Skip to main contentSkip to main content
You have permission to edit this article.
virginia tech

Face time: Virginia Tech research examines AI profiling

  • 0

ROANOKE — A raw first impression or love at first sight might be human intuition, but what if other important life moments, like being hired to a new job, or getting approved for a bank loan, were shaped by computerized decisions about friendly faces?

In the ever-advancing field of artificial intelligence, one consumer behavior researcher at Virginia Tech said she has noticed discussions ramping up about increasingly consequential, and controversial, uses of facial profiling technology.

“It sounds dystopian, like, you know, ‘my future depends on an algorithm.’ But it’s becoming more and more commonplace,” said Shilpa Madan, an assistant professor of marketing. “We don’t know which employers might actually be putting our CV into the scanner.”

Facial profiling goes a step beyond the facial recognition technology that allows someone to unlock their smartphone with a glance. With facial profiling, a type of artificial intelligence, called machine learning, is used to decipher from peoples’ outward appearances their otherwise unseen character traits, like personality, trustworthiness, intelligence, and perhaps even political or sexual preference.

“There are folks on both sides here,” Madan said. “A lot of us do not think that our face has anything to do with our personality, but a lot of people think that our genes are expressed in our face, so how we look may have something to do with our personality.”

While research on the accuracy of facial profiling technology remains inconclusive, the idea is nonetheless gaining traction in some industries and sectors, she said. These are advancements worth questioning.

“Not facial profiling’s innocuous uses, but really consequential uses,” Madan said. “Not things that are fun, you know: upload your selfie to this app, and we’ll tell you how extroverted you are, or how well you match with your partner.”

Perhaps a less fun use, Madan said: What if companies assess candidates’ employability by putting peoples’ photographs through visual profiling algorithms? Or if financial companies use facial profiling to determine loans and interest rates?

“People who want facial profiling to come into the mainstream have many good reasons, but also, this raises so many concerns about bias and discrimination,” Madan said. “Using it for such big decisions, the scope of error is enormous.”

And another concern is that most facial profiling algorithms are black boxes, lacking much, if any, transparency about how the technology works, she said.

“They’re trade secrets. We don’t know how it’s processing things,” Madan said. “There’s little opportunity to actually unravel it, and see what it’s doing.”

It might not be facial profiling, but Virginia lawmakers this year approved certain law enforcement uses of facial recognition technology, in a long-debated bill that passed by a narrow margin. Other facial recognition applications have spurred legal arguments about whether private interests should have access to the technology.

“There are financial firms that are using facial profiling to assess customers’ risk tolerance, and recommend investment strategies to them,” Madan said. “An insurance company allegedly was using facial profiling to determine insurance premiums.”

Listen now and subscribe: Apple Podcasts | Google Podcasts | Spotify | RSS Feed | Omny Studio

A company in Israel called Faception said it used machine learning to create the first facial personality analytics tools, capable of automatically developing a personality profile based on a person’s facial image. Among its uses, the Faception website said the technology can enable “security companies and agencies to more efficiently detect and apprehend potential offenders or criminals before they have the opportunity to do harm.”

Passing through airport security and being singled out by an algorithm that thinks you look criminal? Perhaps an extra heaping of dystopia, even if the technology could shorten lines at security checkpoints.

“It’s a hotly debated idea, even within academia and research,” Madan said. “Research evidence is very mixed. It’s not conclusive, at least not yet. Maybe 10 years down the line, we’ll have more clarity.”

The increase in discussions and contemplated uses of facial profiling evoked a question in Madan, who specializes in lay beliefs, or consumers’ naïve theories about the world. This question became the basis for collaboration on a research project, published recently by the American Psychological Association, Madan said: “At a psychological level, what predicts people’s support for facial profiling, everything else being equal?”

Close to 3,000 people from across the United States took part in 10 related research studies, including questionnaires and experiments for participants who were led to believe they were either very good or quite bad at making appearance-based judgments.

Results showed that support for facial profiling fundamentally depends on how much someone adheres to this lay belief, Madan said: “The more you believe that peoples’ appearance reveals their character, the more you support facial profiling for consequential uses, really big uses, even on yourself.”

People hold the lay belief that appearance reveals character to varying degrees, rather than thinking it is outright true or false, she said. Associations between good-natured and good-looking are often made early in childhood, through popular culture.

As an example, Madan quoted the good witch from “The Wizard of Oz,” who in the film answered a question about her beauty by saying that only bad witches are ugly. Just in more recent children’s movies have villains started to be depicted as good-looking, instead of warty and mean.

“That’s what we grew up on. A lot of us were told ‘don’t judge a book by its cover,’ and ‘all that glitters is not gold,’ and ‘people can be wolves in sheep’s clothing,’” Madan said. “But we also grew up on a diet of very pretty looking Disney princesses.”

But it is possible to shake belief that a person’s appearance reveals character, she said. Just rattle the confidence of someone in their own ability to judge a person by looks alone, and support for ideas like computerized facial profiling starts to wane, according to what’s been found.

“We zeroed in on confidence as the underlying reason. Because when you think that someone’s appearance reveals their character, you feel more confident that you and machines can accurately predict this character from their appearance,” Madan said. “We were able to either increase or reduce support for facial profiling by increasing or reducing peoples’ confidence.”

It was a pretty novel study, one that can contribute to the big discussions taking place online and in halls of business and government about facial profiling, and the larger roles machine learning and artificial intelligence can play in modern society.

“It’s important for the discourse, because that’s how people form their beliefs. When they read about these things in print, they think it can be done, but the evidence is very mixed,” Madan said. “Researchers have a long way to go in resolving this basic idea of whether or not we can do this. Is facial profiling even doable for many traits?”

Those and other lingering questions are what make the topic worth further investigating, she said: “People get plastic surgery. Does that change your character? Clearly it does not, but your facial features have changed, right? People put on beautiful makeup and they look like completely different people, so do you then think your character has changed?

“Lots of questions,” Madan said. “Lots more research to unpack.”

“People who want facial profiling to come into the mainstream have many good reasons, but also, this raises so many concerns about bias and discrimination. Using it for such big decisions, the scope of error is enormous.”

Shilpa Madan, assistant professor of marketing at Virginia Tech

Pull quote
* I understand and agree that registration on or use of this site constitutes agreement to its user agreement and privacy policy.

Related to this story

Most Popular

Listen now and subscribe: Apple Podcasts | Google Podcasts | Spotify | Stitcher | RSS Feed | Omny Studio

Listen now and subscribe: Apple Podcasts | Google Podcasts | Spotify | Stitcher | RSS Feed | Omny Studio

Get up-to-the-minute news sent straight to your device.


Breaking News

News Alert