Facial expressions are extremely important in understanding what the other person is saying, according to PhD research by Naomi Nota. Frowning in particular turns out to play a major role. Nota will defend her PhD thesis at Radboud University on 6 December.
In the past, linguists mainly looked at speech and how it affects language processing. "But in recent years, researchers have extended their investigations to much more than speech alone," Nota explains. "For example, we get a lot of information from people’s lips and how they use their hands when speaking." In her research, she looked specifically at the face: how do we use it when we talk to someone? To this end, Nota and her colleagues analysed video recordings of people engaged in conversation. "We then went through the videos frame by frame and noted every single movement: from frowning, to raised eyebrows or squinting. We wanted to know what happens in people’s faces when they talk." The researchers went through hours of material. "I think we spent 1,000 hours looking at facial expressions."
Eyebrows
Their analysis showed that eyebrows in particular play a major role in conversation. "I noticed that when asking an informative question - like ’What will you be doing tomorrow’’ or ’Where is the train station’’ - people often furrowed their eyebrows. Especially at the start of the question. It happens unconsciously."
In the next step of her research, she used virtual avatars that copied very precisely the timing and eyebrow movement of the test subjects. She also used avatars that did not. The avatars then asked a question, or uttered a statement, and test subjects were asked to determine as quickly as possible which of the two it was. "We discovered that test subjects found it easiest to quickly and accurately detect a question when the avatar frowned. But even if the eyebrows moved up and not down, they found it easier to identify a question."
Nota also looked at the intensity of eyebrow movements. "It makes a big difference whether you use the natural intensity of an eyebrow. The more similar it was to eyebrow movements in real life, the better a test subject was able to tell whether something was a question or a statement."
The practical implementation of Nota’s research may lie, for example, in the development of AI. "I can imagine that it would be good if the avatars or robots used in healthcare, for example, also had eyebrows. It would hopefully help people understand robots faster and better."
