I Can Feel You Are Smiling Happily: Distinguishing between Real and Posed Smiles from Observers' Peripheral Physiology

Abstract

The ability to recognise facial expression is crucial to human social interaction and plays an integral part in most social interaction scenarios. Being able to understand genuine facial expressions is considered a valuable social skill. In this context, a smile is treated as a major facial display that can be recognised easily, but can be confusing. Genuine smiles are usually labelled as both showing and feeling happiness, but posed smiles are far less likely to be understood as feeling happiness than as showing happiness.

Previous studies considered either observers’ verbal responses or smilers’ facial features in distinguishing between real and posed smiles. Some of these studies also considered smilers’ physiological signals in this context, but none of the studies considered observers’ physiological responses. The latter concept is considered in this thesis. This thesis outlines a review of relevant physiological signals and related facial expression recognition as well as affective computing literature. This thesis also presents a novel computational approach, called the independent approach, to train the classifiers with a totally independent dataset that uses a ‘leave-one-subject-and-one-stimulus-out’ cross validation technique. Before discussing observers’ physiological reactions to the viewed smile faces, it addresses observers’ pupillary changes to viewed graphical visualisations. Thus, it discusses non-invasive and unobtrusive physiological sensors and relevant computational techniques for discriminating between observed smiles (real versus posed) with a preparatory examination of some of these physiological signals in observed visualisations (radial graphs versus hierarchical graphs).

At the beginning of this thesis two graphical visualisations (radial and hierarchical) are chosen to distinguish between observers’ verbal responses and their pupillary responses. The graphical visualisations are snapshots of a kind of data used in checking the degree of compliance with corporate governance best practice. The radial visualisation shows the connections between the board members of BHP Titanium Pty Ltd and ICI Australia Petrochemicals. The hierarchical visualisation exhibits the connections between the board members of the National Australia Bank and Sydney 2001 Olympics. Six, very similar, questions were asked from each participant for each visualisation and found that although observers are not able to distinguish between the radial and hierarchical graphs according to their verbal responses, their pupillary responses can. The analysis of the experiment also shows that observers are verbally 81.0% and physiologically 95.8% correct.

The outcomes from the above experiment motivated me to design experiments to analyse the changes of physiological responses to = viewed human facial expressions instead of graphical visualisations. In this regard I have chosen smiles as a human facial expression, because it generally means happiness and/or is used to motivate others. For example, a speaker can be motivated from audience smiles. On the other hand, people can smile from feeling or by acting or posing the smile. Thus discriminating real from posed smiles is important in human centred computing, for computers to ‘understand’ smilers’ mental states.

Experiments are designed and conducted to acquire observers’ physiological signals with their verbal responses while watching smile videos. A number of smile videos are collected from the literature and processed to show them to participants/ observers. The processed smile videos are classified as real/ posed according to their elicitation provenance. The physiological signals in the data sets include pupillary response, galvanic skin response (GSR), electrocardiogram (ECG) and/ or blood volume pulse (BVP).

Observers’ physiological signals are analysed by developing computer programs via signal processing and machine learning techniques. Several methods are used to develop this computational model such as noise removal, feature extraction, feature selection, fusion, classification, ensemble learning, and so on. In this connection, filtering is considered to remove noise from the recorded signals; normalisation is used to overcome age effects on the signals; interpolation is applied to reconstruct the missing values; principal component analysis (PCA) is employed to eliminate the lighting effects from the recorded signals, etc. Neural networks (NN), support vector machines (SVM), relevance vector machines RVM), k-nearest neighbours (KNN), and ensemble classification techniques are employed to develop the classification model in this thesis. Final results show that participants are verbally 52.7% (on average) to 68.4% (by voting) correct whereas they are physiologically 96.1% correct using an independent approach and ensemble technique.

Overall, this thesis contributes a significant dimension to developing a computational model to differentiate between real and posed smiles as well as offering insights into observers’ understanding of radial and hierarchical visualisations based on their peripheral physiology. In other words, this thesis measures observers’ feelings and reactions to the observed images and smiling face visualisations. As physiological signals are not easy to control voluntarily and provide spontaneous and non-conscious outcomes, the outcome of this thesis identifies an authentic and important difference between what observers’ say and feel, that is, reported verbally (say) and reflect physiologically (feel), respectively. Further research suggests including sensor technologies the care-givers or users of avatars to understand human psychology via their recorded physiology. Show more