Using Temporal Features of Observers' Physiological Measures to Distinguish between Genuine and Fake Smiles

Md Zakir Hossain*, Tom Gedeon, Ramesh Sankaranarayana

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

11 Citations (Scopus)


Future affective computing research could be enhanced by enabling the computer to recognise a displayer's mental state from an observer's reaction (measured by physiological signals), using this information to improve recognition algorithms, and eventually to computer systems which are more responsive to human emotions. In this paper, an observer's physiological signals are analysed to distinguish displayers' genuine from fake smiles. Overall, thirty smile videos were collected from four benchmark database and classified as showing genuine or fake smiles. Overall, forty observers viewed videos. We generally recorded four physiological signals: pupillary response (PR), electrocardiogram (ECG), galvanic skin response (GSR), and blood volume pulse (BVP). A number of temporal features were extracted after a few processing steps, and minimally correlated features between genuine and fake smiles were selected using the NCCA (canonical correlation analysis with neural network) system. Finally, classification accuracy was found to be as high as 98.8 percent from PR features using a leave-one-observer-out process. In comparison, the best current image processing technique [1] on the same video data was 95 percent correct. Observers were 59 percent (on average) to 90 percent (by voting) correct by their conscious choices. Our results demonstrate that humans can non-consciously (or emotionally) recognise the quality of smiles 4 percent better than current image processing techniques and 9 percent better than the conscious choices of groups.

Original languageEnglish
Article number8509156
Pages (from-to)178-188
Number of pages11
JournalIEEE Transactions on Affective Computing
Issue number1
Publication statusPublished - Jan 1 2020
Externally publishedYes


  • Affective computing
  • fake smile
  • genuine smile
  • physiological signals
  • temporal features

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Using Temporal Features of Observers' Physiological Measures to Distinguish between Genuine and Fake Smiles'. Together they form a unique fingerprint.

Cite this