This paper integrates facial expression and conversation analyses. The goal of integrating these methods is to cover the individual weaknesses of facial expression and conversation analyses. Facial expression analysis can be used to understand the psychological status of the analysis target. In contrast, it is difficult to understand the context of the conversation behind the situation and/or bargaining between interviewers and guests. While detailed conversation analysis can reveal the content and meaning of an utterance, it is difficult to determine the true intention behind verbal expressions. Creating an integrated analysis will contribute in the following ways.

First, the audience will understand the difference between an utterance and the facial expressions of politicians. If facial expressions coincide with the utterance, the audience will judge the politicians as trustworthy. Conversely, if a difference occurs, the politician will decrease their audience’s political support. Second, understanding the persuasion technique of politicians can be achieved through this analysis. For instance, if politicians emphasize the importance and seriousness of the issue when they show sad facial expressions, we will understand the rhetorical device to attract audience support. Third, we will understand how politicians keep poker faces. Although they may be worried when dealing with tough questions, they might conceal their real attitudes by not using facial expressions that match. We will understand the true intention of politicians by knowing when politicians attempt to conceal the attitudes shown in their facial expressions.

In addition to these goals, we will indicate the strength of qualitative and quantitative analysis using an integrated analysis method. Qualitative analysis can be used to understand specific situations and conditions, but is difficult to generalize. Although quantitative analysis might overlook detailed conditions, it can pursue generalizability. Facial expression analysis as a qualitative method captures the context and psychological status. As a quantitative method, conversation analysis can reveal the link between the strength of a question and facial expression.^[1]

This paper will propose a method of analysis that integrates conversation and facial expression analysis using Japanese political interviews. Also, our qualitative analysis will examine Prime Minister Shinzo Abe’s facial expressions. This paper suggests future political communication research by further developing conversation analysis.

Previous Research

Theoretical Background of Political Communication

Three theories that are relevant to this discussion are the theory of equivocation, the situational theory of communicative conflict, and the face theory. The face theory affects communicative conflict, and it explains why politicians do not answer certain questions. Conversation analysis is used to elaborate on these theories. Subsequently, we will detail the features and the limit of conversation analysis. The theory of equivocation states that politicians equivocate in terms of the sender, receiver, content, and context, and this theory has developed in research on political interviews (Bavelas et al., 1990; Bull, 1998; Bull & Mayer, 1993; Harris, 1991). Bull (1998) indicates that avoidance-avoidance conflict occurs. This conflict implies that the politicians are disadvantaged, no matter their response. Bull (1998) called this the situational theory of communicative conflict (STCC), which analyzes why politicians equivocate in a situation, which has been explained by the concept of threat to face (Bull & Feldman, 2012).

The face theory was introduced by Goffman (1971) and was elaborated on by Brown & Levinson (1987). Bull (1998) have distinguished the faces of politicians, as (1) their own personal face, (2) the face of the party which they represent, and (3) the face of significant others such as supporters, interest groups and so on. Bull & Elliott (1998) stressed that there are several stages of the strength of a threat to face. Feldman & Kinoshita (2017) elaborated that the threat to face varies based on the type of questions and coalition party members or opposition party members based on the strength of threat to face levels.

Facial Expression Research

Facial expressions are seen as reflecting natural feelings, basic emotions, and behavioral intentions Stewart et al. (2009). Discrete emotions theory evolves around six important emotions: surprise, joy, anger, fear, disgust, and guilt; it was presented by Tomkins (1962) and Izard & Malatesta (1987).

Facial expression research has focused on how people recognize facial expressions and how each facial expression has an effect (Ekman & Rosenberg, 1997; McHugo et al., 1985; Sullivan & Masters, 1988). Analyzing facial expressions contributes to the study of emotions in political psychology. The audience can comprehend politicians’ emotions using clues from their facial expressions. The persuasiveness of politicians will increase through effective utterances and facial expressions. However, if the audience understands the discrepancy between utterances and emotions, the possibility of persuasion will decrease, and politicians will lose trust. This paper analyzes the relationship between utterances and facial expressions through qualitative analysis.

Method of Analysis

This paper presents a new analysis method that integrates conversation analysis with facial expression analysis to develop research on political communication. Questions and responses are usually repeated based on the turn-taking system. Thus, the analysis unit is a question-response sequence in conversation analysis. Conversely, during facial expression analysis, it is important to note that momentary expression requires a short time analysis unit. Consequently, the facial expression analysis unit is 0.1 seconds.

The analysis unit is the same due to the difference of the units between conversation and facial expression analysis. When adopting 0.1 seconds as the analysis unit, the content and meaning of the conversation have to be ignored. Conversely, when adapting the question-response sequence as an analysis unit, the momentary expressions are omitted. If the two analysis methods are integrated, it is necessary to overlook their strengths. However, if a question-response sequence is adopted as an analysis unit, momentary facial expressions can be included by using qualitative analysis. Since quantitative analysis does not easily capture a micro-expression, researchers need qualitative analysis for subtle changes in facial expressions.

This paper adopts a question-response sequence as the analysis unit and uses the strength of conversation analysis. The strength of conversation analysis in understanding political rhetoric is to reveal the interaction between the interviewer (e.g., a media person) and guests (e.g., politicians) and what they said in a specific situation (Wooffitt, 2005). A researcher can understand how the content of utterances in natural conversation depends on the context. In other words, we will understand the relationship between an interviewer and a guest, the difference of social positions, and intension of utterance through conversation analysis. When analyzing facial expressions, the expression of the politicians while they talk is used in quantitative analysis. This makes it possible to analyze the expression of politicians seen during an utterance. However, the range of error is different and depends on the duration for which the politician talks. This paper includes an analysis target that shows that it is possible to measure the facial expressions of politicians who speak for more than five seconds.

Face Reader, created by Noldus Information Technology (the Netherlands), is software that can analyze facial expressions. This software is based on the Facial Action Coding System (FACS)^[2] and can specify 491 points on the face and make 3D models. The software calculates seven facial expressions: neutral, happy, sad, angry, surprised, scared, and disgusted. It does so by measuring videos every 0.1 seconds. Previous research using Face Reader exists, such as Otter et al. (2016), analyzed facial expressions using the software and showed different responses occurred between three uncommon and two common situational videos when raters measured real-time responses. Also, Gayoso (2020) compared the speech of Donald Trump and Vladimir Putin and explained that Trump’s speech was delivered with a sad expression because he wanted to emphasize the seriousness of his point.

Analysis Method and Target

First, the subject for the qualitative analysis is Prime Minister Shinzo Abe, who was invited to the Nichiyō Tōron^[3] program that aired on January 8, 2017^[4]. The program was titled “How Politics Changed in 2017: Ask Political Leaders.” Prime Minister Abe’s interview, in which he was asked 28 questions, lasted for 30 minutes, after which other political leaders were interviewed. Second, the subjects of quantitative analysis included ten coalition party members. The programs that were analyzed are Shin Hōdō 2001^[5] aired by Fuji Television, Puraimu Nyūsu^[6] by B.S. Fuji, and Gekiron Kurosufaya^[7] by B.S. Asahi, in addition to Nichiyō Tōron.

Analysis Results

Qualitative Analysis

A qualitative analysis was conducted using an interview with Prime Minister Abe, and the conversation points where his expression is happy, surprised, disgusted, and sad are indicated.

Figure 1.The Flow of Facial Expressions and Topics

Figure 1^[8] shows Prime Minister Abe’s flow of facial expressions. Neutral expressions occupy around 40–50% of the interaction analyzed, indicating he seems to be calm while he answers questions. However, other facial expressions suggest that he is not completely calm. The topics mainly include diplomacy and national security, and he is asked about the Diet’s (The House of Representatives) dissolution towards the end of the interview.

The conversation in this interview relates to current political issues. For instance, questions 6 to 9 are about the relocation of the Futenma air facility, which is an important issue related to national security. The military base burden of Okinawa Prefecture is expensive. The government is considering moving the military base from Futenma to Henoko, but the residents of Okinawa opposed this. The question in this interview is how to resolve this situation. Another example is the question about the diplomatic issue of Japan-Russia relations, which concerns how to proceed with joint economic activities in the northern territories.

Happy Expression

Prime Minister Abe had a happy facial expression two times during this interview. A happy facial expression does not mean that the person is feeling happy but just that he or she is smiling. For example, he smiled when answering the second question on the Liberal Democratic Party (LDP) presidential election^[9].

Shimada: I heard that the LDP presidential term would soon be extended to nine years, which would mean three terms. Do you believe that you will continue to serve as the president of the LDP for the third term starting in autumn 2018?

Abe: To begin with, I think I would like to meet everyone’s expectations first, and I want to promise that. Going forward, I would want to judge what to do by looking at these results.

During this scene, Abe was asked about the expected change in the rule regarding the LDP president’s term, which was changed in March 2017. This decision was made with the understanding that the long-term administration of Prime Minister Abe would gain the understanding of party members and the people. This interview was conducted before the rule was changed.

While answering this question, Prime Minister Abe had a smile as he said, “To begin, I think I would like to meet everyone’s expectation first, and I want to promise that.” Although what he said indicated his attitude towards work, it was said with a smile. Hence, there was a difference between verbal and facial expressions. This indicates that he felt happy and displayed a happy expression when the interviewer discussed his long-term administration.

Surprised Expression

We will now discuss the expression of surprise that is characterized by the raising of eyebrows. Although raised eyebrows are considered an expression of surprise, people also raise their eyebrows when they are emphasizing something^[10].

Shimada: Yes, going to the issue about the U.S. forces in Japan, specifically the Osprey accident that wreaked havoc in Okinawa last month. Onaga, the governor of Okinawa, asked for the withdrawal of Osprey deployment again. What is your opinion about this demand?

Abe: This time, an Osprey had an accident and had to make an emergency landing to refuel. We called for a stop of operations once, but they did not really respond… (Interruption)

This scene dealt with the issue where the governor of Okinawa had demanded the withdrawal of Osprey deployment. Abe moved his eyebrows when he said, “Osprey had an accident.” He moved his eyebrows to emphasize the importance of this issue. While addressing this question, it was evident that he was thinking and responding while looking down, and, after that, he raised his face to explain the situation to the audience. He realized that it was important to explain the situation to the audience, and, consequently, he moved his eyebrows. A surprised expression is useful as a tool to determine what the politicians are focusing on in a political interview.

Disgusted Expression

A disgusted expression indicates a feeling of discomfort; in general, politicians do not express this feeling often. The next example is the scene in which Abe was again asked about the dissolution of the Diet. Abe answered that “we do not think that dissolution is being considered,” before this interaction. Even though he had already explained that there would be no dissolution, the interviewer asked him the same question again.

Shimada: After that, the council of the government will suggest a draft about adjusting the districts of reapportionment of Diet seats by the end of next May. Is there a possibility that you may dissolve the Diet before or after submission of the new draft?

Abe: Just as I mentioned before, I am not considering the dissolution of the Diet at this point. My focus is on the present, and I will try to put all my effort into my work.

Although Abe had already addressed the question during the interview, the interviewer, Shimada, referred to the adjusting districts and again asked him the same question. At this point, Abe showed a disgusted expression as he said, “Just as I mentioned before.” As he was asked the same question again, his expression indicated discomfort.

Sad Expression

A sad expression does not necessarily indicate sadness. Politicians choose their words carefully while using this expression. When a politician looks down and has a troubled expression, it is measured as a sad expression.

Shimada: Regarding the Korean peninsula, North Korea holds the key to East Asian security, and it has an ominous presence. They repeat nuclear tests and launch missiles. The abduction issue^[11] has also not progressed. According to you, what are the factors that can help in solving these problems, and how do economic sanctions have an impact on the issue of abduction?

Abe: Well, North Korea has been conducting nuclear tests and launching missiles since the time of the Kim Jong-un administration. Conversely, Japan has been a part of the United Nations Security Council Resolution 2321 while it has also demonstrated leadership in the United Nations. In that framework, Japan has imposed economic sanctions. Until the abduction issue is resolved, these sanctions will be in place. We would like to do our best to solve the abduction issue under the principles of dialogue and pressure, and action versus action against North Korea. If the abduction issue does not resolve, the relationship between Japan and North Korea will not normalize, and North Korea will not be welcomed as a part of the international society. In other words, if North Korea wants to open the path to the future, the abduction issue has to be resolved. It is imperative that we respond to North Korea so that they can understand the implication of the situation.

Abe’s expression was sad when he said, “If the abduction issue does not resolve, the relationship between Japan and North Korea will not be normalized.” He may have assumed that victims of abduction or stakeholders would be watching his interview, and that was the reason for his sad expression. Prime Minister Abe ensured that his facial expression is related to the context, specifically when the topic was as sensitive as abduction. Besides, to express sadness, he also indicated the intent to look for a solution by saying, “We would like to do our best to solve the abduction issue.” This qualitative analysis suggests that the content of an utterance is related to facial expressions in political interviews.

Quantitative Analysis

Our quantitative analysis used a question-response sequence as an analysis unit, and the data set was made from ten interviews. The interviews focus on coalition party members only.^[12] We analyzed 160 question-response sequences. The degree of question strength is coded from 1 to 5. Level “1” means no threat, while level “5” shows a strong threat. Two coders assessed questions of a threat to face by using an original coding sheet (Feldman & Kinoshita, 2019). The reliability of the threat to face question was confirmed (κ = .814, p < .01).

When the question is a high-level of strength question, the answer is difficult for the politician. We assume that, when the question is tough, the facial expression of politicians will be sad, scared, or disgusted. Conversely, politicians do not show a happy expression.

In identifying the regression model, the dependent variables are six facial expressions: happy, sad, angry, surprised, scared, and disgusted. Threat to face is the independent variable, and the age of politicians and TV programs are the control variables. The TV programs are Shin Hōdō 2001, Gekiron Kurosufaya, and Nichiyō Tōron. These TV programs are a dummy variable (Puraimu Nyūsu is in the reference category). Dependent variables are continuous variables; thus, we will use least squares multiple regression.

Table 1 contains the results of multiple regression analyses. These analyses show how politicians use facial expressions during interviews, and independent variables predict their reasons. Adj-R² shows how the independent variables explain dependent variables. The Adj-R² range from .203 to .561. Our analysis indicates a significant threat to face effect on sad and scared expressions. The coefficient of the threat to face on sad is positive, with a 5% significance level. Politicians show sad expressions when they are asked questions with a high level threat to face. Tough questions annoy politicians and, consequently, they show sad expressions. The coefficient of the threat to face on scared is negative with 1% significance. When questions have a threat to face, politicians do not show that they are scared. This result indicates that politicians try to avoid any facial expression to express that they feel threatened.

The results seen in the TV programs are different. The coefficients of TV programs (Shin Hōdō 2001, Gekiron Kurosufaya, and Nichiyō Tōron) on happy expressions are negatively significant, which indicates that politicians have a happy expression on Puraimu Nyūsu. Puraimu Nyūsu is an almost two-hour political interview; hence the interviewer ensures that a harmonious atmosphere is maintained. Also, sad expressions are seen on Shin Hōdō 2001 and Gekiron Kurosufaya; angry and disgusted expressions are seen in Puraimu Nyūsu; surprised expressions in Shin Hōdō 2001 and Nichiyō Tōron; and scared expressions in Gekiron Kurosufaya. Older politicians usually do not show happy, angry, and surprised expressions but have a disgusted expression.

Discussion

This paper integrated facial expression analysis with conversation analysis. Qualitative and quantitative analysis methods have been analyzed using conversation analysis. The qualitative analysis shows happy, surprised, disgusted, and sad expressions in political interviews with Prime Minister Abe. Qualitative analysis also shows that there can be a difference in verbal and facial expressions. The results indicate that when Abe is asked the same question, he unconsciously has a disgusted expression.

Quantitative analysis using multiple regression indicates a threat to face effect on sad and scared facial expressions. Tough questions make a politician’s face sad, and politicians attempt to avoid a scared expression since the negative coefficient was significantly obtained in the threat to face.

An integrated method of conversation and facial expression analysis suggests new methods to analyze to political interviews. When politicians answer a question, the facial expression shows their psychological feelings. For example, a happy expression might be different in verbal expression, and prudent utterance of politicians can conceal happy feelings. When they are asked difficult questions, they have a sad expression. The TV audience will understand that, when a politician has a sad expression, he or she has been asked a face-threatening question. This quantitative analysis suggests that politicians try to control facial expressions as they attempt to conceal scared expressions.

It will be assumed that there are positive and negative effects on how facial and verbal expressions affect public opinion. Positive effects include molding public opinion by the facial and verbal expressions of a politician. Politicians can make positive images related to their political party by refining their message. Conversely, the negative effect could create a bad image or negative public opinion about their political party. It will be assumed that expressions and utterances that are not received favorably by the voters can lead to loss of political support. The expressions and remarks that have a positive or negative influence on audiences need to be verified through further research.

Political interviews are used as a tool that moves the audience as politicians directly talk to the audience/masses. Voters develop public opinion when they watch political interviews to judge and evaluate the administration and hear the criticism of the opposition parties. Voters should make appropriate judgments based on the verbal and non-verbal behavior of politicians.

Biographical Note

Ken Kinoshita (Ph.D., Doshisha University, Japan) is an assistant professor at the Faculty of Social and Environmental Studies, Fukuoka Institute of Technology, Fukuoka, Japan. His research focuses are in the Japanese parliament and political communication.

He can be reached at the Faculty of Social and Environmental Studies, 3-30-1 Wajirohigashi, Higashi-ku, Fukuoka, Fukuoka Institute of Technology, or by e-mail at kinoshita@fit.ac.jp.

Date of submission: 2019-11-19

Date of the review result: 2020-02-20

Date of the decision: 2020-03-07