top of page

A/B Testing on the Susceptibility of

Fake News, Gender Bias, and Auditory Attention

Slide1.jpeg
Slide1.jpeg

Introduction

With all kinds of information flooding around people, fake news and misleading information have become a topic of public discussion for the last couple of years. Major cases have been linked to political campaigns and social media platforms such as Facebook and Twitter where people easily spread and share misinformation without notice. * 

Determinants of people believing and comprehending information are vast, and we would never be able to find them all. Therefore we decided to target a possible determinant which is the nature of all human-beings: bias. In this research, we will particularly focus on gender bias and auditory attention. 

The best way to stop the spread of misleading information is acquaintance: knowing what makes people biased and what contributes make fake information convincing.

Slide2.jpeg
Slide3.jpeg

*A December 2016 survey by the Pew Research Center suggests that over 23 percent of U.S. adults have shared misinformation with others. 

Questions & Experimental Design

 

With our A/B test, we set out to answer the following three questions:

  1. Does changing the medium (text, male audio, female audio) of content perception affect the perceiver's comprehension?

  2. Does the heterogeneous causal effect of political affiliation, level of education, average daily news consumption and gender have on causality of tending to believe fake news when assigned a medium of information?

  3. If we consider the respondents with at least a score of 2 out of 3 to have understood the article well, what is the heterogeneous effect of treatment on their perception of news using their score as an instrumental variable?


The information that we used to measure the subject's comprehension and bias were prepared by the Atlantic Council's Cyber Statecraft Initiative.

​

Units of Analysis

Slide5.jpeg

Finding Measurable Content

To test our hypothesis, we found a made up (fake) news article from the Atlantic Council's Cyber Statecraft Initiative. We had the content in text form and got a male member of our group to record his voice reading the article aloud, while his wife recorded her voice for the female audio.

​

article.png

Randomization Treatment

To evaluate the causal effect of the medium of learning source, we randomized the source shown to participants in the
experiment. As we used JotForms for our survey distribution, we used the elemental feature on JotForms called "Roll a Dice". Based on the outcome of the dice roll, we assign the participant either text, male audio or female audio voice. For each participant, we updated the medium while maintaining the same content to achieve perfect randomization.

​

Variables

In the survey above, the controlled variable is the medium of the information. For covariates, we polled for gender, political leaning, ethnicity, age, and education level of the subject.
We have two dependent variables:

  • Comprehension - Would the subject get a score of 2 or more out of the three comprehension questions?

  • Credibility - Did the subject believe the mayor was guilty?

Slide6.jpeg

Data Analysis

We obtained 613 total responses to our survey. We went through a data cleansing process to filtered out participants based on a few criteria:

  • Age equal to 100

  • Duplicate IP Address

  • Dice Roll equal to 0 (system error)

Slide7.jpeg

Data Modeling

Since randomization is the key to a successful A/B test, we processed a balance check to make sure our randomization is successful before moving on to further model buildings.

Slide8.jpeg

Data Modeling & Results

​

Causal Effect of Information Medium on the Comprehension Score

We estimated the causal effect of the information medium using ordinal logistic regression with education, sex, age as covariates. Text information is used as a control (reference) group in our model. This model examined the causal effect of the medium of news on the comprehension level or the learning ability of the participant.

We found that getting the information in the form of text document is better than getting the same information through auditory sources irrespective of the gender of the voice. Getting the information in the form of male voice, as opposed to a text document is associated with a higher likelihood of having a poor comprehension, the t-value for InformationMediumMaleVoice is less than -2, and hence statistically significant at 5% level.
Similarly, we can interpret the same conclusion for InformationMediumFemaleVoice, which is also statistically significant at the 5% level.

Slide9.jpeg

Causal Effect of Information Medium and Political Affiliation on Believability

We also estimated the causal effect of information medium and political affiliation using ordinal logistic regression with sex and average news consumption as covariates. Text information is used as a control (reference) group in our model. This model examined the causal effect of the medium of news on how likely was it to believe that the mayor was guilty. We found that as the subjects became increasingly conservative (lean liberal to conservative keeping liberal as a base), the likelihood of believing the fake news - the mayor was guilty - statistically increases. 
An interesting observation is the medium of information - female voice, male voice, texts has no significant impact on the believability of the news.

Slide10.jpeg

Heterogeneous Effect of Comprehension on Believability of the Fake News

In the following heterogeneous regression, we found two things interesting. Firstly, if you were to just look at the male and female voice you are more likely to believe that the mayor was guilty (believe in the fake news). Secondly, if the subject comprehended the article well with a score of greater than or equal to 2, then the subject was more likely to have realized that the article presented was fake. While the believability decreases in the heterogeneous effect of voice and comprehension score, it is statistically significant for the interaction of female and good comprehension. This indicates that if you've paid enough attention to what is being said, you're more likely to catch a woman at it rather than a man.

Slide11.jpeg

Heterogeneous Effect of Post Grad Education on Comprehension and Believability

Another result from the heterogeneous regression is rather conflicting. Graduates were able to comprehend significantly better to a female voice and yet simultaneously were more likely to catch the fault in a man's rendition of the fake news. This conflict would certainly be interesting to explore in the future.

Slide12.jpeg

Conclusions

In this project, we aimed to measure the extent to which gender bias and different delivery mediums can affect the comprehension and belief of misleading information. 

​

  1. The experiment did show that participants who received the text treatment comprehended the article better with a significant difference in scores on the comprehension questions, but we did not see any significant difference across gender or between the treatment groups in terms of believing the article. This suggests that mode of delivery can have an effect on comprehension of information, but we can’t say that we observed any gender bias or that different modes of delivery affected the belief of the fake news story. What we can say though is that certain strata of our subjects did react differently to the fake news that was rendered by male and female voices.

  2. On regressing our covariates on the believability of mayor's guilt, as the subject's political leaning becomes more conservative (Liberal to Lean Liberal to Moderate to Lean Conservative to Conservative), the likelihood of them believing the mayor was guilty increases.

  3. The heterogeneous causal effects were interesting in the sense, that they told us that if you got the treatment of the female voice and if you truly grasped the article you would catch the female voice's lie but not that of the male voice.

Limitations & Next Steps

  1. Strong correlations between variables cannot be confirmed to be causal effect: we observed a strong correlation in self-reported political affiliation and statistics related to believing the article, but we do not have the randomization implementation to confirm the causal effect between the two variables. This might warrant further study of the effect of our different treatments on people of different political affiliations.  

  2. Noisy Dataset: Collecting more data could potentially help us narrow down our conclusions further.

  3. Length of the experimental content: The 2 minutes long audio might be too long in view of the attention spans of the participants. In further researches, we might be able to compare the results between text and audio if we used a shorter content or blurb that would be both quickly listened to and read.

Our Team

  • Akanksha Rawat

  • Ariel Yu

  • Emmanuel Gamma Ibara

  • Sam Duran

  • Sanjana Panicker

  • Vivek Dwivedi

​

My Role

  • Design the experiment

  • Implement the experiment

  • Analyze and interpret the data

​

bottom of page