Do We Need to Disguise Veg Surveys?

Study Design by Peter Hurford

Study Implementation and Analysis by Jacy Anthis

Motivation

.impact, in cooperation with Mercy for Animals, is working on a $70K study aiming to assess the impact of FB veg ads on diet change (details here).  We want a questionnaire for this study that accurately assesses the difference in veg diet change between the treatment group and the control group.

We think it’s plausible that disguising the study (making it less look about vegetarianism) is important for increasing the validity of results, but this comes with the tradeoff of making the survey longer.  Is disguising worth the cost?

Key Question

Does disguising the questionnaire to not look like a study primarily about vegetarianism change how the users respond?

Methodology

To answer this question, we will A-B test two different questionnaires on Amazon’s Mechanical Turk platform.  One questionnaire will be longer, but disguised.  The other questionnaire will be shorter, but not disguised.  We will then compare responses to the two questionnaires, looking for a statistically significant difference between the two groups.

If a statistically significant difference is found and vegetarianism is found to be lower in the non-disguised condition, we can tentatively conclude that disguising the questionnaire does matter.  If a statistically significant difference is not found, we can tentatively conclude that disguising the survey does not matter.

In both cases, more research would be needed to achieve confidence, but this seems good enough for preliminary results to calibrate pre-existing intuitions.  This is intended to be a quick take to inform future research directions.

The Disguised Questionnaire

3.) What is your age? (drop-down)

4.) What is your gender? (drop-down)

The Non-disguised Questionnaire

2.) What is your age? (drop-down)

3.) What is your gender? (drop-down)

Results

300 participants were recruited via MTurk to complete a study with the title "short fun survey :) (<1 minute)" and the description "easy fun psychology experiment".  Participants were then sorted randomly to either get the disguised or non-disguised survey (150 each).  A linear regression was then preformed with a binary variable indicating the condition (disguised vs. non-disguised), the age, and the gender.

While age and gender were statistically significant predictors of consuming less total servings of dairy, chicken, turkey, fish/seafood, pork, beef, and eggs (p < 0.0001 and p = 0.018 respectively), the presence or absence of the disguise did not matter (p = 0.98).

Conclusion

Based on these results, we find preliminary evidence in the direction that disguising the questionnaire does not affect results.  Due to the trade-off described earlier, this preliminary finding is enough for us to conclude it is not worth attempting to disguise the survey.  However, further research should be taken to validate this result.

Raw data can be provided upon request.  Email peter@peterhurford.com.

Possible Errors

Appendix: Reliability of MTurk

See "Doing Important Research on Amazon’s Mechanical Turk?"