Leafleting Outreach Study Analysis (Fall 2013)

This page presents an analysis of the results of ACE’s 2013 study on leafleting, prepared by ACE staff with help from Statistics Without Borders. See the study design page for more details about the methodology of the study.

Pre-analysis Plan

“Survey respondents who report having received either Vegan Outreach leaflet, even if they report also having received the control leaflet, will be considered part of the experimental group. Respondents who report only having received the control leaflet will be considered part of the control group. Respondents who do not report having received any leaflet we distributed will not be considered part of either group.

The primary goal of analysis will be to determine whether rates of conversion to vegetarianism and veganism during the study period are higher among subjects in the experimental group than among subjects in the control group. Depending on the final survey design, the analysis may also consider whether respondents in the experimental group have reduced their rates of meat and animal product consumption more than respondents in the control group, even among those respondents who do consume these products at the time of the survey.”

Main Results

Assignment of Experimental and Control Groups

Our study design required subjects to identify which flyers they had received during the study period, if any, so that we could assign them to experimental or control groups as appropriate. Subjects were shown images of several flyers and asked to indicate any that they had received. The flyers shown included:

  • Two Vegan Outreach flyers, one of which we distributed at each school
  • The back of one of these flyers, because the flyers were distributed with this side facing upwards at one school
  • The control flyer about puppy mills
  • A fictional flyer with a picture of a cat.

Subjects were also able to respond that they were unsure which flyers they had received or that they had not received any flyers.

Our original plan was to assign subjects to the experimental group if they reported receiving the Vegan Outreach leaflet that we distributed at their school, regardless of which other leaflets they reported receiving, and the control group if they reported receiving the control flyer and had not been assigned to the experimental group. Because our volunteers tried to distribute similar numbers of both types of flyer, we expected the control group to be smaller than the experimental group, but not very much smaller. We expected to have a large number of subjects assigned to neither group.

The actual group assignments differed from our expectations, with 123 participants in the experimental group, 23 in the control group, and 477 in neither group. Of the participants assigned to the experimental group, 21 reported receiving the control flyer. Additionally, 101 participants spread throughout all groups reported receiving flyers we did not distribute at their location. We discuss these participants in more detail below.

While our further analysis takes the originally designated control group into account where possible, we also used the group that received neither flyer as a control. The extremely small number of people who reported receiving only the control flyer means that no reasonably probable effect of leafleting would produce statistically significant results when compared to that control group. To better gauge which of our results were meaningful, we used the larger control group despite our concerns that this group would not be as well-balanced against the experimental group in characteristics besides which flyer they had received.

Conversion to Vegetarianism and Veganism

Because previous studies have found that many people who self-report that they are vegetarian also report recently having eaten meat, we measure conversion rates to vegetarianism and veganism by the number of people who reported that they had stopped consuming specific types of animal products. A summary of those results is available in the table below. Percentages are taken out of the number of participants in each group who responded to both questions about a particular food item, which vary by item and in general are slightly smaller than the total group sizes given in the previous section.

experimental control no leaflet
red meat stopped eating 7 0 6
started eating 0 0 7
net change 7 0 -1
net change (%) 6% 0% 0%
poultry stopped eating 5 0 3
started eating 0 0 3
net change 5 0 0
net change (%) 4% 0% 0%
fish stopped eating 3 0 12
started eating 2 1 4
net change 1 -1 8
net change (%) 1% -5% 2%
eggs stopped eating 2 0 7
started eating 2 0 6
net change 0 0 1
net change (%) 0% 0% 0%
dairy stopped eating 1 0 4
started eating 3 0 4
net change -2 0 0
net change (%) -2% 0% 0%

Because of the small absolute numbers of people who started or stopped eating the products in questions, not all these differences are meaningful. Using a generalized linear model and chi-square test, the differences in change of consumption of red meat and poultry (but not fish) between the experimental group and the group that received no flyer were statistically significant (p = 0.0014 and p = 0.0085 respectively). Comparisons between the experimental group and the group that received the control flyer were not statistically significant for red meat, poultry, or fish. We did not run this test on the data regarding eggs or dairy, but we believe that these differences would also not be statistically significant.

We generated 95% bootstrap confidence intervals for the difference in percent net change between the experimental group and the group who received no flyers for each animal product. Positive differences indicate that a higher percentage of subjects in the experimental group quit eating a product than in the no-flyer group. The confidence intervals for red meat and poultry were (2.0%, 11.5%) and (0.9%, 8.7%) respectively. These were the only intervals that did not contain 0; the intervals for fish, eggs, and dairy were (-5.2%, 3.4%), (-4.0%, 3.5%), and (-5.7%, 1.8%) respectively.

Meat Reduction

We also considered overall reduction or increase in meat consumption throughout the sample. We were less certain that the results of this analysis would correspond neatly with reality, because it is more difficult for respondents to remember how often they eat a type of food than whether they eat it at all. However, previous studies had found large numbers of respondents reporting that they had decreased but not eliminated consumption of certain animal products. In our sample, the population overall reported decreasing meat consumption over the three month study period, but that decrease was not significantly associated with the leaflet received. We did not analyze reduction in dairy and egg consumption, which we would expect to be more difficult to interpret in part because eggs and dairy are often present in food in non-obvious ways.

We analyzed meat reduction in two ways. The first categorized respondents’ consumption of red meat, poultry, and fish as having increased, decreased, or stayed the same over the three months, using the frequencies respondents reported for their consumption at the start and end of the three month period. In the sample as a whole, fewer reported increasing than decreasing how often they ate red meat (17% and 21%), poultry (16% and 19%), and fish (10%, 22%). However, chi square tests did not find that the experimental group differed from the other groups in the proportion of people who had increased or decreased consumption of any of these types of meat.

Even if the number of people who increased or decreased their meat consumption did not depend on the leaflet received, it might be the case that people who receive leaflets decrease their consumption by more than people who do not. Our second analysis addressed this possibility by considering the size of the change each respondent reported, in number of times per week a food was consumed. Our models did not find a significant effect of the leaflet received in predicting change in number of times per week an individual consumed red meat, poultry, fish, or all of these meats combined.

Additional Results

Respondent Characteristics

Our sample was reasonably representative of the audience targeted by many leafleting groups, with 57% of respondents being female and the median age being 21 years. There was no significant association between flyer received and age or gender.

Respondents in our study were asked both to describe their diet and to indicate the frequency with which they ate specific foods. As in other samples, a substantial percentage (45%) of those who indicated that they were vegetarian or vegan also reported consuming some type of meat.

Social Desirability

Our survey included a social desirability instrument, the RAND SDRS-5. Respondents who score high on such an instrument are believed to be generally more likely to be answering questions according to what they feel is socially desirable or socially acceptable than respondents who score lower. Accordingly, if responses on another item are correlated with social desirability scores, researchers must consider the possibility that some respondents have provided desirable rather than truthful responses to that item.

In our sample, social desirability scores were on average higher for women and tended to increase with age. In the experimental group, a higher social desirability score was also correlated with a larger decrease in the frequency of meat consumption (all types combined). There were also statistically significant differences between the experimental group and the no-flyer group in the association between frequency of meat consumption and social desirability score for every type of meat except fish alone. These findings suggest that social desirability does play a role in reported meat consumption, especially for respondents who have been exposed to vegan outreach materials. Care should be taken to minimize and track its effects.

More About Group Assignment

We had two concerns about group assignments once we tabulated the data. First, because our intended control group was so small, we needed to consider to what extent the larger no-flyer group is an appropriate substitute. As discussed above, it did not differ significantly in age or gender from the experimental group, which is helpful because age and gender have both been found to correlate with vegetarianism in the past, and theoretically they should also correlate with openness towards diet change and willingness to take a leaflet. The no-flyer group and the experimental group did show different associations between social desirability and responses to diet questions, but it is not clear whether this should be taken to be an effect that would precede or follow acceptance of a flyer.

One set of measurable variables that should have preceded acceptance of a flyer is responses to diet questions about the start of the study period. If the experimental group were more sympathetic to animals before the study started than the no-flyer group was, we might expect to see higher rates of vegetarianism or veganism among them at that time. In the table below we show the percentages of respondents who reported that they never ate each of the products at the start of the study period (three months before the survey took place). These provide some reassurance that the groups were not extremely different in composition, but of course cannot completely resolve all concerns.

experimental no leaflet
red meat 11% 11%
poultry 8% 7%
fish and seafood 17% 13%
eggs 8% 4%
dairy 9% 4%

We also had concerns about respondents’ recall of which flyer they received, given the many respondents who reported receiving flyers we did not think they would have received. Using the Adopt a College website, we searched for distributions of Vegan Outreach leaflets that might explain this pattern. We found such distributions at two schools, and when they were accounted for, 61 of the confusing responses remained. Five of the subjects reported receiving the completely fictional flyer, and based on this we believe that many of the other respondents may also have been erroneously reporting which flyer they received, including some whom we classed in the experimental group. This is especially a concern because after taking a survey about diet, we would expect respondents to be more likely to mistakenly report that they had received a flyer with a picture of a farm animal than one with a picture of a cat. Also, respondents who really received a flyer in a previous semester may have been more likely to report receiving that flyer than to report receiving a flyer they had never seen before. We aren’t sure how best to deal with these concerns.

We did not re-run the full analysis with the new experimental, control, and no-flyer groups based on what we found on Adopt a College. Running multiple analyses of the same data can lead to problems differentiating true effects from effects that appear by chance when many tests are performed. Also, we don’t know whether the distributions by other groups occurred at the same sites and time of week as our leafleting and surveying did, which could affect whether it was plausible for subjects to have received a leaflet, even if it was handed out on their campus. However, to get a sense of how much this would change our analysis, we recalculated the set of confidence intervals for difference in percentage of respondents who stopped eating animal products. The new intervals were: for red meat, (1.5%, 10.0%); for poultry, (0.7%, 7.3%); for fish, (-5.0%, 2.7%); for eggs, (-3.6%, 3.1%); and for dairy, (-4.9%, 1.7%).

Confidence interval plot

Conclusions

We found support for claims that distributing leaflets from Vegan Outreach (or similar leaflets published by other groups) causes a small percentage of respondents to go vegetarian or give up eating specific types of meat. Our study showed surprisingly large effect sizes for respondents giving up red meat and poultry with respect to previous work, and a surprisingly small effect size for respondents giving up fish and seafood. Because our sample of respondents who reported receiving a leaflet was smaller than that in, for example, the study conducted by The Humane League and Farm Sanctuary, our effect sizes are likely to vary more from the true effect size. In that study, the effect sizes ranged from about 1% giving up fish to about 3% giving up red meat. Looking at the two studies together, our study suggests that the effects they found on people giving up red meat and poultry are in fact effects of the leaflets, but effects on fish, egg, and dairy consumption may not be. Because the experimental group in the earlier study was much larger than the experimental group in this study, we do not think the true effects of leafleting in terms of people giving up red meat or poultry are as large as those found in this study; the earlier, lower estimates are probably more accurate. On the other hand, where our study found smaller effect sizes or no effect, our findings reinforce concerns about the previous study’s design.

We did not find support for claims that distributing leaflets from Vegan Outreach (or similar leaflets published by other groups) causes the overall population who receives the leaflets to reduce their meat consumption. Rather, we found an overall pattern of reported reduction in meat consumption throughout the sample. We do not know whether this pattern reflects real decreases in consumption or only systematic errors in recollection. College students may in general be trying to reduce their meat consumption, or they may be reporting different consumption in the past than the present for some other reason. These findings suggest that studies without control groups may find or have found reductions in meat consumption that should not properly be attributed to the intervention being studied, at least in their entirety.

In addition to this finding reinforcing the importance of control groups, we found signs that self-reported exposure to interventions is not always accurate and that social desirability bias does operate on reported meat consumption in ways that interact with exposure to interventions. These findings impose caution on our interpretations of the results of this study. They also underscore the importance of careful study design more generally.

Resources

Survey and Flyer Selection Sheet
Full Dataset
Dataset Interpretation Guide
Statistical Reports (zip file)