index.md (10956B)
1 --- 2 title: "Bayesian estimation of anti-vaccination belief changes" 3 description: How easy is it to change people's minds about vaccinating their children? 4 date: 2015-09-03 5 categories: 6 - Data Science 7 - Science 8 tags: 9 - Statistics 10 - Psychology 11 --- 12 13 ## Introduction 14 15 How easy is it to change people's minds about vaccinating their children? 16 According to a recent study ([Horne, Powell, Hummel & Holyoak, 17 2015](http://www.pnas.org/content/112/33/10321.abstract)), a simple 18 intervention -- which consisted of showing participants images, an anecdote, 19 and some short warnings about diseases -- made participants more likely to 20 support childhood vaccinations. [Here's a good 21 writeup](https://news.illinois.edu/blog/view/6367/234202) of the article if 22 you're unable to read the original. 23 24 The authors [placed their data online](https://osf.io/nx364/), which comprises 25 pre- and post-intervention survey responses for three groups of participants: 26 27 1. A control group 28 2. An "autism correction" group that were shown evidence that vaccines don't 29 cause autism. 30 3. A "disease risk" group that were shown images, an anecdote, and some short 31 warnings about the diseases (such as rubella and measles) that the vaccines 32 prevent. 33 34 I chose to look over this data for a couple reasons. First, I'm friends with 35 two of the authors (University of Illinois Psychologists Zach Horne and John 36 Hummel) and it's good to see them doing cool work. Second, my own research has 37 given me little opportunity to work with survey data, and I wanted more 38 experience with the method. I was excited to try a Bayesian approach because it 39 makes it possible to perform post hoc comparisons without inflating the "type 40 I"" (false positive) error rates (see below). 41 42 Participants were given a surveys with five questions and asked to rate their 43 level of agreement with each on a six-point scale. 44 45 code | question 46 -------------|------------- 47 healthy | Vaccinating healthy children helps protect others by stopping the spread of disease. 48 diseases | Children do not need vaccines for diseases that are not common anymore. *reverse coded* 49 doctors | Doctors would not recommend vaccines if they were unsafe. 50 side_effects | The risk of side effects outweighs any protective benefits of vaccines. *reverse coded* 51 plan_to | I plan to vaccinate my children. 52 53  54 55 The above figure shows the data. Each line represents a single participant's 56 responses before and after the intervention, organized by intervention group 57 and question. Lines are colored by the magnitude of the change in response; 58 blue lines indicate an increase in agreement (toward a more pro-vaccine stance) 59 and red lines indicate a reduction in agreement (a more anti-vaccine stance). 60 61 The JAGS code for the model is part of the source of this document, which is 62 [available through Git](https://git.eamoncaddigan.net/antivax-attitudes/). 63 It uses a Bayesian analog to a three-factor ANOVA, with a thresholded 64 cummulative normal distribution serving as a link function. Such models fit 65 ordinal responses (such as those obtained from surveys) well. The thresholds 66 and variance of the link function were fit independently for each question. 67 The mean of the function was estimated for each response using a linear 68 combination of the levels of the question, the interval (pre-test vs. 69 post-test), the intervention group, and all interactions between these 70 factors. 71 72 ## Results 73 74 ### A "risk" intervention changes attitudes toward vaccination 75 76 When fitting model parameters using Monte Carlo methods, it's important to 77 inspect the posterior distribution to make sure the samples converged. Here's 78 an example of one parameter, the intercept for the mean of the cummulative 79 normal. 80 81  82 83 It's also important to check the predictions made by a model against the data 84 being fit, as "[we cannot really interpret the parameters of the model very 85 meaningfully when the model doesn't describe the data very 86 well](http://doingbayesiandataanalysis.blogspot.com/2015/08/a-case-in-which-metric-data-are-better.html)". 87 Here are response histograms for each question, averaged across the levels of 88 the other factors. Model predictions are superimposed on the histograms, along 89 with the 95% HDI for each response. 90 91  92 93 Since the sampling procedure was well-behaved and the model describes the data 94 well, we can use the parameter estimates to judge the size of the effects. Here 95 are is the estimate of the change in attitude (post-test - pre-test) for each 96 intervention group. 97 98  99 100 These plots highlight the 95% highest density interval (HDI) for the posterior 101 distributions of the parameters. Also highlighted are a comparison value, which 102 in this case is a pre- vs. post-test difference of 0, and a "range of practical 103 equivalence" (ROPE) around the comparison value. The HDI of the posterior 104 distribution of attitude shifts for the "disease risk" group" (but no other 105 group) falls completely outside this ROPE, so we can reasonably conclude that 106 this intervention changes participants' attitudes toward vaccination. 107 108 We can also use the posterior distributions to directly estimate the shifts 109 relative to the control group. Here is the difference between the attitude 110 change observed for both the "autism correction" and "disease risk" groups 111 compared to the attitude change in the control group. 112 113  114 115 The posterior distribution above shows that "disease risk" participants shifted 116 their response about half an interval relative to the control group following 117 the intervention. The "autism correction" participants, however, did not show a 118 credible change in vaccination attitudes. Bayesian estimation replicates the 119 conclusions drawn by Horne and colleagues. 120 121 ### Post hoc comparisons 122 123 An analysis following the tradition of null-hypothesis significance testing 124 (NHST) attempts to minimize the risk of "type I" errors, which occur when the 125 "null" hypothesis (i.e., there is no effect) is erroneously rejected. The more 126 tests performed in the course of an analysis, the more likely that such an 127 error will occur due to random variation. The [Wikipedia article on the 128 "Multiple Comparisons 129 Problem"](https://en.wikipedia.org/wiki/Multiple_comparisons_problem) is an 130 approachable read on the topic and explains many of the corrections that are 131 applied when making mulitple comparisons in a NHST framework. 132 133 Instead of focusing on type I error, the goal of Bayesian estimation is to estimate values of the parameters of a model of the data. The posterior distribution provides a range of credible values that these parameters can take. Inferences are made on the basis of these estimates; e.g., we see directly that the "disease risk" intervention shifts participants' attitude toward vaccination about one half of an interval. Since a single model was fit to all the data, additional comparisons of parameter distributions don't increase the chance of generating false positives. [Gelman, Hill, and Yajima (2008)](http://www.stat.columbia.edu/~gelman/research/unpublished/multiple2.pdf) is a great resource on this. 134 135 For example, we can look at the size of the shift in attitude toward each 136 question for each group. If we used an NHST approach, these 15 additional 137 comparisons would either seriously inflate the type I error rate (using a 138 p-value of 0.05 on each test would result in an overall error rate of 0.54), or 139 require much smaller nominal p-values for each test. 140 141  142 143 The only credible differences for single questions both occur for participants 144 in the "disease risk" group. The "healthy" ("Vaccinating healthy children helps 145 protect others by stopping the spread of disease.") and "diseases" ("Children 146 do not need vaccines for diseases that are not common anymore.") questions show 147 a reliable positive shift, which makes a lot of sense given the nature of the 148 intervention. However, it's important to note that the HDIs are very wide for 149 these posteriors compared to the ones shown earlier. This is driven primarily 150 by the fact that this comparison relies on a three-way interaction, which has 151 greater variance (as is typical in traditional ANOVA models). The posterior 152 mode of the change for the "plan_to" question ("I plan to vaccinate my 153 children") is fairly large for the "disease risk" group, but the wide HDI spans 154 the ROPE around 0. 155 156 ### Expanding the models 157 158 My goal was to examine the conclusions made in the original report of these 159 data. However, this is just one way to model the data, and different models are 160 more appropriate for different questions. For instance, the standard deviation 161 and thereshold values were fit separately for each question here, but these 162 could instead be based on a hyperparameter that could iteself be modelled. I 163 also excluded subject effects from the model; there were many subjects (over 164 300), so a full model with these included would take much longer to fit, but 165 may produce more generalizable results. Bayesian estimation requires an 166 investigator to be intentional about modelling decisions, which I consider to 167 be an advantage of the method. 168 169 ### Prior probabilities 170 171 A defining characteristic of Bayesian analyses is that prior information about 172 the model parameters is combined with their likelihood (derived from the data) 173 to produce posterior distributions. In this analysis, I used priors that put 174 weak constraints on the values of the parameters. If an investigator has reason 175 to assume that parameters will take on certain values (e.g., the results of a 176 previous study), this prior information can -- and should -- be incorporated 177 into the analysis. Again, I like that these decisions have to be made 178 deliberately. 179 180 ## Conclusions 181 182 Concerns about a possible link between childhood vaccination and autism is 183 causing some parents to skip childhood vaccinations, which is dangerous 184 ([Calandrillo, 2004](http://www.ncbi.nlm.nih.gov/pubmed/15568260)). However, an 185 intervention that exposes people to the consequences of the diseases that 186 vaccinations prevent makes them respond more favorably toward childhood 187 vaccination. A separate group of participants did not change their attitudes 188 after being shown information discrediting the vaccination-autism link, nor did 189 a group of control participants. 190 191 ### Acknowledgements 192 193 [Zach Horne](http://www.zacharyhorne.com/) made the data available for analysis 194 (by anyone!), and gave useful feedback on an earlier version of this write-up. 195 Much of the code for Bayesian estimation was cobbled together from programs 196 distributed with Doing Bayesian Data Analysis (2nd ed) by [John K. 197 Kruschke](http://www.indiana.edu/~kruschke/). 198