A few more thoughts regarding weighting on prior vote, and the possible effects it is having on polling estimates in 2024. In figures 1 and 2, we show the comparison between 538 and CHIP50. In figure 1, the comparison is to CHIP50 with demographic weights, but no adjustment for prior vote. In figure 2, we add (simple) weights adjusting for prior vote.
A few key takeaways: (1) generally, we again align quite well with 538, as we did in 2020. But, we clearly overshoot 538 (on average) in figure 1, by about 3 points, with a decent spread (8 points outside of the +/- 5 points). In the equivalent to figure 1 for 2020, we were better aligned, although with slightly higher dispersion. (2) weighting on prior vote greatly increases alignment (average deviation drops to 1.4%) and reduces dispersion (no points outside of the +/- 5 point range).
That second point is really pretty remarkable when we consider in principle how much noise there is in polling (variation due to the randomness in samples; probs vs nonprobs; different methods for turnout, weighting, etc). So many sources of potential variation, and yet so much convergence. One can also see this in the tracking website, Derivative Polling, which shows a high degree of similarity across different polls (generally, almost all polls have been +/- 2 points of the means of all of the polls). Our guess is that this degree of convergence is in significant part driven by weighting by prior vote. It is inserting this source of shared information that is so powerful that it dominates sources of variability, like other methods applied the data, and it irons out weird samples, and so on.
What this also means is that when these polls are aggregated, that what is being aggregated are the modeling assumptions built into the polls. And, as those assumptions become more important in driving the inferences of these polls, and if those modeling assumptions are shared across polls, there is a danger that there is less and less value added in the aggregation process. Survey respondents may be (mostly) independent observations from each other; but pollsters making modeling decisions certainly are not.
