Sunday, November 06, 2022

Methodologies For Predicting The Future Are A Different Thing

Claire McCaskille and Pat McCrory agreed on one thing today on MTP: They had been in races where the polls said they were going to win, and races where the polls said they were going to lose.

The binding thread? The polls were wrong, each time.
There's always an "if" and an excuse. Sort of like the weatherman who predicts a torrential downpour, and there’s not even a drop. Eh, weather forecasts are not a hard science; more a matter of probabilities. But the argument in that tweet is that the polls are right; it’s our expectations that are wrong. Silly us! We should know better than to argue with the polls!

We treat polls as if they were windows opening on the future. The effect of this can be real. The year Katrina struck New Orleans and those left homeless were brought to Houston (among other places), Rita was predicted to slam straight into Houston. That prediction was actually based on probabilities, which are frankly more grounded in meteorology than in polling. It doesn’t take any interpretation to know whether or not the hurricane hit. 

In the case of Rita, it didn’t hit Houston. But the prediction it would created bumper to bumper traffic jam from Houston to Dallas, a distance of about 240 miles. Rita went to Beaumont, and Houston didn’t get so much as wet.

We can’t look into the future. Meteorologists in Houston learned not to pretend they could. Pollsters and pundits, on the other hand…
  How many people didn’t vote because, “Fuck, it’s Hillary, but look! Trump can’t win!”?  And still today:
But it’s science! And numbers! How could it be wrong? Chuck Todd sagely intoned this morning that something something had never happened in this century (IOW, the polls he relied on are right and a window into the post-Tuesday future). No one pointed out that 22 years is a vanishingly small data point in the 233 years we have been a Constitutional republic.

Innumeracy is real. As is the struggle.

Polling is based less on numbers than on models. That’s how 1000 people (or less; the point of that thread) can speak for 330 million. But when the models fail? What then? Note in that thread the sentence: “ 2022 is a midterm.” So, “We have a model for that.” But is it reliable?

The upshot of the thread is: “Well…maybe!” Non-response, after all, is a thing. Telephone polling is based on the old practice of answering a ringing phone. My parents did that ’til the day they died. Until I pulled it, my landline announced my calls and if I didn’t know the caller, I didn’t answer. I grew up reflexively answering a ringing phone. I learned not to. Maybe I missed calls from pollsters. I didn’t miss them at all. Do the models account for this? How does anyone know? We know when the forecast rain fails to fall. How do we know the models are sound?

Weather forecasting can be presented as a matter of probabilities based on knowledge of meteorology and observable data. But polling starts with presumptions and then finds data to support. Well, that’s not quite fair. The data is used to justify the model. And then the prime users of that data (political journalists and pundits) conveniently fail to focus on the inaccuracies of the polls as predictors of election outcomes (because, like Chuck Todd, they have their own models). They also want to use numbers to sound like their opinions (it’s all they are) are not mere conjecture but are science! 

Although really it’s more sciencey than scientific. Science, after all, works with probabilities, at best. It doesn’t predict the future, but can establish likely outcomes. Flip a coin 100 times, and probability can calculate the likelihood of the number of times it comes up heads or comes up tails. But it can’t predict what the next flip will be. The basis of chaos theory is that there are too many unknowable variables (the butterfly’s wing) in a causal chain to ever accurately calculate an outcome of the chain. But pollsters regularly confidently predict the outcome of elections with unverified data which is used to justify models which may not reflect current reality; or reality at all.

What is the matter with Kansas ?

Like the proverbial generals, pollsters are always modeling the last situation. Or rather, what they think the last situation was. 

They justify this with the closeness of their predictions and the outcome, although even that is an interpretation. They take that success as justification of their methodology, and carry forward to the next election. Where they justify their methods again, whatever the result. And they count on the public not really paying attention; or forgetting that they did (and how do you prove either explanation is true?), and relying on the pundits and journalists not to give up the game. And why should they? They have too much invested in it. 

Numbers =science =“hard facts,” and who doesn’t want that authority?

But where does that leave the consumers of this information? Misinformed, mostly.
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)