Election Forecast Accountability Roundup

27

With 6D results in hand, the time is now to look through pre-election forecasts and say a thing or two about how each of us did. The Distortioland/CaracasChron model had a pretty darn good night on 6D, but intellectual honesty demands that we start with the…

The Worst. Forecast. EVER.

This one goes to…yours truly, for my cringeworthy February 16th post, “An Election Year Without an Election.” Yeah, ok, make fun. I went out on a limb, the damn thing broke. What can I say?

In my own (weak, weak) defense, I’ll say that the central idea behind the post – that if they did hold an election they’d lose it by such a catastrophic margin that their hold on power would be fatally compromised – is one that 6D bore out pretty well. I still think they blundered: they’d have had a better shot at keeping power if they’d found an excuse to scuttle the vote. (And yes I say that with my tail firmly tucked between my legs.)

The Lifetime Achievement Award for Services to Uselessly Broad (but then also flat wrong) Forecasts

This one goes to a couple of people, actually. People whose range of “scenarios” was so wide, there was essentially no way they could be proven wrong after the election – and yet still screwed it up. Into this pot goes Hercón, whose strategically brilliant insight that the opposition would get somewhere between 90 and 110 votes was about as useful as an ashtray on a motorcycle and Hinterlaces, whose 84-103 range for the opposition was an even worse mix of uselessly broad and flat wrong.

Special Mention for Not Even Trying

Once more, Luis Vicente Leon and Eugenio Martinez (a.k.a. @Puzkas) deserves a shout-out for his special brand of meta-hedging. Despite being recognized as one of the top elections experts in the country and sitting atop a mass of carefully curated data, he just flat out refused to hazard a guess in public.

Well, that’s one way to make sure you’re not wrong, I guess…

The “You didn’t even bill them for that, did you?” Diploma

In an election where even the government’s paid propaganda shills were going out of their way to avoid going on record predicting an outright PSUV win, it took a special form of shortsightedness to suggest such a thing without even the benefit of a paycheck. Sadly, that’s where Iñaki Sagarzazu found himself a mere few weeks before the vote, forecasting a bare-bones MUD simple majority but hedging aggressively on the downside. Iñaki is a serious researcher, and an independent one at that. But a serious re-think is in order.

And David Smilde could well be said to deserve one of these Diplomas, too – but he didn’t put an actual number to his Chicken-Little scenerio-writing, so he just about gets away with it.

The “How could something so wrong feel so right?” Medal

This one, con todo el pesar, goes to Distortioland/Caracas Chronicles. It pains me to write this, because the urge to do a victory lap is so strong here – too strong for some to resist, in fact.

Our forecast – 111 deputies – was at once freakishly on the mark and entirely useless. It got wrong the one thing that matters: whether the MUD would get a 2/3rd majority.

Granted, it got it wrong by 82 freakin’ votes in Aragua’s 3rd circuit. But I do feel Reverse-Price-is-Right rules have to apply here. The winner is whomever comes closest without going under. We were one under. It sucks. We lose.

You could say Distortioland is being unfairly penalized for having the balls to put out a point estimate rather than hedging with a range. That’s right, as far as it goes. But the case for that would be a lot stronger if @Econ_Vzla hadn’t wobbled right at the end when those two bad Datanalisis/Latinbarómetro polls came out and he suddenly went hyper-rangy...and not in a good way.

“Rightest, Longest” Ribbon

It’s one thing to figure out that the opposition would win a 2/3rds majority of Assembly seats. It’s quite another to have figured that out last April and never looked back.

Francisco Rodríguez was forecasting a 2/3rds MUD supermajority on behalf of Bank of America eight months ago. Noting, rightly, that the majoritarian bias in our “proportional-but-not-proportional” system would load on seats to whomever has a strong majority, FRod was projecting 119-120 oppo seats…in April.

¡Fenómeno!

Later, as our poll lead widened, se volvió loco and he started to talk about the opposition getting up to 148 seats. I always figured that the Distortioland/CaracasChron model captured something about the government’s built in mobilization advantage that an overly literal reliance on generic ballot questions couldn’t, so I was not surprised at all that the real lead was very substantial, but not entirely lopsided.

FRod has a unique claim on the forecasting Gold Medal for being the only forecaster who never once doubted that the opposition would reach 2/3rds, a far-out-of-the-consensus, widely-derided forecast  on a question that, strategically, was the only one that really mattered.

I agonized over this call, but the fact remains that BoA assigned at least 11 and as many as 36  too many seats to the MUD. That’s just too many.

Which is why:

…the Winner Is

Francisco Monaldi, for this tweet from December 2nd which, though still hedged, seems to be the only pre-6D forecast whose forecast range encompassed the right number.

Dr. Monaldi, you’ve won yourself a beer.

27 COMMENTS

  1. Thank you Quico! but like most of those who got the 2/3, I expected a larger vote margin. I think Luis Christianssen deserves credit for stating a couple of days before the election that even though the margin was going to be much smaller than most polls were showing (he said 11 pp), the number of seats will be likely above 100. He noticed that the opposition was growing much more in chavista circuits. Edgar Gutierrez also did a good job at predicting the result.

  2. Quico, Puzkas stated in a Prodavinci forum a few days before the election that he was looking at 98 seats. In the same forum Edgard Gutiérrez forecasted 103.

  3. Not that it will make you feel much better about the non-election call, especially since I’m neither Venezuelan nor a Vzla politics wonk…

    I was completely surprised that they went through with the election. I couldn’t imagine a group of people having their heads buried under so much sand that they couldn’t see, hear or feel what was coming.

    For me, it was just a question of pretext. Even when the elections were ongoing and the anecdotal evidence looked so very bad; I would have sworn they could have managed a “hack” or fraud claim to bail themselves out.

    The consequences were just so great in the event of a loss that it shocked me they actually held elections.

    • I’ve agonized over this, Jeff, cuz we spent lots of time and love and effort on that damn App. But in the end, saying you can’t fault the app for getting it one seat wrong reminds me of nothing so much as that scene from Dr. Strangelove:

      Turgidson: Ahem… The Duty Officer asked General Ripper to confirm the fact that he had issued the go code, and he said, uh, “Yes gentlemen, they are on their way in, and nobody can bring them back. For the sake of our country, and our way of life, I suggest you get the rest of SAC in after them. Otherwise, we will be totally destroyed by Red retaliation.” Uh… “My boys will give you the best kind of start, 1400 megatons worth, and you sure as hell won’t stop them now.” Uhuh. Uh… “So let’s get going, there’s no other choice. God willing, we will prevail, in peace and freedom from fear, and in true health, through the purity and essence of our natural… fluids. God bless you all.” And he hung up.

      [Pause as he realizes the implications of General Ripper’s words]

      Turgidson: Uh, we’re… still trying to figure out the meaning of that last phrase, sir.

      Muffley: There’s nothing to figure out, General Turgidson. This man is obviously a psychotic.

      Turgidson: We-he-ell, uh, I’d like to hold off judgment on a thing like that, sir, until all the facts are in.

      Muffley: General Turgidson! When you instituted the human reliability tests, you assured me there was no possibility of such a thing ever occurring!

      Turgidson: Well, I, uh, don’t think it’s quite fair to condemn a whole program because of a single slip-up, sir.

  4. With that wide interval, FM predicted at the same time simple majority (98), absolute majority (101) and qualified majority (112). That is, he was wrong in two of the three predicted outcomes. To advance the discussion on this issue, it would be appropriate that FM makes public the model which led to such forecast

  5. I was with you on the prediction that there would be no election. Like you, I was dead wrong. But, I won’t apologize for it. I think it is clear to most, in retrospect, that Chavismo would have been far better off, had they found (invented) an excuse to cancel the elections. I am sure they are kicking themselves for not doing it now. So, really the only thing we really got wrong was the degree of Chavismo’s incompetence.

    Now, THAT is something to think about… Traditional wisdom says that you should never underestimate your enemy. But, maybe overestimating your enemy’s capability is just as bad.

  6. Both Dr Monaldi and Distortioland deserve their beers , Of course predictions are part analytic acumen mixed with a dollop of luck and a dash of blind intuition. There are too many imponderables that can deviate a prediction that getting something exactly right is never a matter of pure analysis . One guy who got it right was Francisco in that there were so many voices saying the regime had the means to game the system in its favour through its control of the CNE and its electronic machinery that to have said it wouldn’t change the results was a demonstration of very clear and courageous intellectual conviction !! Congrats to all three!!

    • As someone who earns his living as a number cruncher (with medical data though, I’m not familiar with electoral modeling), reaching a point estimate so close to the real one using a SINGLE PREDICTOR is arrechísimo, I bow to Distortioland’s accuracy. He didn’t post his confidence intervals though, maybe they are quite wide and got the closeanswer just by an incredible luck!

      Dr Monaldi also deserve a price sure, but he didn’t spell how he got there, nothing wrong with that either.

      I say break the casks for both of them!

      Salud!

    • I very much agree with your analysis. But, if those guys deserve a beer, then Francisco surely deserves something even more special for that marvelous forecast. I still can’t believe it. Might I suggest that a bottle of “Baron Gaston Legrand Vintage Seviac Armagnac” be sent to his house forthwith? (instead of a beer)

  7. As Lee Kwan Yew would say, I’ve been wrong before, and I will certainly be wrong again.

    Regarding accountability, I refer anyone to Gustavo’s Coronel’s posts. You must have Justice, above all, or Chavista history will repeat itself. Impunity is a carte-blanche to further crimes.

    Accountability, in the eyes of the LAW, matters. That’s why we hardly have a country anymore. Thieves and crooks KNOW they can get away with anything.

  8. Quico, why are you so harsh on your homegrown initiative? Had Distortioland/CChron predicted 113 instead I would have given the model exactly the same credit. You know why? Because the 2/3 mark is an arbitrary mark foreign to the model (politically it is an important one, of course, but a statistical model that would have predicted 113 is in no way better than a model that predicted 111).
    Regarding the ribbon, I appreciate Francisco Rodriguez posture during all this time supporting the idea of a big majority for the MUD, to make the point of that possibility was certainly a good point, but lets be serious, given his accuracy (or lack of), would you listen to him that carefully the next time around? (lets say for the next presidential election?). You know, FRod prediction is probably what we would have heard from Consultores21 if (IF) they were still around 😉
    But what really surprised me was the big prize. Monaldi´s prediction? Really? Look at it again, his range covers three out of four possible outcomes. Give me 75% of the lottery tickets and I stand a pretty good chance of keeping the prize, but… if I were to buy 75% of the tickets probably I would have spent more than i would win with the prize, right? I think that sumarizes the situation here.

  9. Quico, your criteria have the consistency of warm butter. For me the “winner” is by far @Econ_Vzla. His latest post before the election was just an analysis to show the relevance of the rural vote, not a prediction. You should not hold that against him. It was a simple exercise that has anything to do with his actual prediction. Actually you should look at the accuracy of his individual circuital predictions, which were quite accurate. If this were not enough, his national vote split was right on the money!

    I’m not saying that F. Monaldi doesn’t deserve a special mention but his forecast is a bit rangy too. Plus, his mid-point prediction (106) as well as most of his range are also below the 2/3 threshold that you care so much about.

    The 2/3 majority, by the way, is an arbitrary threshold for assessing the accuracy of a prediction, unless of course that were the event to predict. I’m sure that for most pollsters and analysts, the events to predict were the number of seats for each party and/or the composition of the national vote. The prediction of whether or not the opposition was going to get whatever majority was a simple implication of their seats forecast.

    As for FRod prediction, his latest number disqualifies him. He doesn’t deserve any mention here….and he doesn’t need sponsors. He’s a big boy!

  10. I love this debate in that I can see a good case for all three. I especially like it cuz I can stand back at the distance provided by being on record with the worst forecast ever, so none of it splashes on me.

    For me, though, I’m interested in the political consequences of an outcome much more than the outcome. Saying a forecast of 113 would’ve been exactly as wrong as a forecast of 111 was is like saying the titanic would’ve been just as off course if it had sailed 40 ft. south of the iceberg instead of 40 ft. north, straight into it. That’s…just not how it works.

  11. I called 94 seats. Happy to be wrong. Hats off to Distortioland and his app.

    I do have to say that I didn’t expect the MUD to win by the 20-30 points the pollsters were forecasting. At least I was right in that. I expected the MUD to win with +/- 55% of the vote and they got 56% so there’s that. I thought chavismo wasn’t going to fall below 40%, but they got dangerously close to doing so. I didn’t expect the trashing chavismo took in places like Caracas, Vargas, Aragua. Those where pleasant surprises for sure. Polls where giving 3rd parties, or as they called them “Independents” double digit numbers. As expected, they didn’t even get 3% of the vote. I think Distortioland’s update where needed. The MUD wasn’t going to win by 30 points. It got half of that margin. Why would correcting his analysis to also show 10 and 20 point leads be a bad thing?

  12. Another interesting thing to ponder concerns the predictions per what WILL happen given MUD’s supermajority, Caprillies sounding off, and Maduro apparently digging in. When you contrast an executive branch of government holding court in a tomb and channeling the Great One’s mojo, while a Harvard educated freedom fighter rots in jail, to say nothing of a number two in command (Godgiven) wanted for purported drug dealing, one predicts at their own risk.

  13. I really think you are being unfair to the astonishing accuracy of the Distorsioland prediction. Of particular credit is the stellar job it did at predicting both the total opposition vote and then using it to forecast circuit-level outcomes, which is where you beat Monaldi’s forecast.

    Distorsioland’s final prediction only slightly under-predicted the two-party opposition vote share (56.1% versus 57.9%) whereas Monaldi predicted an opposition two-party vote share of at least 62%, so on this account Distorsioland wins. Distorsioland’s seat projection model also did a better job: Monaldi implicitly predicted 114 oppo seats on a 65% oppo vote share, underestimating the greater efficiency of the opposition vote in this election. His seat projection would have been off if you had used the actual outcome, whereas If you had plugged in the actual oppo vote share into Distorsioland’s seat projection model you would have gotten a 2/3 majority.

    This is not to criticise Monaldi, whose prediction was still very good. Rather, I think a victory lap for you is fully justified: the only mistake you made was not using a range. Even simply taking the margin of error on the final Datanalisis poll and using it to calculate upper and lower bounds would have given you a range of predictions that contained the final outcome.

  14. This is a study in how to come up with a storyline and force your data points into it no matter what. First, your lead into Iñaki’s piece is a giant non sequitir as he never said the government was going to win and should not be compared with those who did. His was not a forecast but a model that looked at how a ten point spread would play out. The point was to explain the logic of the districts and lists and provide some questions regarding what to think about. I don’t think it ever occurred to him or to me that anybody would be so infantile as to think he was actually predicting a ten point spread. The margin ended up being 16 points, so of course the opposition did better than in his 10 pt calculations. My piece you peg as a “forecast” was posted over six months before the election and was framed as a limiting case on how the opposition could lose, paired in the blog with Francisco Monaldi’s piece on how the opposition could win. It include this phrase “Of course it is too early to predict the outcome of an election that does not yet even have a date. But here we will look at some factors that could work against an opposition victory.” By factors we mean variables that could or could not come into play. Several, for example a continued increase in oil prices or renewed US sanctions, did not happen (and we never said they would, only that they could). Perhaps the most disturbing aspect of your post is the guiding assumption that type I error (false positives) are somehow heroic and macho, which type II error (false negatives) are somehow lacking in courage and guts. (Of course the highly masculinist trope undergirding this should be obvious). But this isn’t ESPN Gameday. In the real world it is always a judgment call whether to prefer type I or type II error. In a highly conflictive context in which what you say can have important consequences, it seems obvious to me that one should avoid false positives more than false negatives. In fact one of the many contributions of LVL to Venezuelan political debate is that he avoids false positives with the same numbers that other pollsters use to make bold predictions.

  15. Screw everyone, Quico included. Our elections forecasting app was badass, on point, and pretty to look at, which is way more than Luis Vicente can say for himself.

Leave a Reply