Lately, there have been several occasions where people have compared the accuracy of my forecasts to FiveThirtyEight’s. As a big fan of Nate Silver and the rest of his team, I’m flattered that my work is even being talked about in the same conversation as theirs. I’m actually indebted to FiveThirtyEight anyways, because I use the Facebook data that they publish for free; and that variable is pretty much the cornerstone of my entire model. Granted, I’ve tried contacting the Facebook Data Science team on four different occasions to get the data personally, but a small fish like me can’t get a response (which is just fine, I know they have many other important things they are working on).
With all of that being said, I’d like to set some things straight:
- I deeply respect the work of Nate and FiveThirtyEight, and I think they do a fantastic job.
- It’s nice to be accurate, but if some other institution turns out to be more accurate than myself, it’s not as if I resent them for it. I’m happy for others when it turns out that they’ve done a good job through a solid analysis. That’s what this is all about.
- This is a hobby of mine, and I do this because I think it is a fun mental exercise.
It’s tough to estimate the outcome of any election with perfect accuracy. There are hundreds (or maybe even thousands) of variables that can be used in a model, and the goal is to choose the fewest number of variables that possess the most predictive value. I receive criticism all the time because I’m not taking into account a certain thing, and I totally understand that many of those suggestions have predictive value. For many, however, I’ve already tested them and decided not to include them, most likely because the effect wasn’t statistically significant and/or I’m already capturing that effect through another variable.
Now, to address those who have suggested that I’m outperforming FiveThirtyEight. Simply put, I’m not. FiveThirtyEight has been closer than I have on average. Here is a breakdown of why that is the case:
- FiveThirtyEight has, on average, been more accurate than me in the elections that we have both released projections for, which is 19 out of 32 states. I have published projections for 29 of the Democratic primaries/caucuses (I started after Nevada), and FiveThirtyEight has published projected results for 22.
- FiveThirtyEight’s overall average error for the contests that they publish projections for is 3.2%, and their median error is 2.6%. My average error is 5.8% and my median error is 5.3%. I have only been closer than FiveThirtyEight in seven contests, whereas they have been closer than me twelve times. See the following graph to visualize this.
- As for calling wins and losses correctly: For Missouri and Illinois, FiveThirtyEight projected Hillary to win, and I was projecting her to lose. FiveThirtyEight was correct both of those times. On the flip side, in Michigan and Oklahoma I projected Bernie to win, and FiveThirtyEight was projecting him to lose. Bernie won both of those. In Minnesota and Arizona, FiveThirtyEight didn’t publish any projections that I’m aware of.
- If you wish to recreate these numbers yourself, keep in mind that I publish projections for the two candidates excluding votes for other candidates. This means that my projections between Hillary and Bernie always add up to 100. FiveThirtyEight publishes projections based on polling, so that necessarily includes votes for other candidates. Thus, my projections must be measured against the results after adjusting them to the aforementioned scale (=100/(BernieVote + HillaryVote)*BernieVote), while FiveThirtyEight’s can be compared outright.
- I have underestimated Bernie Sanders overall. The sum of all of my errors (AdjBernieResult – MyBernieProjection) is 1.4. The sum of all of the absolute values of my errors is 174.1. See the graph below (excludes the first three states because I started publishing results after Nevada). The x-axis denotes the contest number, i.e. Iowa would be 1, New Hampshire 2, and so on.
So if I’m doing worse overall, why are my projections valuable?
I think my work is worth something. Maybe you don’t, and that’s perfectly fine. However, I know that I would want to get an idea of what would happen in some states when Nate hasn’t been able to publish any projections, and that has happened eleven times so far (mostly in caucus states). Let me be clear that it’s through no fault of FiveThirtyEight when they don’t publish projections, and they would likely publish projections for every state if there was always enough recent polling data, but that’s not always the case.
This is, in my opinion, the beauty of using the data sources I’m using. There is zero reliance on polls. Not that polls are a bad thing, because FiveThirtyEight demonstrates that enough polling data is an extremely powerful predictor all the time (See their Ohio, Vermont, Georgia, Virginia estimates all within one point of the result(!)). But, once again, polling isn’t always performed everywhere, and that’s where I think my work has the most value. Obviously my work isn’t particularly great all the time and I’ve had some major misses, but it’s still pretty cool to be in the infancy of this new methodology. I’m confident that this approach will become the new standard for political forecasting in the future and replace polling as the primary data source for predicting elections, and until then I will continue to refine my work to produce better results. Thank you everyone for the interest in what I do.
-Tyler
Tyler Pedigo 
Nice that you finally posted this. Confused why you chose not to link directly to Facebook Primary Map so everyone could see exactly what this data is you pull from.
LikeLike
You are selling yourself short. Another awesome thing about what you do is that it doesn’t require expensive polling. 538 basically takes some weighted averages. Big deal. The fact that your method comes close to a lot of expensive and time consuming polls is incredible! Also, echoing what was said above, your stats on predicting Bernie victories blow everyone else’s out of the water. That’s not a coincidence — it’s because your methods are more attuned to younger and independent voters than anyone else’s (antiquated) methods. The polling industry needs creative approaches — that actually work — like yours. In your shoes, I would start a consulting firm, at least. Keep it up!
LikeLike
Tyler I really respect your work, I am constantly checking your website for poll updates. Whats New York looking like?!!
LikeLike
Nice job Tyler. Also true that Bernie has won 15 states so far, you projected 13 of them, 538 projected only 3 of them: https://twitter.com/ericbroder/status/717869405749714944
So basically you predicted 87% of Bernie state victories, and 538 only 20%. Good work!
LikeLike
You aren’t giving yourself enough credit. You got Michigan right when they all got it wrong. 538 was way off in the last several caucuses with their percentages..you were very close. 538 had Sanders with about a 50/50 chance to win Wisconsin…that was way off..he won in a landslide..you had him with 59 percent of the vote and he got 57…superb! Keep going man..
LikeLike
I think in the end your methods will be far more valuable than 538
LikeLiked by 1 person
Is it not suspicious as hell that 538 published NO projections for Arizona? And why not Minnesota? I believe the public consumption polls had Sanders down 10% when the cognoscenti could see he was likely heading for an easy victory there.
LikeLike
Keep it up! Like Cecilia, I also appreciate & look forward to each of your posts! 🙂
LikeLike
As someone who’s very new to forecasting, projections and predictions, and just politics overall. I can say I was pretty sceptical of 538 for a long time. That changed after March 15th though, I started to realize they’re just trying to forecast as best as possible with the help of polling and some other variables. Now I certainly respect their work a lot more, and daily check their website to see how each candidate is doing.
And while it’s right that they’ve been more accurate, I still find your data much more valuable, for the following reasons:
– Contrary to 538, who mainly just use polling data to predict a vote share and chance to win a contest, you do it without the use of any pollsters. Instead you use Facebook and Google trends data, and a lot of your own data to project every single state. The beauty of this is that you’ve basically made an entirely new system apart from polling to predict states. And while you still have a 2.6% higher error overall, that’s an absolutely beautiful thing.
– Contrary to 538, you can publish and project every state, regardless of how much polls are available, or how accurate or trusted that poll is. That’s a gigantic bonus for all of your work.
– You can predict caucus states, which have been notoriously hard to predict even for respected pollsters (except for Iowa, there’s really some geniuses there.) And what’s even more amazing is how close you get. Your Washington State & Idaho predictions are both frighteningly close in that regard. And you still had very decent projections in Utah and Hawaii, while pollsters didn’t even dare to poll the states to lose their reputations. I do feel you could have predicted Alaska a little better, but the problem here is that it’s hard to factor into account making Hillary non-viable in specific places, for that you’d need to get data on the precinct or district level. All in all this has been the most amazing thing your work has done.
– Unlike 538, and I swear, this is important to me, and a lot of other people: You’re not arrogant. You’re humble, and just use data to try and predict stuff. You admit you can make errors, and that you’re only human, just like anyone else. Apart from Michigan, where 538 ate a stack of humble pie, they have been as arrogant as can be, and sometimes just flat out refuse to admit they make errors and they usually refuse to correct them. That’s where I still lack a lot of respect for them. Now don’t get me wrong, they’re still mostly right. But when they’re wrong, they try to cover it up. Like after Bernie won Nebraska and Kansas. “Because those states are so conservative those places probably have more liberals who vote for Bernie compared to other states.” What the hell…
I just want you to remember this, as a fellow maths and data enthusiast: “An error doesn’t become a mistake until you refuse to correct it.”
Keep being yourself, Tyler. And keep trying to make your data more accurate with every passing state as much as you can. Never lose yourself to thinking you know it all, because no man or woman does.
I greatly respect your work and will keep following your work for the rest of the political season, and I’ll probably look forward to what you’ll do after that too.
With much respect,
– Adriharu
LikeLike
“Unlike 538, and I swear, this is important to me, and a lot of other people: You’re not arrogant. You’re humble, and just use data to try and predict stuff.”
Yeah. What that person said.
LikeLike
You are awesome Tyler! I appreciate and look forward to your posts:)
LikeLiked by 1 person