The announcement that 2013 will now have a draw to allocate songs into halves of the final did sort of ruin the impassioned, statistically-backed blog post I’d planned about the running order. I was all set to dissect the data and examine whether songs in the second half scored significantly higher than the first. The EBU then went and acted on my findings before I’d even done anything with them. So, instead of a post angrily exclaiming the need for some random element to the running order, I’m offering insight into the effects of the random element they’ve put back in.
You don’t need to do too much maths to see the running order has an effect. A quick look at the graph on the right makes it clear as day. There is an obvious upward trend in points received as you get further into the contest. These figures are the average points per country for each draw position since televoting became widespread in 1998. The red line shows the overall average of 2.44. Bear in mind here that the number of finalists – and so the position of the last entry – has changed over time. So the right side of the graph doesn’t give a fully rigorous picture. It’s just a rough guide.
So how much of an effect does the draw have? Simplifying my problem down to halves, I found that songs in the first half of the final scored a measly average of 2.10 points per country, while songs in the second half got 2.83. In a contest of 39 countries, like 2013, that gap amounts to 28 points.
“Fluke!” I hear some cry. “That’s just how the songs happen to fall. Loreen just happened to get spot number 17 last year and she was always going to win. Sometimes the good songs just come later”. Maybe it is a fluke. Luckily, statistics offers us a way of measuring the likelihood of such a fluke.
Pasha looks happy.
Significance testing seeks to find a probability that an observed result is a true reflection of the actual position, as opposed to it occurring through random fluctuation. 95% is the key figure. If your result lies within a 95% confidence interval, mathematicians will consider it to be statistically significant. If you want to be really strict, you look for 99%.
Putting the above averages through a t-test to see whether the 1st and 2nd half averages are significantly different to the overall average gives us p-values of 99.57% and 98.98%. Both fall well within the standard measure of significance. It’s very unlikely that the relationship between draw and points scored is coincidental. The half of the draw a finalist picks out of the hat in May will have a real impact on their result.
Does this mean that, by instituting a draw for halves, the EBU get off scot-free? Not really. There’s still room for the running order to affect the outcome within the halves. I split the draw again into quarters and ran my t-test again. The results, in the table below, show again that the later the position in the running order, the higher the points tally. However, only the songs in the first and last quarters show a statistically significant difference from the overall average.
I then ran a two-sample t-test to find if there was a significant difference between quarters. There is. When Christer Bjorkman and pals sit down and devise the running order in May, their decisions will still influence the results. It won’t be as big an impact as it could have been, but it’s still unfair on the entries.
When the removal of the draw was first announced, the great and the powerful from EBU were in complete denial about the running order having statistical significance. That position seems to have been adjusted. Maybe there was a great outcry at the heads of delegation meeting, or maybe someone just did the maths.
By having a draw for halves, EBU may well avoid an outcry over producers rigging the order. At the very least it will give them a handy excuse when complaints come in. They still don’t really have much of a leg to stand on, though. When someone says “It’s fine, we had a draw for halves, so everything’s fair,” point them my way.
Thank You Europe 