Saturday, 7 October 2017

The World Keeps Spinning and I cannot Stop It

I was expecting a little down time before needing to update here again, during which I have been working remotely (kinda) in the unicameral state. Yet it seems electoral news is unavoidable right now. Obviously the marriage postal survey is an ongoing story, and one I will dedicate at least one full post to (spoilers, I’m predicting a victory for the Yes campaign). But also local (Queensland) radio is expecting an election to be called any day now up here in the north-east, the NT’s LGA elections are not really on anyone else’s radar but are definitely a thing in this country, Xenophon is pulling out of federal politics to prepare for a lower-house run in my home state--so that race is already on foot--and overseas Catalan’s independence is being probably-not-decided by people risking their safety to illegally vote in Spain while the German election is being viewed as the possible start of the Fourth Reich. In other elections I didn’t get a chance to look into recently, Norway re-elected it’s conservative government, the Swiss passed one and rejected two referenda, Russia held gubernatorial elections and Singaporeans missed out on voting for their president due to a combination of peculiar constitutional provisions restricting this year’s candidates to those of Malay descent and only one eligible candidate standing. Upcoming this month, but also going to be largely ignored by me, are elections in Argentina, Austria, the Czech Republic, Iceland, Japan, Liberia, Kenya, Kyrgyzstan, Slovenia and Venezuela.

You get some democracy! And you get some democracy!
All of that, and I’m two elections behind in my wrap-up analysis. First, we have a very overdue reflection on the UK general election, and then the New Zealand general.

A very overdue reflection on the UK general election.

Out of 650 seats, I got 78 wrong. That’s an 88% success rate, which isn’t too bad even before you consider the large number of successful minor parties that are notoriously hard to predict. The full table will be in the data dump below, but there is little to be learned from the predictions we got right. The 78 wrong predictions are:

Constituency Prediction Result
Aberdeen South Scottish National Conservative
Arfon Labour Plaid Cymru
Ayr, Carrick & Cumnock Scottish National Conservative
Banff & Buchan Scottish National Conservative
Barrow & Furness Conservative Labour
Bath Conservative Lib Dem
Battersea Conservative Labour
Bedford Conservative Labour
Brentford & Isleworth Conservative Labour
Brighton Kemptown Conservative Labour
Bristol North West Conservative Labour
Bury North Conservative Labour
Caithness, Sutherland & Easter Ross Scottish National Lib Dem
Canterbury Conservative Labour
Cardiff North Conservative Labour
Carshalton & Wallington Conservative Lib Dem
Ceredigion Lib Dem Plaid Cymru
City of Chester Conservative Labour
Coatbridge, Chryston & Bellshill Scottish National Labour
Colne Valley Conservative Labour
Copeland Labour Conservative
Crewe & Nantwich Conservative Labour
Croydon Central Conservative Labour
Derby North Conservative Labour
Dewsbury Conservative Labour
Ealing Central & Acton Conservative Labour
East Dunbartonshire Scottish National Lib Dem
East Lothian Scottish National Labour
East Renfrewshire Labour Conservative
Eastbourne Conservative Lib Dem
Edinburgh West Scottish National Lib Dem
Enfield North Conservative Labour
Enfield Southgate Conservative Labour
Foyle Social Democrat Sinn Féin
Glasgow North East Scottish National Labour
Gordon Scottish National Conservative
Gower Conservative Labour
Halifax Conservative Labour
Hampstead & Kilburn Conservative Labour
High Peak Conservative Labour
Hove Conservative Labour
Ilford North Conservative Labour
Ipswich Conservative Labour
Keighley Conservative Labour
Kensington Conservative Labour
Kingston & Surbiton Conservative Labour
Kirkcaldy & Cowdenbeath Scottish National Labour
Lancaster & Fleetwood Conservative Labour
Leeds North West Lib Dem Labour
Lincoln Conservative Labour
Mansfield Labour Conservative
Middlesbrough South & East Cleveland Labour Conservative
Midlothian Scottish National Labour
Newcastle-under-Lyme Conservative Labour
North East Derbyshire Labour Conservative
Ochil & South Perthshire Scottish National Conservative
Oxford West & Abingdon Conservative Lib Dem
Perth & North Perthshire Conservative Scottish National
Peterborough Conservative Labour
Plymouth, Sutton & Devonport Conservative Labour
Portsmouth South Conservative Labour
Reading East Conservative Labour
Rutherglen & Hamilton West Scottish National Labour
Sheffield Hallam Lib Dem Labour
South Antrim Ulster Unionist
South Down Social Democrat Sinn Féin
Southport Lib Dem Conservative
Stockton South Conservative Labour
Stoke-on-Trent South Labour Conservative
Stroud Conservative Labour
Twickenham Conservative Lib Dem
Vale of Clwyd Conservative Labour
Walsall North Labour Conservative
Warrington South Conservative Labour
Warwick & Leamington Conservative Labour
Weaver Vale Conservative Labour
Wirral West Conservative Labour
Wolverhampton South West Conservative Labour

Of these, 7 are examples of the Conservatives pulling through against a predicted Labour win, 6 are Conservatives beating minor parties, 14 are unexpected minor party wins (half of those against other minor party predictions and half against Labour or Conservatives), 8 are from Labour doing better than expected against a minor party and the whopping 43 remaining are Labour polling much better than expected and slapping the Conservative party down. That accounts for more than 55% or our wrong calls in this election!

Much has been said of Jeremy Corbyn’s unexpectedly strong polling against Theresa May, in the face of a hostile media. Explanations range from identity politics to economics to evolutionary psychology. But personally, I’m not so much interested in why he did so well as why people didn’t see it coming.

With the election of Trump, the polls appeared to have been flawed by media vilification causing Republican voters to keep their intentions hidden. With Brexit, it has been claimed, there was an over-confidence in a ‘Bremain’ victory that reduced turn out for one side while fuelling protest votes for the other. Neither of these explanations appear to apply in the UK general election. What is of note is that younger generations turned out in large numbers and overwhelmingly supported Labour. Whether the Labour campaign spoke to them, they had resolved to become more active in politics after losing Brexit (as most younger voters appear to see it) or there was some other drive, the youth vote for Corbyn was an unexpected, progressive force.

With all of these inaccurate polls, I’m looking for a common denominator to adjust for; I cannot see one. What we appear to be facing is larger political engagement from groups that normally don’t turn out to vote (frustrated blue-collar workers for Trump, protest voters in Brexit and the youth in this election), stronger political engagement from all sides, and a failure of traditional survey methodology to adapt to a shifting political landscape.

And Now, to New Zealand

My analysis of the Kiwi election was brief and relied on basically estimating the nation-wide vote as a % per party. The actual result was National 44%, Labour 37%, NZ First 7%, Green 6% and others picking up the scraps. This is pretty well on par with our calculation of a mid-40s% National force against a comparable Labour-Greens alliance based on the polling. I, however, stepped away from this polling—based on the recent unreliabilities mentioned above—to suggest the Labour would poll even better than this. Of my three stated predictions:

1) a Labour-Green combined result outperforming the Nationals did not eventuate (58:51 seats in the Nationals’ favour), though I was right on the sub-prediction that Labour would gain at the expense of the Greens.

2) the Labour-Green combined result being sufficient to form government did not eventuate (10 seats short of a majority).

3) NZ First will refuse to cooperate with the Greens IF they end up holding the balance of power; unless Labour can form a >50% coalition with NZ First, they will see NZ First crown the Nationals to public outcry. This is still to be determined, but (with the possible exception of the public outcry) this looks pretty solid.

So, with a somewhat unusual question, I find myself asking ‘why did the traditional, well tested polling methods accurately predict the NZ polling result?’ It would seem the people turning out to vote were exactly those who were expected to. While I had looked at the poll inaccuracies as a left/right or willing-to-speak-out/unwilling-to divides the real focus—particularly applicable to the voluntary postal vote in Australia—should be on:
1) Who is being polled?
2) Who is not being polled?
3) Who should be being polled?

Data Dump


Friday, 22 September 2017

The Other Land Down Under

I must provide the normal excuses for not having yet looked at the results of the UK election and comparing these to my predictions. And, obviously, looking forward I will be spending a little time going ahead regarding the same-sex marriage poll in Australia. But right now, polls are open in New Zealand so I have a very small window to discuss all things Kiwi.

The New Zealand Voting System

One of the more curious of the electoral systems I am likely to cover for some time, the NZ parliament is elected using a mixed-member proportional representation system to elect its lower house. (Although NZ used to have an upper house, they pulled a Queensland in 1950 and abolished it.)

There are 71 electorates in New Zealand, which are elected by first-past-the-post systems, so whoever gets the most votes wins each of these seats. However, there are 120 seats in the parliament, sometimes more. This is because New Zealand voters also get to vote in a proportional representation system. This may be best illustrated by a 2-party example:

Party A wins 31 seats, and 40% of the proportional vote. Party B wins the remaining 40 seats and 60% of the proportional vote. Using a simple allocation of the seats (rather than the more complex Sainte Lague method they actually use that favours minor parties) Party A is entitled to 40%*120 = 48 seats, so adds 17 more members from its party list, and Party B is entitled to 60%*120 = 72 seats so adds 32.

This means that for predictive purposes the electorate votes are largely irrelevant; and miscalculation will generally be covered by the additional proportional vote. In other words, predict the proportional vote, and you predict the government.

The Popular Vote

In order to win any seats, a party must either gain 5% or the proportional vote or win a seat outright. Looking at the polls taken throughout the month of September, only four parties were expected to exceed 5% at any point: the ruling National Party, the opposition Labour Party, the Labour-alligned Greens Party and the free-floating NZ first:

Poll concluded: National Labour Green NZ First
20 Sep 45.8% 37.3% 7.1% 7.1%
19 Sep 46% 37% 8% 5%
13 Sep 40% 44% 7% 6%
11 Sep 47.3% 37.8% 4.9% 6.0%
10 Sep 40% 39.5% 9% 6%
6 Sep (Newsroom) 30% 45% 6% 11%
6 Sep (One News) 39% 43% 5% 9%
5 Sep 38.9% 41.1% 6.7% 8.9%

Importantly, given the left-leaning nature of the Green vote, the latest polls are very close for a Labour-Greens coalition:

Poll concluded: National Coalition
20 Sep 45.8% 44.4%
19 Sep 46% 45.0%
13 Sep 40% 51.0%
11 Sep 47.3% 42.7%
10 Sep 40% 48.5%
6 Sep (Newsroom) 30% 51.0%
6 Sep (One News) 39% 48.0%
5 Sep 38.9% 47.8%

Nevertheless, on the most recent data it would look like a narrow National lead, depending on the NZ First to play king/queen-maker. However, we've seen a lot of inaccurate, narrow polling in the last 18 months: Brexit, US Presidential and UK National. The French Presidential election--by contrast--was reasonably well predicted but far from close.

The first two of these surprise results favoured the right, the third favoured the left, so this is not necessarily a phenomena measured well on the traditional left-right spectrum. All three, however, supported the underdog and all three swung further than expected in favour of change. To say all three favoured the "unpopular" choice would be misleading, since in Brexit the "unpopular" vote won first-past-the-post (and in the US Trump also won, though not on the popular vote); it may be accurate to suggest, however, that these hidden votes supported the politically-incorrect/media-ridiculed position. Although polling is anonymous it would seem that people with these 'socially unacceptable' positions (for lack of a better word) are still hesitant to disclose these positions to polsters.

What does this mean in NZ? Support for the underdog would play into Labour's hands--especially considering that two months ago before a leader change their polling was about 20 percentage points lower. Support for change would also support Labour. However the rise of Jacinda Ardern, the new Labour Leader, has made her something of a media-darling. As a result there is less motivation for a 'hidden' vote.

Is the appetite for change fully represented in the polling above? There will be no way to tell until tonight. But it should not be as dramatic as the Brexit vote (probably in the area of around 10 percentage points).

My predictions, therefore, are:
1) a Labour-Green combined result outperforming the Nationals, but probably with Labour taking ground from Greens.
2) the Labour-Green combined result being sufficient to form government, but
3) NZ First will refuse to cooperate with the Greens IF they end up holding the balance of power; unless Labour can form a >50% coalition with NZ First, they will see NZ First crown the Nationals to public outcry.

Wednesday, 7 June 2017

Once More unto the Breach, Dear Friends

Having promised to deliver a (very) quick analysis of the UK general election, I guess I had better deliver.

There are 650 electorates to consider here—more than 4 times the size of an Australian federal election—with three countries and a principality to consider: England, Scotland, Northern Ireland and Wales. Fortunately for the speed-psephologists of the world, but unfortunately for the British, each seat is determined on a first-past-the-post basis. This means we can generally consider the two main contenders in each seat and ignore all of the others. This is what I will be doing for 568 of these seats. I have made 79 exceptions where the third-place party placed within 10% of the second party in the previous election (as these might need a three-way consideration of likely winners); an exception for Belfast South where the two main parties combined held less than 50% of the votes and thus cannot be considered entirely safe; an exception for North Down which is held by an independent (as swings to independents cannot be calculated without seat-specific polling); an exception for East Devon where the second party is an independent (for the same reason); and an exception for Buckingham as this is the speaker's seat (and as Jay Foreman, informative stand-up comic and one half of the Map Men, explains here this seat is uncontested by convention (except UKIP and the Greens sneakily contested it anyhow last election (and are doing so again this time))).

Taking these 568 seats, however, we can consider the two main contenders, the number of votes each held last election and then calculate their percentage of the two-party vote (2P%):

This boils down to 18 different contests. I've listed each below, the number of votes each party won in total in the previous election and the 2P% of the first party alphabetically. I have also taken the latest ICM/Guardian polling (more recent Opinium polling is now available) and worked out the two-party share held by the first party. The difference between the two percentages is the swing in percentage points.

Party 1
Party 2 Party 1 Party 2 Party 1% Party 1 Party 2 Party 1% Percentage Points
Conservative v Labour 11,299,959 9,344,328 54.74% 45 34 56.96% Conservative +2.23
Conservative v Lib Dem 11,299,959 2,415,888 82.39% 45 8 84.91% Conservative +2.52
Conservative v Scottish National 434,097 1,454,436 22.99% 27 40 40.30% Conservative +17.31
Conservative v UKIP 11,299,959 3,881,129 74.43% 45 5 90.00% Conservative +15.57
Conservative v Plaid Cymru 407,813 181,704 69.18% 35 8 81.40% Conservative +12.22
Green v Labour 1,157,613 9,344,328 11.02% 3 34 8.11% Green -2.91
Labour v Lib Dem 9,344,328 2,415,888 79.46% 34 8 80.95% Labour +1.50
Labour v Plaid Cymru 552,473 181,704 75.25% 46 8 85.19% Labour +9.93
Labour v Scottish National 707,147 1,454,436 32.71% 25 40 38.46% Labour +5.75
Labour v UKIP 9,344,328 3,881,129 70.65% 34 5 87.18% Labour +16.53
Lib Dem v Plaid Cymru 97,783 181,704 34.99% 5 8 38.46% Lib Dem +3.47
Lib Dem v Scottish National 219,675 1,454,436 13.12% 6 40 13.04% Lib Dem -0.08
Alliance v Unionist 61,556 184,260 25.04% 9.8 28.8 25.39% Alliance +0.35
Sinn Féin v Social Democrat 176,232 99,809 63.84% 27.9 13.7 67.07% Sinn Féin +3.22
Sinn Féin v Ulster 176,232 102,361 63.26% 27.9 15.8 63.84% Sinn Féin +0.59
Sinn Féin v Unionist 176,232 184,260 48.89% 27.9 28.8 49.21% Sinn Féin +0.32
Social Democrat v Unionist 99,809 184,260 35.14% 13.7 28.8 32.24% Social Democrat -2.90
Ulster v Unionist 102,361 184,260 35.71% 15.8 28.8 35.43% Ulster -0.29

Note that contests involving an exclusively Scottish party (in thistle purple), Northern Irish party (in shamrock green) or Welsh party  (in leek... beige?) do not have meaningful UK wide 2015 results or 2017  polling. These are compared with the votes and polling of the other parties only within their respective countries: sources for voting in Scotland, Northern Ireland and Wales. New polling for Scotland was available at the time of posting.

Applying this two-party swing, or modifier, to the 2P% in our first table yields a quick prediction: