There ain't no policy gold in them there data mountains

Governments need to rethink both the extent to which they invest in data gathering and to focus on higher level outcomes such as economic growth, better health or education and perceptions of crime

We make decisions all of the time and also, as our guilt about procrastination tells us, we avoid making decisions all of the time. Indeed, when we consider a decision we adopt a range of strategies designed to avoid the decision or any responsibility for the decision. We might simply pass the choice to somebody else: ‘do we go left or right here, Mary?’. Or we will seek to share responsibility for the decision: ‘I think we need to go through the main door, what do you think Fred?’. And we might simply refuse to make the decision: ‘I don’t know, Geoff, I’m going home now’.

While most of the decisions we make are simple (left or right, Netflix or Prime, chicken or beef), all are made easier when we have more information. We turn left because we have looked at a map, the documentary on the pyramids is on Prime not Netflix, we haven’t defrosted the chicken. Many choices we make are solved with this simple level of information but there are a set of choices, often more substantial and even irreversible, where we dawdle, dodge and put off the choice. Very commonly we say things like ‘I’m not sure’ and ‘I want to know more’ implying that the information we have isn’t sufficient to allow us to make the right choice.

So how much information do we need to make a good decision? Remember that most decisions are binary: you either do something or you don’t do that something. The problem is less the commitment to act than the nature of the choices available, the ‘choice architecture’. The manner in which we approach the selection from the suite of options that make up this choice architecture is usually an analytical one in so far as we appraise each option so as to allow us to reach the most optimal decision. And to conduct this analysis we make use of information and apply reason. So faced with a choice of action from a suite of available options, we need to be happy with the information we have about those options. The problem, however, is that we seldom have complete information meaning that instead of deciding we ask for more information, more research, better data. Procrastination becomes a strategy and we rationalise that choice with the argument that, by seeking better information or by reviewing the available information and option, we will arrive at a better decision.

I recall the process by which Bradford prepared its Local Plan, something that began in 2008 and, as I write, remains incomplete. Sixteen years into the process and the council has yet to identify, as the local plan process requires, allocations for housing and other development. The reason for this taking so long is, in part, down to the constant chopping and changing of national rules, but mostly it’s the result of procrastination. What would happen is that part of the plan, say the distribution of housing, would require a base of data before any plan is written. But because demography and economic geography are not static there is always the desire for comfort in the form of reviewing the base of information informing the planning process. Officers would explain that, if this didn’t happen (and it would take months, sometimes years) then the plan would fall foul of the planning inspector at his examination.

This picture is repeated at every level of government and across every function. The default response from any bureaucracy faced with the demand for ‘change’ is to call for more research, argue there’s insufficient data, or propose a review (often all three). For politicians making the choice to research and review is always presentable as a ‘decision’ even when it is, in truth, avoiding a decision. Resource is directed towards research, data collection and analysis that, too often, changes nothing in our understanding of the ‘choice architecture’. For some politicians and bureaucrats this process is critical because ‘the right choice must be made’. Of course, the result is that no choice is made while the room fills up with data and the analysis of data. Another set of bureaucrats see the gathering of data as an end in itself believing that by collecting data on everything, everywhere the process of decision-making will be made quicker and easier. Except as we noted with Bradford’s Local Plan, data doesn’t keep well, especially data about the real world in which ordinary people (occasionally escaping the trap of procrastination) make billions of decisions about their lives.

There are those who argue that the government should collect and publish more data (we’ve had a silly row in the UK about the government refusing to publish the racial origins and immigration status of people in social housing) often to suit their own political or academic preferences rather than from a genuine desire to improve the state’s decision-making. Who lives in a council house is less important, in public policy terms, than the fact that Britain is 4 million homes short of what it needs. Issuing data on how many heads of socially housed households are first generation immigrants doesn’t resolve this problem (although it does raise a question about why so many immigrants are in social housing) and, in some ways, it sets off a wild goose chase of blaming immigrants for high rents rather than the chronic lack of housing.

Even when the operation of a system provides large amounts of data, we should not assume that this data gives us a good enough basis for making choices. If we collect data on the people who ride buses it can prove useful for the management of the bus service but it tells us nothing about the people who don’t use the bus but might do so. Boston has an app called Street Bump that gathers data about the city’s roads using the sensor on smartphones. As cars drive around the amount the smartphones of the drivers and passengers tell the city authority about the state of the roads. Great idea except that the results were skewed towards the wealthier parts of the city where people had smartphones and downloaded the app. This was ‘remedied’ by changing the way data was analysed but is a reminder that data doesn’t always tell us what we think it is telling us.

This isn’t an argument against the gathering of data but rather an attempt to get people to appreciate that they can’t subcontract policy choices to computers, however clever and sophisticated their algorithms. And that the people, in a democracy, devolve these choices to politicians who are obliged to use the powers of reasoned argument to arrive at decisions. Moreover, I’ve tried to show that while making decisions is a simple thing, in operational terms most people find it very difficult because of the risks. As a result poor decisions are made either to put off making a decision or else, for reasons of political or cultural comfort, to ignore the choice that reason and information point to as the best option.

The world is flooded with information and with ever better tools for analysing that information but this doesn’t seem to result in better choices because expedience and decision-maker preference override what analysis points to as the best option. Choices are made either to give the appearance something is being done despite everyone knowing that something is ineffectual (e.g. advertising bans) or else to use rhetoric to pretend that the best choice is being made (e.g. planning reform). Those presenting policies also make selective use of information in order to rationalise their decision - or non-decision - preferring case study and qualitative data over quantitative data because the latter disagrees with their choice.

We probably need less information than we think to make good decisions. But today’s world is so data rich that we will always fall for the temptation of seeking more data or new data analysis as a prop for our decision-making. Not only does this result in data being misunderstood but it provides the excuse for delay and prevarication through the argument that new information is needed. Moreover decisions that should be dictated by political choice and operationalised using data end up being made on the basis of the data analysis rather than a reasoned choice.

As Duncan Robinson in The Economist observes:

“A naive idea that more data inevitably leads to better governance should also be abandoned. It is too easy for a government to get sidetracked by what it can see”

Robinson cites the example of England’s improved data on sewage discharges as one where British politics is now obsessed “with crap, with 1,700 mentions of ‘sewage’ in hansard over the past four years”. Nothing had changed except for us having data where we didn’t have data before but it is likely that this data on one part of our water and sewage system will skew policy around water supply and management, probably to the long-term detriment of the industry and the public.

Governments need to rethink both the extent to which they invest in data gathering (there is probably little good reason to collect data on ethnicity, religion and gender for example) and to focus on higher level outcomes (economic growth, health outcomes, educational attainment, perceptions of crime) rather than on trying to mine vast mountains of data for policy answers - there ain’t no policy gold in them there data mountains.