Let’s create a nation of social scientists

How might social science become more influential, more relevant and more useful in the years to come?

Recent debates about impact have largely assumed a model of social science in which a cadre of specialists, based in universities, analyse and interpret the world and then feed conclusions into an essentially passive society. But a very different view sees specialists in the academy working much more in partnership with a society that is itself skilled in social science, able to generate hypotheses, gather data, experiment and draw conclusions that might help to answer the big questions of our time, from the sources of inequality to social trust, identity to violence.

There are some powerful trends to suggest that this second view is gaining traction. The first of these is the extraordinary explosion of new ways to observe social phenomena. Every day each of us leaves behind a data trail of who we talk to, what we eat and where we go. It’s easier than ever to survey people, to spot patterns, to scrape the web or to pick up data from sensors. It’s easier than ever to gather perceptions and emotions as well as material facts and easier than ever for organisations to practice social science – whether investment organisations analysing market patterns, human resources departments using behavioural science, or local authorities using ethnography.

That deluge of data is a big enough shift on its own. However, it is also now being used to feed interpretive and predictive tools using artificial intelligence to predict who is most likely to go to hospital, to end up in prison, which relationships are most likely to end in divorce.

Governments are developing their own predictive tools, and have also become much more interested in systematic experimentation, with Finland and Canada in the lead,  moving us closer to Karl Popper’s vision of “methods of trial and error, of inventing hypotheses which can be practically tested…”

This revolution in data, experiment and prediction brings with it all sorts of challenges, such as how to get the right data, because many of the most important facts are not captured, or how to avoid algorithms reflecting the biases of past actions or how to handle issues of ethics and privacy. But it’s hard to see how any social scientist, or faculty, can avoid embracing at least some of the new tools for gathering data, interpreting them and then using them to make predictions.

The second revolution is less visible but could be no less profound. This is the hunger of many people to be creators of knowledge, not just users; to be part of a truly collective intelligence. At the moment this shift towards mass engagement in knowledge is most visible in neighbouring fields.  Digital humanities mobilise many volunteers to input data and interpret texts – for example making ancient Arabic texts machine-readable. Even more striking is the growth of citizen science – eBird had 1.5 million reports last January; some 1.5 million people in the US monitor river streams and lakes, and SETI@home has 5 million volunteers. Thousands of patients also take part in funding and shaping research on their own conditions.

There has so far been much less of this in social science, despite traditions such as Mass Observation. Yet there are obvious parallels and no shortage of fascination – perhaps to track what’s happening on the streets; the prevalence of hate crime or speech; the emergence of new kinds of economic life.

Technology now makes it much easier to embed social science in daily life and realise the dreams of past thinkers, from John Stuart Mill and John Dewey to Donald Campbell in the 1960s who advocated a truly experimental society, “a process utopia, not a utopian social structure per se… [that] seeks to implement Popper’s recommendation of a social technology for piecemeal social engineering…”

Within some professions, versions of this are gaining ground, blurring the boundaries between research and action. There is already a network of police officers using experimental methods – the Society of Evidence Based Policing. In some countries schoolteachers see their role as both teaching and research, using rough and ready devices with their peers to try out variations to curriculum or teaching methods. Thousands of charities now try to work out their “theory of change” and collect data to make sense of their impact.

Imagine these multiplied ten or hundredfold and we start to see an emancipatory vision of a society that is hungry to know itself and is helped by the academy to do so.

Add these two revolutions together – the one based on data and machine intelligence, the other on collective intelligence – and we can start to see a possible future for social science where the old idea of impact, and a linear flow from the academy out into society,  becomes rather anachronistic.

We’re all familiar with the old idea that it’s better to teach a man to fish than just to give him fish. In essence these trends ask us a simple question: why not apply the same logic to social science, and why not reorient social sciences to enhance the capacity of society itself to observe, analyse and interpret?

Geoff Mulgan is chief executive of Nesta, a charity dedicated to encouraging innovation and is also a senior visiting scholar at Harvard University. His latest book Big Mind is published by Princeton University Press later this year.