Scaling up: digital data services for the social sciences
Author: Nathan Cunningham, Darren Bell, Deidre Lungley, Herve l'Hours, Louise Corti, University of Essex, UK Data Service
Date: 22 November 2017
Type of case study: Research
About the data
The arrival of ‘big data’ has changed social scientists’ expectations, and brought technological and infrastructure challenges for data services and repositories. The UK Data Archive (UKDA), lead organisation of the UK Data Service (UKDS), now needs to ingest sizable treams of real-time data, and enable exploration and linkage of a variety of data assets. For us, this is nothing new. UKDA has been operating since 1967, when ‘data storage technology’ meant punch cards. We’ve adapted to magnetic tape, floppy disks and modern online databases – and users have moved from examining printed statistical tables to downloading survey data files, and to exploring and visualising over the web. In the era of big data, we need to completely review repository architecture and infrastructure – and by focusing first on the field of energy research, we’ve been able to plan for the challenges we face.