Social science research benefits from accountability and transparency, which can usefully be underpinned by high quality and trustworthy data. Rigorous data curation practices are still sometimes viewed as a dark art, and easy-to-use tools to correct and clean numeric data are not widely used, despite awareness of the desire to make data FAIR. The tasks of checking, cleaning and documenting data by repository staff can be all too manual and time-consuming.
The UK Data Service developed a free easy-to-use open source tool known as QAMyData that provides a health check for numeric data. The tool uses automated methods to detect and report on some of the most common problems in survey or numeric data, such as missingness, duplication, outliers and direct identifiers. Requirements were scoped through a series of engagement exercises with the Service’s own data curation team, other data publishers, managers and quantitative researchers to create a comprehensive list of ‘tests’ that are typically used when quality assessing numeric data files.
The tool offers a number of configurable tests that have been categorised into four types: file, metadata, data integrity, and identifiers, which can be run on popular file formats, including SPSS, Stata, SAS and CSV. A standard config file has default settings for each test, such as a threshold for pass or fail on various tests (e.g. detect value label that are truncated, email addresses identified as a string, or undefined missing values) which can be easily adapted to meet the user’s own desired thresholds. The configuration feature allows the creation of a unique Data Quality Profile. The software creates a ‘data health check’ that details errors and issues as both a summary and detailed report, providing a location of the failed test. New tests can easily be added. Data depositors and publishers can act on the results and resubmit the file until a clean bill of health is produced.
The choice of technology for the tool went through at least four months of research, experimenting with different open source programming languages and libraries of statistical functions, including R, Python and Clojure, focussed initially on SPSS and Stata files. The agile programming language, Rust, was selected as the best choice, building on the established Readstat library, which is gaining recognition in the statistical community, where we hope it will be developed further. The QAMyData software is easily downloaded to a laptop or server and can be quickly used and integrated into data cleaning and processing pipelines. It is available to download from the UK Data Service Github pages under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
The grant also delivered a training module on what makes a clean and well-documented numeric dataset. A user guide, training exercise and purposely-erroneous dataset were produced and road-tested during early training sessions.
Data curator and researcher workshops:
Principal Investigator: Louise Corti, UK Data Service
Co-Investigator: Vernon Gayle, AQMen, University of Edinburgh
Researchers: Jon Johnson, Myles Offord, Cristina Magder, Anca Vlad, UK Data Service
Funders: National Centre for Research Methods (NCRM), Economic and Social Research Council (ESRC)
Project dates: 8 January 2018 – 31 July 2019 (19 months)