Accessible data curation and analytics for international-scale citizen science datasets

Benjamin Murray; Eric Kerfoot; Liyuan Chen; Jie Deng; Mark S Graham; Carole H Sudre; Erika Molteni; Liane S Canas; Michela Antonelli; Kerstin Klaser; Alessia Visconti; Alexander Hammers; Andrew T Chan; Paul W Franks; Richard Davies; Jonathan Wolf; Tim D Spector; Claire J Steves; Marc Modat; Sebastien Ourselin

doi:10.1038/s41597-021-01071-x

Accessible data curation and analytics for international-scale citizen science datasets

Benjamin Murray, Eric Kerfoot, Liyuan Chen, Jie Deng, Mark S Graham, Carole H Sudre, Erika Molteni, Liane S Canas, Michela Antonelli, Kerstin Klaser, Alessia Visconti, Alexander Hammers, Andrew T Chan, Paul W Franks, Richard Davies, Jonathan Wolf, Tim D Spector, Claire J Steves, Marc Modat, Sebastien Ourselin

Research output: Contribution to journal › Article › peer-review

10 Citations (Scopus)

Abstract

The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. As of May 23rd, 2021, over 5 million participants have collectively logged over 360 million self-assessment reports since its introduction in March 2020. The success of the Covid Symptom Study creates significant technical challenges around effective data curation.

The primary issue is scale. The size of the dataset means that it can no longer be readily processed using standard Python-based data analytics software such as Pandas on commodity hardware. Alternative technologies exist but carry a higher technical complexity and are less accessible to many researchers.

We present ExeTera, a Python-based open source software package designed to provide Pandas-like data analytics on datasets that approach terabyte scales. We present its design and capabilities, and show how it is a critical component of a data curation pipline that enables reproducible research across an international research group for the Covid Symptom Study.

Original language	English
Article number	297
Pages (from-to)	297
Journal	Scientific Data
Volume	8
Issue number	1
Early online date	22 Nov 2021
DOIs	https://doi.org/10.1038/s41597-021-01071-x
Publication status	Published - 22 Nov 2021

Keywords

Big Data
COVID-19/epidemiology
Citizen Science
Data Curation
Data Science
Datasets as Topic
Epidemiological Monitoring
Humans
Mobile Applications
Smartphone
Software

Access to Document

10.1038/s41597-021-01071-x

Cite this

Murray, B., Kerfoot, E., Chen, L., Deng, J., Graham, M. S., Sudre, C. H., Molteni, E., Canas, L. S., Antonelli, M., Klaser, K., Visconti, A., Hammers, A., Chan, A. T., Franks, P. W., Davies, R., Wolf, J., Spector, T. D., Steves, C. J., Modat, M., & Ourselin, S. (2021). Accessible data curation and analytics for international-scale citizen science datasets. Scientific Data, 8(1), 297. Article 297. https://doi.org/10.1038/s41597-021-01071-x

@article{191d8e1db995450e9456d0e1a75303f1,

title = "Accessible data curation and analytics for international-scale citizen science datasets",

abstract = "The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. As of May 23rd, 2021, over 5 million participants have collectively logged over 360 million self-assessment reports since its introduction in March 2020. The success of the Covid Symptom Study creates significant technical challenges around effective data curation.The primary issue is scale. The size of the dataset means that it can no longer be readily processed using standard Python-based data analytics software such as Pandas on commodity hardware. Alternative technologies exist but carry a higher technical complexity and are less accessible to many researchers.We present ExeTera, a Python-based open source software package designed to provide Pandas-like data analytics on datasets that approach terabyte scales. We present its design and capabilities, and show how it is a critical component of a data curation pipline that enables reproducible research across an international research group for the Covid Symptom Study.",

keywords = "Big Data, COVID-19/epidemiology, Citizen Science, Data Curation, Data Science, Datasets as Topic, Epidemiological Monitoring, Humans, Mobile Applications, Smartphone, Software",

author = "Benjamin Murray and Eric Kerfoot and Liyuan Chen and Jie Deng and Graham, {Mark S} and Sudre, {Carole H} and Erika Molteni and Canas, {Liane S} and Michela Antonelli and Kerstin Klaser and Alessia Visconti and Alexander Hammers and Chan, {Andrew T} and Franks, {Paul W} and Richard Davies and Jonathan Wolf and Spector, {Tim D} and Steves, {Claire J} and Marc Modat and Sebastien Ourselin",

note = "Funding Information: ZOE Limited provided in-kind support for all aspects of building, running, and supporting the ZOE app and service to all users worldwide. Support for this study was provided by the National Institute for Health Research (NIHR)-funded Biomedical Research Centre based at Guy{\textquoteright}s and St Thomas{\textquoteright} (GSTT) NHS Foundation Trust. This work was supported by the UK Research and Innovation London Medical Imaging & Artificial Intelligence Centre for Value-Based Healthcare (104691) and the Chronic Disease Research Foundation award CDRF-22/2020. Investigators also received support from the Wellcome Trust (WT203148/Z/16/Z, WT213038/Z/18/Z, and W212904/Z/18/Z), Medical Research Council (MRC; MR/V005030/1 and MR/M004422/1), British Heart Foundation, Alzheimer{\textquoteright}s Society, EU, NIHR, COVID-19 Driver Relief Fund, Innovate UK, the NIHR-funded BioResource, and the Clinical Research Facility and Biomedical Research Centre based at GSTT NHS Foundation Trust, in partnership with Kings College London. This work was also supported by the National Core Studies, an initiative funded by UK Research and Innovation, NIHR, and the Health and Safety Executive. The COVID-19 Longitudinal Health and Wellbeing National Core Study was funded by the MRC (MC_PC_20030). The work performed on the Swedish study was supported by grants from the Swedish Research Council, Swedish Heart-Lung Foundation and the Swedish Foundation for Strategic Research (LUDC-IRC 15-0067). SO was supported by the French government, through the 3IA C.te d{\textquoteright}Azur Investments in the Future project managed by the National Research Agency (ANR-19-P3IA-0002). ATC was supported by a Stuart and Suzanne Steele MGH Research Scholar Award and by the Massachusetts Consortium on Pathogen Readiness and M Schwartz and L Schwartz. Publisher Copyright: {\textcopyright} 2021, The Author(s).",

year = "2021",

month = nov,

day = "22",

doi = "10.1038/s41597-021-01071-x",

language = "English",

volume = "8",

pages = "297",

journal = "Scientific Data",

publisher = "Nature Publishing Group",

number = "1",

}

Murray, B , Kerfoot, E, Chen, L, Deng, J, Graham, MS , Sudre, CH , Molteni, E, Canas, LS, Antonelli, M , Klaser, K , Visconti, A , Hammers, A, Chan, AT, Franks, PW, Davies, R, Wolf, J, Spector, TD , Steves, CJ , Modat, M & Ourselin, S 2021, 'Accessible data curation and analytics for international-scale citizen science datasets', Scientific Data, vol. 8, no. 1, 297, pp. 297. https://doi.org/10.1038/s41597-021-01071-x

TY - JOUR

T1 - Accessible data curation and analytics for international-scale citizen science datasets

AU - Murray, Benjamin

AU - Kerfoot, Eric

AU - Chen, Liyuan

AU - Deng, Jie

AU - Graham, Mark S

AU - Sudre, Carole H

AU - Molteni, Erika

AU - Canas, Liane S

AU - Antonelli, Michela

AU - Klaser, Kerstin

AU - Visconti, Alessia

AU - Hammers, Alexander

AU - Chan, Andrew T

AU - Franks, Paul W

AU - Davies, Richard

AU - Wolf, Jonathan

AU - Spector, Tim D

AU - Steves, Claire J

AU - Modat, Marc

AU - Ourselin, Sebastien

N1 - Funding Information: ZOE Limited provided in-kind support for all aspects of building, running, and supporting the ZOE app and service to all users worldwide. Support for this study was provided by the National Institute for Health Research (NIHR)-funded Biomedical Research Centre based at Guy’s and St Thomas’ (GSTT) NHS Foundation Trust. This work was supported by the UK Research and Innovation London Medical Imaging & Artificial Intelligence Centre for Value-Based Healthcare (104691) and the Chronic Disease Research Foundation award CDRF-22/2020. Investigators also received support from the Wellcome Trust (WT203148/Z/16/Z, WT213038/Z/18/Z, and W212904/Z/18/Z), Medical Research Council (MRC; MR/V005030/1 and MR/M004422/1), British Heart Foundation, Alzheimer’s Society, EU, NIHR, COVID-19 Driver Relief Fund, Innovate UK, the NIHR-funded BioResource, and the Clinical Research Facility and Biomedical Research Centre based at GSTT NHS Foundation Trust, in partnership with Kings College London. This work was also supported by the National Core Studies, an initiative funded by UK Research and Innovation, NIHR, and the Health and Safety Executive. The COVID-19 Longitudinal Health and Wellbeing National Core Study was funded by the MRC (MC_PC_20030). The work performed on the Swedish study was supported by grants from the Swedish Research Council, Swedish Heart-Lung Foundation and the Swedish Foundation for Strategic Research (LUDC-IRC 15-0067). SO was supported by the French government, through the 3IA C.te d’Azur Investments in the Future project managed by the National Research Agency (ANR-19-P3IA-0002). ATC was supported by a Stuart and Suzanne Steele MGH Research Scholar Award and by the Massachusetts Consortium on Pathogen Readiness and M Schwartz and L Schwartz. Publisher Copyright: © 2021, The Author(s).

PY - 2021/11/22

Y1 - 2021/11/22

N2 - The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. As of May 23rd, 2021, over 5 million participants have collectively logged over 360 million self-assessment reports since its introduction in March 2020. The success of the Covid Symptom Study creates significant technical challenges around effective data curation.The primary issue is scale. The size of the dataset means that it can no longer be readily processed using standard Python-based data analytics software such as Pandas on commodity hardware. Alternative technologies exist but carry a higher technical complexity and are less accessible to many researchers.We present ExeTera, a Python-based open source software package designed to provide Pandas-like data analytics on datasets that approach terabyte scales. We present its design and capabilities, and show how it is a critical component of a data curation pipline that enables reproducible research across an international research group for the Covid Symptom Study.

AB - The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. As of May 23rd, 2021, over 5 million participants have collectively logged over 360 million self-assessment reports since its introduction in March 2020. The success of the Covid Symptom Study creates significant technical challenges around effective data curation.The primary issue is scale. The size of the dataset means that it can no longer be readily processed using standard Python-based data analytics software such as Pandas on commodity hardware. Alternative technologies exist but carry a higher technical complexity and are less accessible to many researchers.We present ExeTera, a Python-based open source software package designed to provide Pandas-like data analytics on datasets that approach terabyte scales. We present its design and capabilities, and show how it is a critical component of a data curation pipline that enables reproducible research across an international research group for the Covid Symptom Study.

KW - Big Data

KW - COVID-19/epidemiology

KW - Citizen Science

KW - Data Curation

KW - Data Science

KW - Datasets as Topic

KW - Epidemiological Monitoring

KW - Humans

KW - Mobile Applications

KW - Smartphone

KW - Software

UR - http://www.scopus.com/inward/record.url?scp=85119660232&partnerID=8YFLogxK

U2 - 10.1038/s41597-021-01071-x

DO - 10.1038/s41597-021-01071-x

M3 - Article

C2 - 34811392

AN - SCOPUS:85119660232

VL - 8

SP - 297

JO - Scientific Data

JF - Scientific Data

IS - 1

M1 - 297

ER -

Accessible data curation and analytics for international-scale citizen science datasets

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this