@inbook{07a2ca601f7f4d26b72a8f347ad6fe44,
title = "Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratings",
abstract = "Human ratings are abstract representations of segmentation quality. To approximate human quality ratings on scarce expert data, we train surrogate quality estimation models. We evaluate on a complex multi-class segmentation problem, specifically glioma segmentation, following the BraTS annotation protocol. The training data features quality ratings from 15 expert neuroradiologists on a scale ranging from 1 to 6 stars for various computer-generated and manual 3D annotations. Even though the networks operate on 2D images and with scarce training data, we can approximate segmentation quality within a margin of error comparable to human intra-rater reliability. Segmentation quality prediction has broad applications. While an understanding of segmentation quality is imperative for successful clinical translation of automatic segmentation quality algorithms, it can play an essential role in training new segmentation models. Due to the split-second inference times, it can be directly applied within a loss function or as a fully-automatic dataset curation mechanism in a federated learning setting.",
keywords = "automatic quality control, BraTS, glioma, quality estimation, segmentation quality metrics",
author = "Florian Kofler and Ivan Ezhov and Lucas Fidon and Izabela Horvath and {de la Rosa}, Ezequiel and John LaMaster and Hongwei Li and Tom Finck and Suprosanna Shit and Johannes Paetzold and Spyridon Bakas and Marie Piraud and Jan Kirschke and Tom Vercauteren and Claus Zimmer and Benedikt Wiestler and Bjoern Menze",
note = "Funding Information: Acknowledgments. This work was partially supported by the Ministry of Science and Technology (MoST), the National Center for Theoretical Sciences, and Big Data Computing Center of Southeast University. W.-W. Lin and T.M. Huang were partially supported by MoST 110-2115-M-A49-004-and MoST 110-2115-M-003-012-MY3, respectively. T. Li was supported in part by the National Natural Science Foundation of China (NSFC) 11971105. Funding Information: Acknowledgement. BM, BW and FK are supported through the SFB 824, subproject B12. Supported by Deutsche Forschungsgemeinschaft (DFG) through TUM International Graduate School of Science and Engineering (IGSSE), GSC 81. LF, SS, EDLR and IE are supported by the Translational Brain Imaging Training Network (TRABIT) under the European Union{\textquoteright}s {\textquoteleft}Horizon 2020{\textquoteright} research & innovation program (Grant agreement ID: 765148). IE and SS are funded by DComEX (Grant agreement ID: 956201). Supported by Anna Valentina Lioba Eleonora Claire Javid Mamasani. With the support of the Technical University of Munich - Institute for Advanced Study, funded by the German Excellence Initiative. EDLR is employed by icometrix (Leuven, Belgium). JP and SS are supported by the Graduate School of Bioengineering, Technical University of Munich. JK has received Grants from the ERC, DFG, BMBF and is Co-Founder of Bone-screen GmbH. BM acknowledges support by the Helmut Horten Foundation. Research reported in this publication was partly supported by the National Institutes of Health (NIH) under award numbers NIH/NCI:U01CA242871 and NIH/NINDS:R01NS042645. Research reported in this publication was partly supported by AIME GPU cloud services. Funding Information: Acknowledgements. This research was supported by the Capacity Enhancement Program for Scientific and Cultural Exhibition Services through the National Research Foundation of Korea (NRF) funded by Ministry of Science and ICT (No. NRF-2018X1A3A1069693) and Korea Institute of Energy Technology Evaluation and Planning (KETEP) grant funded by the Korea government (MOTIE) (20201510300280, Development of a remote dismantling training system with force-torque responding virtual nuclear power plant). Funding Information: Acknowledgments. Research reported in this publication was partly supported by the National Institutes of Health (NIH) under award numbers NIH/NCI:U01CA242871 and NIH/NINDS:R01NS042645. The content of this publication is solely the responsibility of the authors and does not represent the official views of the NIH. Funding Information: Acknowledgments. JN was supported by the Silesian University of Technology funds through the grant for maintaining and developing research potential. This work was supported by the Polish National Centre for Research and Development grant: POIR.01.01.01-00-0092/20 (Methods and algorithms for automatic coronary artery calcium scoring on cardiac computed tomography scans). Funding Information: Acknowledgements. This work was partially funded through NIH/NIBIB grant under award number R01EB020683. The authors would like to acknowledge partial support of this work by the National Science Foundation Grant No. 1828593. Funding Information: study was conducted retrospectively using human subject data made available in open access by BraTS-Reg Challenge organizers [2]. Ethical approval was not required as confirmed by the license attached with the open access data. The authors declare no conflict of interest. The authors would like to thank the organizers for their help with the submission of the Singularity container. This research was supported in part by PLGrid Infrastructure. Funding Information: Acknowledgements. This work was supported by an NSERC Discovery Grant to JL. Funding was also provided by a Nova Scotia Graduate Scholarship and a StFX Graduate Scholarship to JW. Computational resources were provided by Compute Canada. Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Proceedings of the 8th International MICCAI Brainlesion Workshop, BrainLes 2022 ; Conference date: 18-09-2022 Through 22-09-2022",
year = "2023",
month = jul,
day = "18",
doi = "10.1007/978-3-031-33842-7_1",
language = "English",
isbn = "9783031338410",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "3--13",
editor = "Spyridon Bakas and Ujjwal Baid and Bhakti Baheti and Alessandro Crimi and Sylwia Malec and Monika Pytlarz and Maximilian Zenk and Reuben Dorent",
booktitle = "Brainlesion",
address = "Germany",
}