Assessing the Reliability of Crowdsourced Speech Data for Objective Quality Prediction
* Presenting author
Abstract:
The increasing demand for large and diverse speech datasets required for modern deep neural network-based speech quality prediction models has made crowdsourcing an attractive alternative to traditional data collection, that typically relies on studio recordings judged by listeners in laboratory tests. While offering a cost-effective and scalable source of diverse speech data, the reliability and validity of crowdsourcing in this domain remain uncertain due to uncontrolled recording conditions and equipment, as well as inconsistent participant behaviour. In this paper, we investigate the reliability of crowdworkers by collecting speech data under simulated crowdsourcing conditions featuring diverse acoustic environments and recording devices, and comparing the resulting recordings with previously collected crowdsourced data. We present the recording methodology and procedure, along with the instrumental analyses of speech data. Furthermore, objective speech quality prediction models were applied to both datasets to assess the influence of recording method on model performance. The findings provide insights into the reliability of crowdsourced speech data and its implications for speech quality prediction.