How to benchmark objective quality metrics from paired comparison data? - Nantes Université Access content directly
Conference Papers Year : 2016

How to benchmark objective quality metrics from paired comparison data?

Abstract

The procedures commonly used to evaluate the performance of objective quality metrics rely on ground truth mean opinion scores and associated confidence intervals, which are usually obtained via direct scaling methods. However, indirect scaling methods, such as the paired comparison method, can also be used to collect ground truth preference scores. Indirect scaling methods have a higher discriminatory power and are gaining popularity, for example in crowdsourcing evaluations. In this paper, we present how the classification errors, an existing analysis tool, can also be used with subjective preference scores. Additionally, we propose a new analysis tool based on the receiver operating characteristic analysis. This tool can be used to further assess the performance of objective metrics based on ground truth preference scores. We provide a MATLAB script with an implementation of the proposed tools and we show one example of application of the proposed tools.
Fichier principal
Vignette du fichier
QoMEX2016.pdf (575.17 Ko) Télécharger le fichier
Origin Publisher files allowed on an open archive
Loading...

Dates and versions

hal-01395449 , version 1 (07-12-2019)

Identifiers

Cite

Philippe Hanhart, Lukáš Krasula, Patrick Le Callet, Touradj Ebrahimi. How to benchmark objective quality metrics from paired comparison data?. 8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbon, Portugal. ⟨10.1109/QoMEX.2016.7498960⟩. ⟨hal-01395449⟩
95 View
341 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More