Fair & Responsible AI Workshop @ CHI2020

Bias as a Distinct Factor in Human Ratings of Machine Labeling


Workshop Paper


Eric P.S Baumer, Amin Hosseiny Marani

Abstract
This paper uses an excerpt from a larger analysis to argue that human assessments of machine labeling can reveal bias as a distinct measure separate from other perceptions of label quality. Human subjects were asked to assess the quality of automatically generated labels for a trained topic model. Quality assessments were gathered using 15 distinct self-report questions. Exploratory factor analysis identified a distinct “bias” factor. This point is likely relevant for a wide variety of machine labeling tasks.

PDF

Cite

APA
Baumer, E. P. S., & Marani, A. H. Bias as a Distinct Factor in Human Ratings of Machine Labeling.

Chicago/Turabian
Baumer, Eric P.S, and Amin Hosseiny Marani. “Bias as a Distinct Factor in Human Ratings of Machine Labeling” (n.d.).

MLA
Baumer, Eric P. S., and Amin Hosseiny Marani. Bias as a Distinct Factor in Human Ratings of Machine Labeling.