[SEM] Perspectivism in NLP

ABSTRACT:

Contemporary Natural Language Processing is largely rooted in language resources, e.g., for training models in supervised machine learning.
Even is few- or zero-shot settings, the importance of good quality data for benchmarking is paramount. Typically, data come in the form of large manually annotated datasets. The harmonization of the annotation, however, is often problematic, especially when highly subjective annotation tasks are performed, such as sentiment, irony, undesirable language, and generally anything involving pragmatics. Discarding and averaging discordant opinions carries the risk of losing the rich knowledge coming from different annotators' perspectives. A recent line of research proposes to never aggregate annotations [1], but rather to leverage the worth of knowledge found in disagreement for building models [2] and evaluating them [3]. In this talk, I will present the perspectivist paradigm, focusing on NLP, and a few works exploring its implications on data annotation, model evaluation, and interoperability, with particular focus on the perspectivist modeling of hate speech and irony.

[1] The Perspectivist Manifesto. https://pdai.info/

[2] Basile et al. 2021 "We Need to Consider Disagreement in Evaluation" https://aclanthology.org/2021.bppf-1.3/

[3] Cabitza et al. "Toward a perspectivist turn in ground truthing for predictive computing". AAAI-23. https://arxiv.org/pdf/2109.04270.pdf