How Lexicographers Evaluate User Contributions in The Thesaurus of Modern Slovene in Comparison to Dictionary Users

Abstract

User involvement can be a valuable asset in expediting the process of language resource development, given that a thoughtful methodology is implemented. A successful example is the Thesaurus of Modern Slovene, which incorporates user participation to improve its automatically generated content. To shed light on the otherwise invisible lexicographic decision-making processes and to develop editorial protocols based on the needs of dictionary users, we investigated how differently lexicographers evaluate user-suggested synonyms compared to other user groups. We conducted an evaluation of nearly 1,000 user-suggested synonyms, assessed by a total of 42 evaluators from 7 user groups, and tested four hypotheses about lexicographers as evaluators. After evaluation, the Inter-Annotator Agreement (IAA) in all groups was calculated using Krippendorff's alpha and entropy, the evaluators’ comments were classified into bottom-up categories, and the data were statistically analysed. In accordance with our assumptions, the lexicographers provided the most detailed arguments and identified the highest number of potential shortcomings of the suggested synonyms. However, they also scored the second lowest IAA among all groups and were more opposed to discarding user suggestions. We discuss the possible reasons for these results and emphasise their value for the further development of responsive dictionaries.