In this paper, we propose SEMCAT (Semantic Evaluation Metric Conforms to AMR Theory), a novel similarity measuring method for Abstract Meaning Representation (AMR). AMR is a semantic structure used to explicitly express the truth-conditional meaning aspect of a natural language sentence. Our evaluation strategy is mainly designed to reflect the theoretical basis of AMR. Specifically, based on the concept of semantic similarity derived from neo-Davidsonian semantics, we enhance the $$\textsc {Smatch}^\sharp $$ metric with the Weisfeiler-Leman algorithm to reflect both the local and global structure of an AMR graph. In this work, we conducted two experiments to verify the usefulness of the SEMCAT metric. In Experiment 1, we conducted a series of statistical tests to confirm whether SEMCAT conforms to AMR guidelines using our simulated data, which were based on AMR 3.0. The experimental result showed that SEMCAT exhibited clear conformance to AMR guidelines compared to existing metrics. Also, we re-evaluate current state-of-the-art AMR parsers using SEMCAT to report new baselines. In Experiment 2, we examine the usefulness of SEMCAT by evaluating the correlation between AMR metrics and human evaluation scores for the paraphrase task, which is one of the main downstream tasks for AMR. In four benchmark data sets for the evaluation, SEMCAT achieved the highest average correlation among all existing AMR metrics with graph standardization.

Abstract This study examines whether the psycholinguistic and demographic characteristics of authors of online texts are correlated with the way harmful language, such as toxicity and hate speech, is judged. We apply artificial intelligence models to two harmful language datasets, Jigsaw’s Special Rater Pool dataset and the Measuring Hate Speech dataset, to generate probabilities for different text aspects, namely inferring demographic information of the author behind the suspicious text in terms of age and gender, as well as the expressed emotions, emotionality, sentiment and communication style. We then perform a statistical regression analysis to examine how these text aspects correlate with the perception of hate speech and toxicity during the annotation process. The study shows that while the frequency of the psycholinguistic text aspects that can be derived from the author’s personality does not differ significantly between harmful and non-harmful classes, the inferred text aspects are statistically associated with the annotators’ perception of harmful language and could potentially influence the way annotators label the texts.