Judgment Aggregation: A Primer

Computational social choice is a new interdisciplinary research area at the intersection of social choice theory and computer science that studies computational aspects of social choice mechanisms as well as applications of social choice ideas in the design and analysis of computing systems. Research on computational social choice now encompasses a variety of topics, including the algorithmic aspects of voting rules, computational barriers to strategic behaviour, collective decision-making in multi-agent systems, preference elicitation and communication issues in voting, fair division, computational aspects of weighted voting games, collective decision-making in combinatorial domains, logic-based formalisms for social choice problems, belief and judgement aggregation, and social software. The aim of this special issue is to build on the spectacular success of two workshops on computational social choice (COMSOC’06, Amsterdam and COMSOC’08, Liverpool), and provide a forum for reseachers in the area to present their ideas on a variety of topics. This special issue is composed of nine research articles and a short note that span a fairly wide range of issues, including voting, mechanism design, judgment aggregation, and fairness, and a similarly wide range of techniques, such as logic, experiments, and computational complexity. The paper by Agotnes, van der Hoek and Wooldridge defines a new modal logic for representing and reasoning about preference and judgment aggregation. The authors show that their logic can be used to express a variety of concepts and results from the social choice domain, including majority voting, the discursive paradox, Arrow’s theorem and Condorcet’s paradox. Baharad, Goldberger, Koppel and Nitzan propose a maximum likelihood estimation-based approach for aggregating voters’ opinions over several issues. They argue that their method is vastly superior to simple majority voting, and use both analytical and empirical evaluation to support their argument. Balan, Richards and Luke deal with the problem of selecting a sequence of actions when the actions have multiple beneficiaries. Their goal is to select the actions according to a lexicographic fairness criterion. The explore the complexity of this problem for both finite and infinite sequences of actions and present both algorithms and intractability results for this problem. The paper by Caminada and Pigozzi applies argumentation theory to judgment aggregation, and proposes aggregation operators satisfying not only the usual property of collective rationality, but another highly desirable property,

Preference aggregation and judgment aggregation are two basic research models of group decision making. And preference aggregation has been deeply studied in social choice theory. However, researches of social choice theory gradually focus on judgment aggregation which appears recently. Judgment aggregation focuses on how to aggregate many consistent logical formulas into one, from the perspective of logic. We try to start with judgment aggregation model based on logic and then explore different solutions to problem of judgment aggregation.

International audience

Judgment aggregation is a field in which individuals are required to vote for or against a certain decision (the conclusion) while providing reasons for their choice. The reasons and the conclusion are logically connected propositions. The problem is how a collective judgment on logically interconnected propositions can be defined from individual judgments on the same propositions. It turns out that, despite the fact that the individuals are logically consistent, the aggregation of their judgments may lead to an inconsistent group outcome, where the reasons do not support the conclusion. However, in this paper we claim that collective irrationality should not be the only worry of judgment aggregation. For example, judgment aggregation would not reject a consistent combination of reasons and conclusion that no member voted for. In our view this may not be a desirable solution. This motivates our research about when a social outcome is `compatible' with the individuals' judgments. The key notion that we want to capture is that any individual member has to be able to defend the collective decision. This is guaranteed when the group outcome is compatible with its members views. Judgment aggregation problems are usually studied using classical propositional logic. However, for our analysis we use an argumentation approach to judgment aggregation problems. Indeed the question of how individual evaluations can be combined into a collective one can also be addressed in abstract argumentation. We introduce three aggregation operators that satisfy the condition above, and we offer two definitions of compatibility. Not only does our proposal satisfy a good number of standard judgment aggregation postulates, but it also avoids the problem of individual members of a group having to become committed to a group judgment that is in conflict with their own individual positions.

Artificial intelligence (AI) systems are increasingly being used in consumer as well and business applications. Companies are using them, Governments are using them, public security, medical researchers, you and I – everybody seems to be somehow touched. As it becomes more advanced and ubiquitous in our lives, it is critical that AI developers and users understand the ethical implications and challenges posed by this powerful technology. This paper explores the key ethical skills and considerations that are essential for navigating the new world of AI. Artificial Intelligence (AI) presents significant ethical dimensions across various sectors, including research, business, and daily life. It is important to appreciate the major ethical dimensions of AI and how they impact AI systems. Policies and guidelines needs to be laid out for development and usage of such systems. Automated Decisions/Responsibility: No matter how intelligent a AI system might perceived to be, it is always prone to making errors. So, AI system that make automated decisions will always be dangerous. Therefore, it is important to ensure that there is always "human intervention" or "human review" for outcomes/decisions made by AI systems. Determining accountability for AI-generated actions or content remains a significant challenge, emphasizing the need for ethical governance frameworks. There has to be human accountability.Fairness/Bias: AI systems can perpetuate existing biases present in training data, leading to discriminatory outcomes.Transparency: The opaque nature of many AI algorithms complicates accountability, making it difficult to understand decision-making processes.Privacy: AI's capability to process vast datasets can infringe on individual privacy rights, necessitating stringent data protection measuresHuman Dignity: The risk of AI-generated content resembling existing works raises ethical issues regarding originality and authorship.

Although the categorization of ultrasound using the Breast Imaging Reporting and Data System (BI-RADS) has become widespread worldwide, the problem of inter-observer variability remains. To maintain uniformity in diagnostic accuracy, we have developed a novel artificial intelligence (AI) system in which AI can distinguish whether a static image obtained using a breast ultrasound represents BI-RADS3 or lower, or BI-RADS4a or higher, to determine the medical management that should be performed on a patient whose breast ultrasound shows abnormalities. To establish and validate the AI system, a training dataset consisting of 4,028 images containing 5,014 lesions and a test dataset consisting of 3,166 images containing 3,656 lesions were collected and annotated. We selected a setting that maximized the area under the curve (AUC) and minimized the difference in sensitivity and specificity by adjusting the internal parameters of the AI system, achieving an AUC, sensitivity, and specificity of 0.95, 90.0%, and 88.5%, respectively. Furthermore, based on 30 images extracted from the test data, the diagnostic accuracy of 20 clinicians and the AI system was compared, and the AI system was found to be significantly superior to the clinicians (McNemar test, p < 0.001). Then, we conducted a trial to introduce the system for use in clinical practice. Physicians reviewed the images and determined whether they were BI-RADS3 or lower, or BI-RADS4a or higher. Next, the classification was performed again for the same images concerning the AI diagnosis. At this time, the initial judgment was allowed to be overturned. We checked whether there was any difference in the diagnostic accuracy, sensitivity, and specificity before and after reviewing to the AI diagnosis. Reviews by 24 physicians were evaluated: 4 Japanese Breast Cancer Society breast specialists, 5 non-specialists and physicians with experience treating more than 40 cases of breast cancer, and 15 non-specialists and physicians with no experience treating more than 40 cases of breast cancer. The average rate of accuracy before confirming the AI diagnosis increased to 73.1% after confirming the AI diagnosis (p=0.00548), compared to 69.3% on average before the AI diagnosis. Compared to practice experience, the accuracy increased from an average of 77.1% to 79.6% for the 9 physicians who were breast specialists or who had treated 40 or more cases of breast cancer. For the 15 physicians with less than 40 breast cancer cases, the average rate of accuracy increased from 64.7% to 69.2%. Furthermore, sensitivity increased significantly to an average of 99.7% after reviewing of the AI diagnosis from an average of 88.8% prior to reviewing the AI-diagnosis.(p< 0.01). Specificity increased from an average of 62.4% to 63.8% (p=0.433) after reviewing AI diagnosis. We showed that our AI system, when applied to clinical practice and used by physicians, contributes to the improvement of diagnostic accuracy. Our results indicated that our AI diagnostic system was sufficiently accurate to be used in the clinical practice. Citation Format: Erina Odani, Tetsu Hayashida, masayuki kikuchi, Aiko Nagayama, tomoko seki, maiko takahashi, Akiko Matsumoto, Takeshi Murata, Rurina Watanuki, Takamichi Yokoe, Ayako Nakashoji, Hinako Maeda, Tatsuya Onishi, Sota Asaga, Takashi Hojo, Hiromitsu Jinno, Keiichi Sotome, Akira Matsui, Akihiko Suto, Shigeru Imoto, Yuko Kitagawa. Establishment of the breast ultrasound support system using deep-learning system [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P1-05-06.

We discuss how over the last 30 to 50 years, Artificial Intelligence (AI)\nsystems that focused only on data have been handicapped, and how knowledge has\nbeen critical in developing smarter, intelligent, and more effective systems.\nIn fact, the vast progress in AI can be viewed in terms of the three waves of\nAI as identified by DARPA. During the first wave, handcrafted knowledge has\nbeen at the center-piece, while during the second wave, the data-driven\napproaches supplanted knowledge. Now we see a strong role and resurgence of\nknowledge fueling major breakthroughs in the third wave of AI underpinning\nfuture intelligent systems as they attempt human-like decision making, and seek\nto become trusted assistants and companions for humans. We find a wider\navailability of knowledge created from diverse sources, using manual to\nautomated means both by repurposing as well as by extraction. Using knowledge\nwith statistical learning is becoming increasingly indispensable to help make\nAI systems more transparent and auditable. We will draw a parallel with the\nrole of knowledge and experience in human intelligence based on cognitive\nscience, and discuss emerging neuro-symbolic or hybrid AI systems in which\nknowledge is the critical enabler for combining capabilities of the\ndata-intensive statistical AI systems with those of symbolic AI systems,\nresulting in more capable AI systems that support more human-like intelligence.\n

There are two different fundamental conceptions of the task of social aggregation: interest aggregation and judgment aggregation. While the former aims at the fair and efficient trade-off of personal interests, the latter aims the best resolution of disagreement. Both are possible interpretations of the standard preference aggregation and preference-based voting problems. This note makes two broad points. First, the judgment aggregation conception is not yet well understood and remains undertheorized. Second, the difference between the two conceptions matters; it is a difference that makes a difference. To substantiate the first point, we argue that judgment aggregation as resolution of disagreement cannot be subsumed under statistical aggregation in the manner of Condorcet’s jury theorem, and sketch a more specific articulation as “reflexive consensus under arguable disagreement”. To substantiate the second point, we illustrate the difference in the context of two important voting rules, the Kemeny rule and Majority Judgment.

The primary challenge in the development of large-scale artificial intelligence (AI) systems lies in achieving scalable decision-making—extending the AI models while maintaining sufficient performance. Existing research indicates that distributed AI can improve scalability by decomposing complex tasks and distributing them across collaborative nodes. However, previous technologies suffered from compromised real-world applicability and scalability due to the massive requirement of communication and sampled data. Here we develop a model-based decentralized policy optimization framework, which can be efficiently deployed in multi-agent systems. By leveraging local observation through the agent-level topological decoupling of global dynamics, we prove that this decentralized mechanism achieves accurate estimations of global information. Importantly, we further introduce model learning to reinforce the optimal policy for monotonic improvement with a limited amount of sampled data. Empirical results on diverse scenarios show the superior scalability of our approach, particularly in real-world systems with hundreds of agents, thereby paving the way for scaling up AI systems.

Computational Social Choice is an interdisciplinary research area involving Economics, Political Science, and Social Science on the one side, and Mathematics and Computer Science (including Artificial Intelligence and Multiagent Systems) on the other side. Typical computational problems studied in this field include the vulnerability of voting procedures against attacks, or preference aggregation in multi-agent systems. Parameterized Algorithmics is a subfield of Theoretical Computer Science seeking to exploit meaningful problem-specific parameters in order to identify tractable special cases of in general computationally hard problems. In this paper, we propose nine of our favorite research challenges concerning the parameterized complexity of problems appearing in this context. This work is dedicated to Jianer Chen, one of the strongest problem solvers in the history of parameterized algorithmics, on the occasion of his 60th birthday.

Artificial intelligence and radiologists in prostate cancer detection on MRI (PI-CAI): an international, paired, non-inferiority, confirmatory study

BackgroundThe surgical difficulty of laparoscopic cholecystectomy (LC) for acute cholecystitis (AC) and the risk of bile duct injury (BDI) depend on the degree of fibrosis and scarring caused by inflammation; therefore, understanding these intraoperative findings is crucial to preventing BDI. Scarring makes it particularly difficult to perform safely and increases the BDI risk. This study aimed to develop an artificial intelligence (AI) system to indicate intraoperative findings of scarring in LC for AC.Materials and methodsAn AI system was developed to detect scarred areas using an algorithm for semantic segmentation based on deep learning. The training dataset consisted of 2025 images extracted from LC videos of 21 cases with AC. External evaluation committees (EEC) evaluated the AI system on 20 cases of untrained data from other centers. EECs evaluated the accuracy in identifying the scarred area and the usefulness of the AI system, which were assessed based on annotation and a 5-point Likert-scale questionnaire.ResultsThe average DICE coefficient for scarred areas between AI detection and EEC annotation was 0.612. The EEC’s average detection accuracy on the Likert scale was 3.98 ± 0.76. AI systems were rated as relatively useful for both clinical and educational applications.ConclusionWe developed an AI system to detect scarred areas in LC for AC. Since scarring increases the surgical difficulty, this AI system has the potential to reduce BDI.

As artificial intelligence (AI) systems continue to be developed, humans will increasingly participate in human-AI interactions. Humans interact with AI systems to achieve particular goals. To ensure that AI systems contribute positively to human-AI interactions, it is important to examine human-AI tasks with an emphasis on human goals and outcomes. The AI Use Taxonomy aims to provide a flexible means of classifying how an AI system contributes to an outcome. The taxonomy sets forward 16 AI use “activities” which are independent of AI techniques and domains. Tasks are combinations of one or more AI use activities. Future research includes applying the taxonomy to better understand measurement challenges for each activity. The taxonomy can contribute to an improved understanding of the architecture of human-AI tasks and help to foster positive, human-centered interactions with AI systems and optimal outcomes in the following ways: • Provides common terminology for describing outcome-based human-AI activities independent of AI techniques and domains • Enables cross-domain insights based on shared human-AI activities • Highlights commonalities in measurement and evaluation needs across disparate AI techniques • Facilitates the development of use cases • Facilitates the evaluation of trustworthiness characteristics and usability.

This scoping study explores the role of Human-in-the-Loop Artificial Intelligence (HITL AI) in precision agriculture and evaluates the benefits of using human expertise in combination with Artificial Intelligence (AI) systems in decision-making within modern smart agricultural environments. The development of Artificial Intelligence, Machine Learning, Internet of Things, and robotics has significantly impacted modern agriculture by providing automated crop monitoring, disease detection, yield prediction, and smart farm management systems. However, Artificial Intelligence systems also face challenges in terms of understanding, interpretability, flexibility, and trustworthiness in modern smart agricultural environments. This study is based on the literature regarding human-in-the-loop systems, human-centric Artificial Intelligence systems, and collaborative robotics systems in the context of smart agriculture. The structured scoping study methodology has been followed to identify and evaluate studies regarding Artificial Intelligence systems in smart agricultural environments, with a focus on automation-centric Artificial Intelligence systems and human-centric Artificial Intelligence systems within the context of Agriculture 5.0 concepts. The study concludes that although automation-centric AI systems show high accuracy in simulated smart agricultural environments, Human-In-The-Loop (HITL) AI systems show higher robustness in smart agricultural environments. Explainability in AI has shown significant potential in supporting the effectiveness of HITL AI systems in decision-making within smart agricultural environments. The study also identifies some important gaps in the literature regarding HITL AI systems in smart agriculture. The study concludes that for the development of modern smart precision agriculture, collaborative intelligence within smart agricultural environments is necessary to create sustainable smart agriculture systems.

This chapter explores relationships between the science and profession of psychology and the development of artificial intelligence (AI) and robotic systems. It briefly examines the role of psychology in the development of AI systems and theories. There is a long and intimate history of such a relationship, exemplified in the arts and the social sciences as well as in computer science and engineering. The chapter examines the role that psychology has played in the understanding of the ways in which humans interact with contemporary AI systems, to influence the behaviour of the systems and, in turn, to be influenced by them. Human decision making has been shown to be improved by factors other than intelligence and thereby appears to be a quality which would be retained by humans rather than machines, for example motivational processes, emotions and experience. The chapter examines the state of the profession of psychology and helping professions in the light of developments in AI.