• Home
  • Search
  • CP-Bench: Evaluating Large Language Models for Constraint Modelling
  • Cite Icon1
  • https://doi.org/10.3233/faia250890Copy DOI Icon

CP-Bench: Evaluating Large Language Models for Constraint Modelling

Show More
  • Abstract
  • Literature Map
  • Citations
  • Similar Papers
Abstract

Constraint Programming (CP) is widely used to solve combinatorial problems, but its core process, namely constraint modelling, requires significant expertise and is considered to be a bottleneck for wider adoption. Aiming to alleviate this bottleneck, recent studies have explored using Large Language Models (LLMs) to transform combinatorial problem descriptions into executable constraint models. However, the existing evaluation datasets for constraint modelling are often limited to small, homogeneous, or domain-specific instances, which do not capture the diversity of real-world scenarios. This work addresses this gap by introducing CP-Bench, a novel benchmark that includes a diverse set of well-known combinatorial problems sourced from the CP community, structured explicitly for evaluating LLM-driven CP modelling. With this dataset, and given the variety of constraint modelling frameworks, we compare and evaluate the modelling capabilities of LLMs for three distinct constraint modelling systems, which vary in abstraction level and underlying syntax. Notably, the results show higher performance when modelling with a high-level Python-based framework. Additionally, we systematically evaluate the use of prompt-based and inference-time compute methods across different LLMs, which further increase accuracy, reaching up to 70% on this highly challenging benchmark.

Similar Papers
  • Research Article

Unlocking the Potential of Large Language Models in Education: Factors Influencing Adoption by Instructional Designers and Academics

  • Jan 01, 2026
  • Journal of Information Technology Education: Research
  • Katherine L Fourie +2
  • Research Article

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

  • Nov 03, 2025
  • Blood
  • Samantha Rizzo +5
  • Supplementary Content

Large language and vision-language models for robot: safety challenges, mitigation strategies and future directions

  • Jul 29, 2025
  • Industrial Robot: the international journal of robotics research and application
  • Xiangyu Hu +1
  • Research Article

Research and selection of Large Learning Models for automation of ABAP-code migration

  • Sep 24, 2025
  • Management of Development of Complex Systems
  • Oleg Pozdnyakov +1
  • Research Article

Readability & quality of large language model responses in CAR-T patient education

  • Nov 03, 2025
  • Blood
  • Sridhar Balasubramanian +3
  • Research Article

Application of Large Language Models (LLMs) to Geriatric Practice and Its Evaluation at 4 VA GRECCs

  • Dec 01, 2025
  • Innovation in Aging
  • Huai Cheng +2
  • Research Article

1401 Bringing Medicine Expertise to Your Screen: A New Frontier in Curbside Sleep Consultation Leveraging Large Language Models?

  • May 19, 2025
  • SLEEP
  • Nina Kuei +4
  • Front Matter
  • Citations1

Editorial: Large language models in work and business.

  • Nov 29, 2024
  • Frontiers in artificial intelligence
  • Şadi Evren Şeker
  • Research Article
  • Citations15

Large language models in neurosurgery: a systematic review and meta-analysis.

  • Nov 23, 2024
  • Acta neurochirurgica
  • Advait Patil +5
  • Research Article

#2924 Comparison of large language models and traditional natural language processing techniques in predicting arteriovenous fistula failure

  • May 23, 2024
  • Nephrology Dialysis Transplantation
  • Suman Lama +6
  • Research Article
  • Citations1

Optimization of traditional methods for determining the similarity of project names and purchases using large language models

  • Apr 01, 2024
  • Litera
  • Aleksei Aleksandrovich Golikov +2
  • Supplementary Content
  • Citations114

Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review

  • Nov 07, 2024
  • Journal of Medical Internet Research
  • Leyao Wang +7
  • Research Article

Large language model use in oral and maxillofacial surgery training: a national resident survey.

  • Feb 21, 2026
  • Oral and maxillofacial surgery
  • Nolan Kranc +7
  • Research Article
  • Citations4

Documenting Disclosure: Limited Reporting of Generative AI Usage in Radiology Research Manuscripts.

  • Oct 01, 2025
  • Academic radiology
  • D Jonah Barrett +2
  • Conference Article
  • Citations18

Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

  • Jan 01, 2024
  • Daniel P Jeong +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.