• Home
  • Search
  • OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
  • Open Access IconOpen Access
  • Cite Icon7
  • https://doi.org/10.18653/v1/2024.findings-acl.808Copy DOI Icon

OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models

  • Jan 1, 2024
  • Shuai Wang +5 more
Show More
  • Abstract
  • Literature Map
  • Citations
  • Similar Papers
Abstract

Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favour of functional programming (FP), e.g., HumanEval and MBPP.To address this, our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k metric. Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field.Our benchmark and scripts are publicly released at: https: //github.com/alphadl/OOP-eval.

Similar Papers
  • Research Article
  • Citations14

Should ML be Object-Oriented?

  • Jul 01, 2002
  • Formal Aspects of Computing
  • David Macqueen
  • Single Book
  • Citations395

Modern Compiler Implementation in C

  • Dec 13, 1997
  • Andrew W Appel +1
  • Supplementary Content

Co-Contextual Type Systems: Contextless Deductive Reasoning for Correct Incremental Type Checking

  • Mar 02, 2020
  • TUbilio (Technical University of Darmstadt)
  • Edlira Kuci
  • Research Article
  • Citations39

Object-Oriented Programming, Functional Programming and R

  • May 01, 2014
  • Statistical Science
  • John M Chambers
  • Research Article
  • Citations18

Large Language Model-Aware In-Context Learning for Code Generation

  • Feb 28, 2025
  • ACM Transactions on Software Engineering and Methodology
  • Jia Li +7
  • Research Article
  • Citations2

Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values

  • May 30, 2025
  • Aerospace
  • Rui He +4
  • PDF
  • Conference Article
  • Citations7

LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

  • Apr 14, 2024
  • Bruno Pereira Cipriano +1
  • Research Article
  • Citations1

Functional Programming at Work in Object- Oriented Programming.

  • Jan 01, 2009
  • The Journal of Object Technology
  • Philippe Narbel
  • Conference Article
  • Citations6

Comparative analysis of functional and object-oriented programming

  • May 01, 2016
  • Dino Alic +2
  • Research Article
  • Citations3

A Comparative Study of Object-Oriented, Procedural, and Functional Programming Paradigms in Microservice Architecture

  • Sep 24, 2025
  • VFAST Transactions on Software Engineering
  • Mustafa Ahmed Khan +5
  • Research Article
  • Citations5

HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent

  • Jan 27, 2025
  • ACM Transactions on Software Engineering and Methodology
  • Jie Jw Wu +1
  • Research Article
  • Citations32

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

  • Jun 22, 2025
  • Proceedings of the ACM on Software Engineering
  • Ruiqi Wang +5
  • Research Article
  • Citations1

Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation

  • Jan 01, 2026
  • IEEE Transactions on Software Engineering
  • Binquan Zhang +6
  • Research Article
  • Citations27

A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages

  • Oct 07, 2025
  • ACM Transactions on Software Engineering and Methodology
  • Sathvik Joel +2
  • Research Article
  • Citations24

AceCoder : An Effective Prompting Technique Specialized in Code Generation

  • Nov 21, 2024
  • ACM Transactions on Software Engineering and Methodology
  • Jia Li +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.