• Home
  • Search
  • Large Language Model-Aware In-Context Learning for Code Generation
  • Cite Icon18
  • https://doi.org/10.1145/3715908Copy DOI Icon

Large Language Model-Aware In-Context Learning for Code Generation

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Large Language Models (LLMs) have shown impressive In-Context Learning (ICL) ability in code generation. LLMs take a prompt context consisting of a few demonstration examples and a new requirement as input, and output new programs without any parameter update. Existing studies have found that the performance of ICL-based code generation heavily depends on the quality of demonstration examples and thus arises research on selecting demonstration examples: given a new requirement, a few demonstration examples are selected from a candidate pool, where LLMs are expected to learn the pattern hidden in these selected demonstration examples. Existing approaches are mostly based on heuristics or randomly selecting examples. However, the distribution of randomly selected examples usually varies greatly, making the performance of LLMs less robust. The heuristics retrieve examples by only considering textual similarities of requirements, leading to sub-optimal performance. To fill this gap, we propose a L arge language model- A ware selection approach for I n-context- L earning-based code generation named LAIL. LAIL uses LLMs themselves to select examples. It requires LLMs themselves to label a candidate example as a positive example or a negative example for a requirement. Positive examples are helpful for LLMs to generate correct programs, while negative examples are trivial and should be ignored. Based on the labeled positive and negative data, LAIL trains a model-aware retriever to learn the preference of LLMs and select demonstration examples that LLMs need. During the inference, given a new requirement, LAIL uses the trained retriever to select a few examples and feed them into LLMs to generate desired programs. We apply LAIL to four widely used LLMs and evaluate it on five code generation datasets. Extensive experiments demonstrate that LAIL outperforms the state-of-the-art (SOTA) baselines by 11.58%, 3.33%, and 5.07% on CodeGen-Multi-16B, 1.32%, 2.29%, and 1.20% on CodeLlama-34B, and achieves 4.38%, 2.85%, and 2.74% improvements on Text-davinci-003 in terms of Pass@1 at MBJP, MBPP, and MBCPP, respectively. In addition to function-level code generation, LAIL improves the performance of LLMs on DevEval, a repository-level code generation dataset, which achieves 10.04%, 8.12%, and 4.63% improvements compared to the SOTA baselines at Pass@1, 3, and 5 on CodeLlama-7B. Human evaluation further verifies that the generated programs of LAIL are superior in correctness, code quality, and maintainability. Besides, LAIL has satisfactory transferability across different LLMs and datasets, where the retriever learned on one LLM (dataset) can be transferred to other LLMs (datasets).

Similar Papers
  • Conference Article

CrossLM: A Data-Free Collaborative Fine-Tuning Framework for Large and Small Language Models

  • Jun 23, 2025
  • Yongheng Deng +5
  • Research Article
  • Citations32

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

  • Jun 22, 2025
  • Proceedings of the ACM on Software Engineering
  • Ruiqi Wang +5
  • Research Article
  • Citations11

How Can IJDS Authors, Reviewers, and Editors Use (and Misuse) Generative AI?

  • Apr 01, 2023
  • INFORMS Journal on Data Science
  • Galit Shmueli +7
  • Research Article
  • Citations6

Tax Intelligent Decision-Making Language Model

  • Jan 01, 2024
  • IEEE Access
  • Yan Zhong +2
  • Research Article
  • Citations2

Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values

  • May 30, 2025
  • Aerospace
  • Rui He +4
  • Research Article
  • Citations1

AI-Powered Code Generation Evaluating the Effectiveness of Large Language Models (LLMs) in Automated Software Development

  • Mar 31, 2023
  • Journal of Artificial Intelligence & Cloud Computing
  • Ravikanth Konda
  • Research Article
  • Citations5

HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent

  • Jan 27, 2025
  • ACM Transactions on Software Engineering and Methodology
  • Jie Jw Wu +1
  • Research Article
  • Citations1

Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation

  • Jan 01, 2026
  • IEEE Transactions on Software Engineering
  • Binquan Zhang +6
  • Research Article
  • Citations1

How well will LLMs perform for graph layout tasks?

  • Mar 01, 2026
  • Visual Informatics
  • Yilun Fan +5
  • Research Article
  • Citations27

A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages

  • Oct 07, 2025
  • ACM Transactions on Software Engineering and Methodology
  • Sathvik Joel +2
  • Research Article
  • Citations24

AceCoder : An Effective Prompting Technique Specialized in Code Generation

  • Nov 21, 2024
  • ACM Transactions on Software Engineering and Methodology
  • Jia Li +4
  • Conference Article
  • Citations3

Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming

  • Nov 25, 2025
  • Humza Ashraf +4
  • Research Article
  • Citations7

Evaluating the performance of large language & visual-language models in cervical cytology screening

  • May 23, 2025
  • npj Precision Oncology
  • Qi Hong +15
  • Supplementary Content
  • Citations1

Large language and vision-language models for robot: safety challenges, mitigation strategies and future directions

  • Jul 29, 2025
  • Industrial Robot: the international journal of robotics research and application
  • Xiangyu Hu +1
  • Research Article
  • Citations2

Evaluating the Accuracy and Diagnostic Reasoning of Multimodal Large Language Models in Interpreting Neuroradiology Cases From RadioGraphics.

  • Jan 01, 2026
  • Korean journal of radiology
  • Pae Sun Suh +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.