• Home
  • Search
  • Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing
  • Cite Icon1
  • https://doi.org/10.1145/3728972Copy DOI Icon

Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Deep learning (DL) frameworks are essential to DL-based software systems, and framework bugs may lead to substantial disasters, thus requiring effective testing. Researchers adopt DL models or single interfaces as test inputs and analyze their execution results to detect bugs. However, floating-point errors, inherent randomness, and the complexity of test inputs make it challenging to analyze execution results effectively, leading to existing methods suffering from a lack of suitable test oracles. Some researchers utilize metamorphic testing to tackle this challenge. They design Metamorphic Relations (MRs) based on input data and parameter settings of a single framework interface to generate equivalent test inputs, ensuring consistent execution results between original and generated test inputs. Despite their promising effectiveness, they still face certain limitations. (1) Existing MRs overlook structural complexity, limiting test input diversity. (2) Existing MRs focus on limited interfaces, which limits generalization and necessitates additional adaptations. (3) Their detected bugs are related to the result consistency of single interfaces and far from those exposed in multi-interface combinations and runtime metrics (e.g., resource usage). To address these limitations, we propose ModelMeta, a model-level metamorphic testing method for DL frameworks with four MRs focused on the structure characteristics of DL models. ModelMeta augments seed models with diverse interface combinations to generate test inputs with consistent outputs, guided by the QR-DQN strategy. It then detects bugs through fine-grained analysis of training loss/gradients, memory/GPU usage, and execution time. We evaluate the effectiveness of ModelMeta on three popular DL frameworks (i.e., MindSpore, PyTorch, and ONNX) with 17 DL models from ten real-world tasks ranging from image classification to object detection. Results demonstrate that ModelMeta outperforms state-of-the-art baselines from the perspective of test coverage and diversity of generated test inputs. Regarding bug detection, ModelMeta has identified 31 new bugs, of which 27 have been confirmed, and 11 have been fixed. Among them, seven bugs existing methods cannot detect, i.e., five wrong resource usage bugs and two low-efficiency bugs. These results demonstrate the practicality of our method.

Similar Papers
  • Research Article
  • Citations38

A comprehensive empirical study on bug characteristics of deep learning frameworks

  • Nov 01, 2022
  • Information and Software Technology
  • Yilin Yang +3
  • Research Article

MirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing

  • Jan 01, 2026
  • IEEE Transactions on Software Engineering
  • Shiwen Ou +8
  • Conference Article
  • Citations8

Generating source inputs for metamorphic testing using dynamic symbolic execution

  • May 14, 2016
  • Eman Alatawi +2
  • Research Article

Automated segmentation of the primary tumor in nasopharyngeal carcinoma using a deep learning framework in positron emission tomography imaging: a comparative study

  • Oct 23, 2025
  • Quantitative Imaging in Medicine and Surgery
  • Meina Liang +9
  • Research Article
  • Citations1

Semi-supervised physics guided deep learning framework: An application in modeling of gallium nitride based high electron mobility transistors

  • Jul 22, 2022
  • Journal of Applied Physics
  • Shivanshu Mishra +2
  • Research Article
  • Citations12

PyRaDiSe: A Python package for DICOM-RT-based auto-segmentation pipeline construction and DICOM-RT data conversion

  • Jan 28, 2023
  • Computer Methods and Programs in Biomedicine
  • Elias Rüfenacht +6
  • Research Article
  • Citations8

Transfer learning for fluence map prediction in adrenal stereotactic body radiation therapy

  • Dec 06, 2021
  • Physics in Medicine & Biology
  • Wentao Wang +8
  • Conference Article
  • Citations1

Physics-guided Deep Learning for Branch Current Distribution System State Estimation

  • Dec 17, 2022
  • Y Raghuvamsi +2
  • PDF
  • Research Article
  • Citations58

DLBench: a comprehensive experimental evaluation of deep learning frameworks

  • Feb 07, 2021
  • Cluster Computing
  • Radwa Elshawi +3
  • Research Article

High-fidelity mechanical property confirmation of aluminum–agro-waste composites via integrated analytical, experimental, and deep learning framework

  • Apr 01, 2026
  • Next Materials
  • Stephen Ndubuisi Nnamchi +3
  • Conference Article

Test Input Validation for Vision-Based DL Systems: An Active Learning Approach

  • Apr 27, 2025
  • Delaram Ghobari +5
  • Research Article
  • Citations4

A Novel Ant Colony Based DBN Framework to Analyze the Drug Reviews

  • Dec 08, 2021
  • International Journal of Intelligent Systems and Applications
  • Nazia Tazeen +1
  • Conference Article
  • Citations1

Exploring Metamorphic Testing for Self-learning Functions with User Interactions

  • Jan 01, 2024
  • AHFE international
  • Marco Stang +3
  • Research Article
  • Citations1940

DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics

  • Mar 21, 2018
  • Computer Physics Communications
  • Han Wang +3
  • Research Article
  • Citations168

Blockchain for deep learning: review and open challenges

  • Mar 14, 2022
  • Cluster Computing
  • Muhammad Shafay +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.