• Home
  • Search
  • Significantly reducing the processing times of high-speed photometry data sets using a distributed computing model
  • https://doi.org/10.1117/12.924863Copy DOI Icon

Significantly reducing the processing times of high-speed photometry data sets using a distributed computing model

  • Abstract
  • Literature Map
  • Similar Papers
Abstract

The scientific community is in the midst of a data analysis crisis. The increasing capacity of scientific CCD instrumentation and their falling costs is contributing to an explosive generation of raw photometric data. This data must go through a process of cleaning and reduction before it can be used for high precision photometric analysis. Many existing data processing pipelines either assume a relatively small dataset or are batch processed by a High Performance Computing centre. A radical overhaul of these processing pipelines is required to allow reduction and cleaning rates to process terabyte sized datasets at near capture rates using an elastic processing architecture. The ability to access computing resources and to allow them to grow and shrink as demand fluctuates is essential, as is exploiting the parallel nature of the datasets. A distributed data processing pipeline is required. It should incorporate lossless data compression, allow for data segmentation and support processing of data segments in parallel. Academic institutes can collaborate and provide an elastic computing model without the requirement for large centralized high performance computing data centers. This paper demonstrates how a base 10 order of magnitude improvement in overall processing time has been achieved using the ACN pipeline, a distributed pipeline spanning multiple academic institutes.

Similar Papers
  • Research Article
  • Citations11

Development of Data Processing and Analysis Pipeline for the Ricochet Experiment

  • Nov 21, 2022
  • Journal of Low Temperature Physics
  • J Colas +4
  • Book Chapter
  • Citations2

Chapter 11 - Technical design: data processing pipeline in eHealth: The case of ECG data sets

  • Jan 01, 2022
  • Anomaly Detection and Complex Event Processing Over IoT Data Streams
  • Patrick Schneider +1
  • Research Article
  • Citations222

Improving classification accuracy using data augmentation on small data sets

  • Jul 15, 2020
  • Expert Systems with Applications
  • Francisco J Moreno-Barea +2
  • PDF
  • Research Article
  • Citations2

Comparison of sequencing methods and data processing pipelines for whole genome sequencing and minority single nucleotide variant (mSNV) analysis during an influenza A/H5N8 outbreak

  • Feb 20, 2020
  • PLoS ONE
  • Marjolein J Poen +12
  • Research Article
  • Citations12

Efficient reuse of replicated parallel data segments in computational grids

  • Jan 16, 2008
  • Future Generation Computer Systems
  • Sandip Tikar +1
  • Supplementary Content
  • Citations1

Very Long Baseline Interferometry in ItalyWide-field VLBI imaging and astrometry and prospects for an Italian VLBI network including the Sardinia Radio Telescope

  • Apr 19, 2010
  • AMS Dottorato Institutional Doctoral Theses Repository (University of Bologna)
  • John S Morgan
  • Conference Article
  • Citations1

An Innovative Machine Learning Method for Predicting Well Performance in Unconventional Reservoirs with a Relatively Small Data Set

  • Mar 07, 2023
  • Hui-Hai Liu +3
  • Research Article
  • Citations2

A hybrid approach based on transfer and ensemble learning for improving performances of deep learning models on small datasets

  • Nov 30, 2021
  • TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES
  • Tunç Gültekin +1
  • Research Article
  • Citations104

Machine learning-based inverse design methods considering data characteristics and design space size in materials design and manufacturing: a review.

  • Jan 01, 2023
  • Materials Horizons
  • Junhyeong Lee +6
  • Conference Article
  • Citations2

Combining Remote and In-situ Sensing for Persistent Monitoring of Water Quality

  • Feb 21, 2022
  • Cesar A Rojas +5
  • Research Article
  • Citations3

Evolutionary Strategies AI Addresses Multiple Technical Challenges in Deep Learning Deployment: Proof-of-Principle Demonstration for Neuroblastoma Brain Metastasis Detection.

  • Jun 17, 2024
  • Journal of imaging informatics in medicine
  • Subhanik Purkayastha +9
  • Preprint Article

Rescuing the Winds: Up-to-date Open Access to the Historic QBO Time Series

  • Jul 16, 2025
  • Tobias Kerzenmacher +1
  • Conference Article
  • Citations2

Methods of voxel data rendering for visualizing on multi-layer volumetric displays

  • May 16, 2018
  • Gatis Valters +7
  • Research Article
  • Citations216

Missing values in mass spectrometry based metabolomics: an undervalued step in the data processing pipeline

  • Oct 08, 2011
  • Metabolomics
  • Olga Hrydziuszko +1
  • Conference Article
  • Citations1

Improving Grid Scheduling of Pipelined Data Processing by Combining Heuristic Algorithms and Simulated Annealing

  • Jun 01, 2006
  • Qingjiang Wang +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.