Optimization of Image B-spline Interpolation for GPU Architectures

Image processing pipelines combine the challenges of stencil computations and stream programs. They are composed of large graphs of different stencil stages, as well as complex reductions, and stages with global or data-dependent access patterns. Because of their complex structure, the performance difference between a naive implementation of a pipeline and an optimized one is often an order of magnitude. Efficient implementations require optimization of both parallelism and locality, but due to the nature of stencils, there is a fundamental tension between parallelism, locality, and introducing redundant recomputation of shared values. We present a systematic model of the tradeoff space fundamental to stencil pipelines, a schedule representation which describes concrete points in this space for each stage in an image processing pipeline, and an optimizing compiler for the Halide image processing language that synthesizes high performance implementations from a Halide algorithm and a schedule. Combining this compiler with stochastic search over the space of schedules enables terse, composable programs to achieve state-of-the-art performance on a wide range of real image processing pipelines, and across different hardware architectures, including multicores with SIMD, and heterogeneous CPU+GPU execution. From simple Halide programs written in a few hours, we demonstrate performance up to 5x faster than hand-tuned C, intrinsics, and CUDA implementations optimized by experts over weeks or months, for image processing applications beyond the reach of past automatic compilers.

Advances in computer hard- and software have enabled the automated extraction of biomarkers from large scale imaging studies by means of image processing pipelines. For large cohort studies, ample storage- and computing resources are required: pipelines are typically executed in parallel on one or more High Performance Computing Clusters (HPC). As processing is distributed, it becomes more cumbersome to obtain detailed progress and status information of large-scale experiments. Especially in a research-oriented environment, where image processing pipelines are often in an experimental stage, debugging is a crucial part of the development process that relies heavily on a tight collaboration between pipeline developers and clinical researchers. Debugging a running pipeline is a challenging and time-consuming process for seasoned pipeline developers, and nearly impossible for clinical researchers, often involving parsing of complex logging systems and text files, and requires special knowledge of the HPC environment. In this paper, we present the Pipeline Inspection and Monitoring web application (PIM). The goal of PIM is to make it more straightforward and less time-consuming to inspect complex, long running image processing pipelines, irrespective of the level of technical expertise and the workflow engine. PIM provides an interactive, visualization-based web application to intuitively track progress, view pipeline structure and debug running image processing pipelines. The level of detail is fully customizable, supporting a wide variety of tasks (e.g. quick inspection and thorough debugging) and thereby facilitating both clinical researchers and pipeline developers in monitoring and debugging.

Muscle fiber images play an important role in the medical diagnosis and treatment of many muscular diseases. The number of nuclei in skeletal muscle fiber images is a key bio-marker of the diagnosis of muscular dystrophy. In nuclei segmentation one primary challenge is to correctly separate the clustered nuclei. In this article, we developed an image processing pipeline to automatically detect, segment, and analyze nuclei in microscopic image of muscle fibers. The pipeline consists of image pre-processing, identification of isolated nuclei, identification and segmentation of clustered nuclei, and quantitative analysis. Nuclei are initially extracted from background by using local Otsu's threshold. Based on analysis of morphological features of the isolated nuclei, including their areas, compactness, and major axis lengths, a Bayesian network is trained and applied to identify isolated nuclei from clustered nuclei and artifacts in all the images. Then a two-step refined watershed algorithm is applied to segment clustered nuclei. After segmentation, the nuclei can be quantified for statistical analysis. Comparing the segmented results with those of manual analysis and an existing technique, we find that our proposed image processing pipeline achieves good performance with high accuracy and precision. The presented image processing pipeline can therefore help biologists increase their throughput and objectivity in analyzing large numbers of nuclei in muscle fiber images.

Remote photoplethysmography (rPPG) enables non-contact vital sign measurements using standard smart device cameras, opening up the potential of scalable health applications on consumer smart devices. However, rPPG signal quality is highly sensitive to camera sensor characteristics and image processing pipelines, which can vary between devices. This variation limits reproducibility and generalisation of rPPG-based algorithms beyond specific hardware platforms. This work presents a reproducible test environment for the validation of the camera sensor response in the context of rPPG signals. A microcontroller-driven illumination system and mechanically constrained setup are used to generate controlled, repeatable optical signals. Two characterisation tests are introduced: a time domain morphology analysis and a frequency domain attenuation analysis. Pulse timing consistency, pulse waveform morphology and normalised frequency responses are compared to assess sensor similarity. This method is applied to selected consumer devices and demonstrates consistent camera response patterns under the controlled test conditions. By explicitly addressing validation of the camera sensor and image processing pipeline, this work supports the development of more robust and transferable rPPG-based vital sign applications across a wider range of consumer devices.

Pipeline image processing systems are increasingly being used in industrial vision systems, in particular in manufacture visual inspection. There has been little work done in the study of the appropriate software environment for rapid program development for these machines. In this paper, an approach to improve software programmability of these systems was proposed. The software features reflecting the hardware capabilities as well as the nature of image processing applications were discussed. A preprocessor was developed to support the high level programming environment and the abstract data structures. The preprocessor also analyzed the image processing tasks expressed as expressions and generate the hardware setup and connections necessary for multipass operations through the pipeline processor. A prototype implementation of the preprocessor and an example were also discussed.

High-throughput microscopy makes it possible to observe the morphology of zebrafish on large scale to quantify genetic, toxic or drug effects. The image acquisition is done by automated microscopy, images are evaluated automatically by image processing pipelines, tailored specifically to the requirements of the scientific question. The transfer of such algorithms to other projects, however, is complex due to missing guidelines and lack of mathematical or programming knowledge. In this work, we implement an image processing pipeline for automatic fluorescence quantification in user-defined domains of zebrafish embryos and larvae of different age. The pipeline is capable of detecting embryos and larvae in image stacks and quantifying domain activity. To make this protocol available to the community, we developed an open source software package called „ZebrafishMiner“ which guides the user through all steps of the processing pipeline and makes the algorithms available and easy to handle. We implemented all routines in an MATLAB-based graphical user interface (GUI) that gives the user control over all image processing parameters. The software is shipped with a manual of 30 pages and three tutorial datasets, which guide the user through the manual step by step. It can be downloaded at https://sourceforge.net/projects/scixminer/.

Image processing pipelines are ubiquitous and we rely on them either directly, by filtering or adjusting an image post-capture, or indirectly, as image signal processing (ISP) pipelines on broadly deployed camera systems. Used by artists, photographers, system engineers, and for downstream vision tasks, traditional image processing pipelines feature complex algorithmic branches developed over decades. Recently, image-to-image networks have made great strides in image processing, style transfer, and semantic understanding. The differentiable nature of these networks allows them to fit a large corpus of data; however, they do not allow for intuitive, fine-grained controls that photographers find in modern photo-finishing tools. This work closes that gap and presents an approach to making complex photo-finishing pipelines differentiable, allowing legacy algorithms to be trained akin to neural networks using first-order optimization methods. By concatenating tailored network proxy models of individual processing steps (e.g. white-balance, tone-mapping, color tuning), we can model a non-differentiable reference image finishing pipeline more faithfully than existing proxy image-to-image network models. We validate the method for several diverse applications, including photo and video style transfer, slider regression for commercial camera ISPs, photography-driven neural demosaicking, and adversarial photo-editing.

Light-sheet fluorescence imaging is a cutting-edge 3-dimensional imaging technique that allows the analysis of biological structures and processes in-vivo and in-vitro with exceptional image resolutions [1].In recent years, this technique has evolved to increase the microscope field of view and imaging throughput resulting in the generation of huge amounts of data in the range of hundreds of Gigabytes to Terabytes per specimen [2].Consequently, this surge in data volumes presents challenges related to storage, data organization, image processing, analysis, metadata management, and visualization.Thus far, laboratories have predominantly relied on on-premise computing resources and storage solutions for data processing.These solutions afford them complete control over data and resources, thereby facilitating superior latency in data accessibility and customized security measures.Nevertheless, this approach is fraught with limitations, including constrained flexibility of compute resources, external data accessibility, scalability, and disaster recovery capabilities [3].At the Allen Institute for Neural Dynamics, we are studying large populations of specimens and sharing the data to the scientific community as early as possible which led us to rely on cloud computing resources and cloud friendly formats (OMEZarr) to develop our image processing pipelines.We are engaged in the imaging process of whole mouse brains mounted on the SmartSPIM, a Selective Plane Illumination Microscope (SPIM), and commercial light-sheet microscope system developed by LifeCanvas Technologies.Imaging is usually conducted at a resolution of 2.0 m z step, with an x-y pixel size of 1.8 m.This system facilitates the acquisition of highresolution images through the utilization of various lasers and adjustable laser powers, enabling the generation of multiple channels for dataset acquisition.In this work, we introduce the SmartSPIM pipeline, an end-to-end automated cloud light-sheet image processing pipeline which uses Code Ocean and Amazon Web Services to efficiently handle vast quantities of data.To the best of our knowledge, it is the first automated cloud-based light-sheet image processing pipeline that has been successfully deployed for this purpose.This pipeline includes rapid data transfer from a high-performance on-premise storage system to the cloud, creation of metadata, flatfield correction, image correction to remove stripe noise, large-scale image stitching facilitated by TeraStitcher [4], image registration to the Allen Mouse Brain Common Coordinate Framework (CCF) [5], a cell localization module using Cellfinder [6], quantification of cell counts per CCF region, and a cloud-native visualization of the stitched 3D volume and cell distributions categorized by brain region with Neuroglancer.It is worth mentioning that raw data, code, and results are made fully accessible to the public as an integral component of fair, open and reproducible science.Our approach involves structuring our raw data with experiment metadata.Furthermore, the results of each processed dataset include metadata derived from both raw data and all subsequent image processing modules.This derived metadata comprehensively encompasses inputs, algorithmic parameters, utilized tools, timestamps, and software versions employed at each stage of the pipeline, facilitating future reproducibility efforts.Lastly, all our code is publicly available on Github.This pipeline is capable of concurrently processing multiple datasets.Typically, the dataset size varies between 1 and 2 Terabytes, contingent upon the number of channels and nature of the experiment.For instance, in the case of a dataset of 1.5 Terabytes and 3 channels, the processing times are currently as follows: 3.3 hours for applying flatfield correction and removing stripe noise, 0.2 hours for image stitching while exclusively considering image registration within overlapping image stacks in a single channel, 1.7 hours for fusing the data with OMEZarr format, 0.1 hours for registering a single channel to the CCF atlas, 5 hours for whole-brain cell localization in multiple channels, and 0.2 hours for mapping identified cells to each CCF region.

In the domain of image processing, deep learning has assumed the position of the spearhead of innovation, creating revolutionary changes in numerous fields, such as image categorization, object recognition, and image generation. This white paper aims at developing a more detailed overview of the modern deep learning methods which are primarily designed for image processing. Analyzing the essential aspects of CNNs, GANs, and RNNs we wish to reveal innovative achievements in this area. In this white paper, it highlights a broad variety of domains, where these deep learning techniques have demonstrated remarkable progress. In the field of medical imaging, where CNNs help in the diagnosis and prognosis of the disease, to satellite imagery analysis, where GANs improve the generation of synthetic realistic data, and computer vision, where RNNs allow for better video analysis, deep learning is redefining the limits of image processing. Development should also be highlighted as an issue that the integration of deep learning into the image processing pipeline may bring along with it, including data accessibility, model interpretability, and computing resources. Accordingly, acknowledging such issues and staying updated with new trends, researchers and practitioners will be able to fully utilize deep learning and use it to solve complex image processing problems and carry out innovation in many different spheres of life. Deep learning is here to stay at the cutting edge of image processing, transforming our abilities and perspectives in this ever - changing discipline.

Programming a Pipelined Image Processor

Spatial image transformation is a commonly used component in many image processing pipelines. It enables the correction of optical distortions, image registration onto a common reference plane, electronic image stabilisation, digital zoom, video mosaicking, etc. With the growing tendency to embed image processing in low-power devices, attaining an efficient transformation solution becomes increasingly decisive. Furthermore, interpolation is the key operation in achieving the high quality of the transformed data from the original data. Fortunately, different implementations have already seen several efficiency improvements in recent years. However, interpolation relies on sampling a set of neighbouring points from memory, which has yet to be addressed efficiently for smaller computational platforms with limited memory resources. In this work, we derive a generic mathematical model and circuit design principles for the spatial transformation accelerator design for N-dimensional data. Furthermore, we present an efficient simultaneous access scheme for high-quality signal reconstruction. Finally, the introduced ideas are verified in field programmable gate arrays using one-dimensional and two-dimensional data transformation use cases. The presented solution is able to transform images with sizes ranging from 256 × 256 to 8192 × 8192 and achieves a transfer rate of 275 frames per second with 512 × 512 images.

A study on the possibility of implementing a real-time stereoscopic 3D rendering TV system

Introduction:  The JAXA Martian Moons Exploration (MMX) mission [1] will deploy the French-German IDEFIX rover (Fig. 1) to the surface of Phobos [2]. The IDEFIX rover will attempt wheeled-locomotion on a low-gravity surface for the first time thus providing a unique opportunity to study the surface properties of Phobos and the behaviour of regolith on small-bodies.     Figure 1 (left) The MMX IDEFIX rover (image credit: JAXA). (right) A WheelCam (image credit: CNES)The WheelCams:  The IDEFIX rover includes two WheelCams, placed on the underside of the rover and each aimed at a different rover wheel [3]. The WheelCam image sensors are panchromatic and consist of a 2048 by 2048 array. The optics provide a field of view of 32.5° and a pixel resolution of approximately 100 μm at the centre of the image. The WheelCams are also equipped with LEDs to illuminate the scene; white LEDs to be used while driving, and three colour LEDs (590, 720 and 880 nm) to allow for multispectral imaging. The WheelCams can be operated in both an imaging and a movie mode. Typical movie frame rates are expected to be 1 image per mm moved for the front WheelCam, 1 image per cm moved for the rear WheelCam (the anticipated rover speed is ~0.1-4 mm/s).Science Objectives of the WheelCams: The WheelCam images will be used to characterize the general grain properties of the regolith (size distribution, morphological parameters [3]), within the limits of the resolution. The WheelCam images will also be used to determine the adhesive properties of regolith particles stuck to the rover wheels and the depth of the wheel sinkage, which is closely linked to the load bearing strength [4] and friction angle of the regolith [5].  The linear and angular velocities of the wheel will be calculated from the WheelCam images in order to provide measurements of the traction and slippage of the wheel. In addition to providing important information about the performance of the locomotion, this also provides the shearing characteristics of the regolith [6,7]. Topographical reconstruction of the rover tracks, talus and tailings behind the wheels [8] will provide additional constraints on the physical properties of the regolith.  Finally, using the coloured LEDs it will be possible to study the mineralogical composition of the regolith from reflectance and albedo measurements, and to asses space weathering by comparing inside and outside of the rover tracks.Preparing for the WheelCam analyses: In anticipation of the MMX mission, we have developed the MMX WheelCam testbed (Fig. 2, [9]) that recreates the scene that the WheelCams will observe during the mission (Fig. 3). The main objective of the testbed is to develop the image processing tools for the WheelCams. The testbed is instrumented with multiple sensors allowing measurements to be made of the sinkage of the wheel into the soil, and the slippage (essentially the loss of traction) of the wheel. The testbed also includes the MMX rover wheel, camera baffles, the LEDs and blackout panels to perform trials in representative lighting conditions.  Figure 2. Two different views of the MMX WheelCam testbed at ISAE-SUPAERO.Figure 3. WheelCam approximate field of views, as generated by the WheelCam testbed. (left) Front WheelCam perspective. (right) Rear WheelCam perspective.We also perform Discrete Element Method simulations [10] in order to understand the influence of the low gravity environment on sinking and driving behaviour (Fig. 4; [11, 12]]. This is essential to ensure an accurate determination of the regolith properties from the wheel – surface interactions.Figure 4. Snapshot from a rolling simulation where gravity is 0.006 m/s2 and the rotational velocity is 0.65 rad/s [11,12].Conclusions: This presentation will discuss the expected science return from the MMX IDEFIX rover WheelCams and provide a status of the preparatory activities and image processing pipelines under development.Acknowledgments: We acknowledge CNES funding.

A New Wavelet-based image denoising using undecimated discrete wavelet transform and least squares support vector machine

Binary images are often used in image processing pipelines, and are usually stored in the unpacked format, i.e., 8 bits per pixel. The packed format uses 1 bit per pixel, 8 times less memory, and is a good option when dealing with images too big to fit in RAM. This paper presents a parallel implementation of pixelwise and window operators for packed binary images. The implementation, written in OpenCL, can run in GPUs or multiple core CPUs. The proposed Destination Word Accumulation (DWA) implementation of morphological operations is faster than Leptonica in 2D and up to two orders of magnitude faster than Python and MATLAB in 1D to 5D.