Hardware-friendly universal demosaick using non-iterative map reconstruction

There is a recent trend change in using non-Bayer Color Filter Array (CFA) patterns, allowing superior image quality in low-light conditions. However, these new CFA patterns have weak point on full image reconstruction showing severe visual artifacts and low details reconstruction after demosaicing. In this work, we address aforementioned problems by using deep learning approach for new CFA type - Nonacell or Nonapixel, introduced by Samsung 108MP HMX CMOS image sensor. Experimental results show that proposed method not only allows suppression of visual artifacts and perfect details and edge restoration, but also shows superior objective image quality, exceeding 40dB in CPSNR for popular Kodak dataset. Finally, our method is computationally efficient due to network structure and feasible for on-device deployment after optimization.

This paper describes a demosaicking method for hexagonally-structured color filter array. Demosaicking is essential to acquire color images using color filter array (CFA) in single sensor imaging. Thus, CFA patterns have been discussed in order to improve image quality in single sensor imaging after the Bayer pattern are introduced. Advancements in imaging sensor technology recently introduce a hexagonal CFA pattern. The hexagonal CFA can be considered to be a 45-degree rotational version of the Bayer pattern, thus demosaicking can be implemented by an existing method with backward and forward 45-degree rotations. However, this approach requires heavy computing power and memory in image sensing devices because of the image rotations. To overcome this problem, we proposes a demosaicking method for a hexagonal Bayer CFA without rotations. In addition, we introduce a weighting parameter in our demosaicking method to improve image quality and to unifying exiting method with our method. Experimental results indicate that the proposed method is superior to conventional methods in terms of PSNR. In addition, some optimized values for the weighting parameter are provided experimentally.

Digital still cameras use a single electronic sensor overlaid with a color filter array (CFA) such that each sensor pixel only samples one of the three primary color values. To render a full-color image from the CFA samples, the two missing color values are estimated from its neighbors. This process is called CFA demosaicking. This paper introduces and compare between eight demosaicking methods using peak signal to noise ratio and mean square error in RGB color space and perceived error in CIELAB color space.

This paper introduced a problem of the modified Bayer pattern color filter array (CFA). A demosaicking method is also known as color interpolation, which is a digital camera process employed to restore a full-color imagery from an image with missing color information. In general, a CFA pair contains two green pixels and one red and blue pixel (RGGB). However, there exist alternatives such as RRGB or RGBB. In this paper, we study the effect of three different color arrays. Simulation results show that the obtained filters give good performance.

We present a convolutional neural network architecture for performing joint design of color filter array (CFA) patterns and demosaicing. Our generic model allows the training of CFAs of arbitrary sizes, optimizing each color filter over the entire RGB color space. The patterns and algorithms produced by our method provide high‐quality color reconstructions. We demonstrate the effectiveness of our approach by showing that its results achieve higher PSNR than the ones obtained with state‐of‐the‐art techniques on all standard demosaicing datasets, both for noise‐free and noisy scenarios. Our method can also be used to obtain demosaicing strategies for pre‐defined CFAs, such as the Bayer pattern, for which our results also surpass even the demosaicing algorithms specifically designed for such a pattern.

In this paper, we propose fast color interpolation methods for color filter array (CFA). The proposed methods exploit correlations among the three color components. It has been reported that the quality of color interpolation can be significantly improved by utilizing correlation among color components. However, the complexity of such methods may increase significantly compared to those methods which separately interpolate each color component. On the other hand, the major application of CFA is portable imaging devices such as digital cameras or mobile phones. For portable devices, power consumption is an important issue. In this paper, we propose high quality color interpolation methods with reduced complexity. Experimental results show that the proposed methods provide noticeable improvements without significantly increasing the system complexity.

Demosaicking is reconstructing a RGB image from the mosaicked image recorded by a single sensor with a color filter array (CFA), which has achieved great progress via deep learning. Some works on joint CFA design and demosaicking via deep learning have been presented. However, almost all existing approaches focus only on optimizing filter arrangements without considering the used spectral sensitivity functions (SSFs). The commonly used SSFs are to mimic human trichromatic perception, which are not optimal for deep-learning-based de-mosaicking. In this paper, we simultaneously learn the CFA and demosaicking using a deep convolutional neural network, where both the filter arrangement and used SSFs are optimized. By modeling the applying of SSFs as a convolutional layer with physical constraints, we formulate our joint learning approach as an end-to-end autoencoder, which can be trained as the standard convolutional neural networks. Our approach allows the designing of CFA patterns with arbitrary sizes, where the filter arrangements can be predefined or not. We demonstrate the effectiveness and advantages of our approach by comparing with fixed Gaussian functions and various predefined filter arrangements.

Signal CCD/CMOS sensors capture image information by covering the sensor surface with a color filter array(CFA). For each pixel, only one of three primary colors(red, green and blue) can pass through the color filter array(CFA). The other two missing color components are estimated by the values of the surrounding pixels. In Bayer array, the green components are half of the total pixels, but both red pixel and blue pixel components are quarter, so green components contain more information, which can be reference to color interpolation of red components and blue components. Based on this principle, in this paper, a simple and effective color interpolation algorithm based on green components and signal correlation for Bayer pattern images was proposed. The first step is to interpolate R, G and B components using the method-bilinear interpolation. The second step is to revise the results of bilinear interpolation by adding some green components on the results of bilinear interpolation. The calculation of the values to be added should consider the influence of correlation between the three channels. There are two major contributions in the paper. The first one is to demosaick G component more precisely. The second one is the spectral-spatial correlations between the three color channels is taken into consideration. At last, through MATLAB simulation experiments, experimental pictures and quantitative data for performance evaluation-Peak Signal to Noise Ratio(PSNR) were gotten. The results of simulation experiments show, compared with other color interpolation algorithms, the proposed algorithm performs well in both visual perception and PSNR measurement. And the proposed algorithm does not increase the complexity of calculation but ensures the real-time of system. Theory and experiments show the method is reasonable and has important engineering significance.

A new edge-adaptive demosaicing algorithm for color filter arrays

The main idea behind wavelet based demosaicing with spatial refinement is to reconstruct the full resolution color image from the mosaiced image. In this study, a new effective wavelet based demosaicing algorithm for interpolating the missing color components in Bayer’s Color Filter Array (CFA) pattern is proposed. This interpolation technique uses the interchannel correlation among the high frequency subbands to determine the missing pixels in each color channel, followed by a refining step in spatial domain which uses non-iterative technique that enforces color difference rule with fewer computations. As a result, the proposed demosaicing method yields better performance than bilinear, edge based and subband based demosaicing methods.

A new zooming algorithm suitable for single-sensor imaging devices is introduced. The proposed method is capable of zooming and enlarging data acquired by single sensor cameras operating on the Bayer color filter array (CFA) pattern. The approach allows for operations on noise-free data at the hardware level. Complexity and cost implementation are thus greatly reduced. Based on a color difference model and an adaptive edge-sensing mechanism, the proposed method produces sharp, visually pleasing outputs. At the same time, it yields excellent performance, in terms of both subjective and objective image quality measure.

Lossless compression of HDR color filter array image for the digital camera pipeline

Most digital cameras use a single sensor coupled with a Color Filter Array (CFA) to capture images, and apply demosaicking to interpolate the full color images. In reality, the CFA image is noisy, which causes problems in the demosaicking process. This paper proposes a Joint Denoising and Demo-saicking based on inter-Color correlation (JDDC) scheme. We propose a new framework that linearly combines an extracted luminance image and a low-passed RGB images to get a full color image. Given the noise in the extracted luminance image and the low-passed RGB images are non-stationary and partially correlated, we modify the classical Non-Local Means (NLM) filter to denoise the extracted luminance image and the low-passed RGB images before the combination. Experimental results verify the effectiveness of the proposed scheme both objectively and subjectively.

Medical endoscopic video processing requires real-time execution of color component acquisition, color filter array (CFA) demosaicing, and high dynamic range (HDR) compression under low-light conditions, while adhering to strict thermal constraints within the surgical handpiece. Traditional hardware-aware neural architecture search (NAS) relies on fixed hardware design spaces, making it difficult to balance accuracy, power consumption, and real-time performance. A collaborative “power-accuracy” optimization method is proposed for hardware-aware NAS. Firstly, we proposed a novel hardware modeling framework by abstracting FPGA heterogeneous resources into unified cell units and establishing a power–temperature closed-loop model to ensure that the handpiece surface temperature does not exceed clinical thresholds. In this framework, we constrained the interstage latency balance in pipelines to avoid routing congestion and frequency degradation caused by deep pipelines. Then, we optimized the NAS strategy by using pipeline blocks and combined with a hardware efficiency reward function. Finally, color component acquisition, CFA demosaicing, dynamic range compression, dynamic precision quantization, and streaming architecture are integrated into our framework. Experiments demonstrate that the proposed method achieves 2.8 W power consumption at 47 °C on a Xilinx ZCU102 platform, with a 54% improvement in throughput (vs. hardware-aware NAS), providing an engineer-ready lightweight network for medical edge devices such as endoscopes.

An impartial framework to investigate demosaicking input embedding options