Adaptive demosaicking with improved edge detection

Digital still cameras use a single electronic sensor overlaid with a color filter array (CFA) such that each sensor pixel only samples one of the three primary color values. To render a full-color image from the CFA samples, the two missing color values are estimated from its neighbors. This process is called CFA demosaicking. This paper introduces and compare between eight demosaicking methods using peak signal to noise ratio and mean square error in RGB color space and perceived error in CIELAB color space.

Since the birth of charge coupled devices (CCD) and the complementary metal-oxide-semiconductor (CMOS) active pixel sensors, pixel pitch of digital image sensors has been continuously shrinking to meet the resolution and size requirements of the cameras. However, shrinking pixels reduces the maximum number of photons a sensor can hold, a phenomenon broadly known as the full-well capacity limit. The drop in full-well capacity causes drop in signal-to-noise ratio and dynamic range.The Quanta Image Sensor (QIS) is a class of solid-state image sensors proposed by Eric Fossum in 2005 as a potential solution for the limited full-well capacity problem. QIS is envisioned to be the next generation image sensor after CCD and CMOS since it enables sub-diffraction-limit pixels without the inherited problems of pixel shrinking. Equipped with a massive number of detectors that have single-photon sensitivity, the sensor counts the incoming photons and triggers a binary response “1” if the photon count exceeds a threshold, or “0” otherwise. To acquire an image, the sensor oversamples the space and time to generate a sequence of binary bit maps. Because of this binary sensing mechanism, the full-well capacity, signal-to-noise ratio and the dynamic range can all be improved using an appropriate image reconstruction algorithm. The contribution of this thesis is to address three image processing problems in QIS: 1) Image reconstruction, 2) Threshold design and 3) Color filter array design.Part 1 of the thesis focuses on reconstructing the latent grayscale image from the QIS binary measurements. Image reconstruction is a necessary step for QIS because the raw binary measurements are not images. Previous methods in the literature use iterative algorithms which are computationally expensive. By modeling the QIS binary measurements as quantized Poisson random variables, a new non-iterative image reconstruction method based on the Transform-Denoise framework is proposed. Experimental results show that the new method produces better quality images while requiring less computing time.Part 2 of the thesis considers the threshold design problem of a QIS. A spatially-varying threshold can significantly improve the reconstruction quality and the dynamic range. However, no known method of how to achieve this can be found in the literature. The theoretical analysis of this part shows that the optimal threshold should match with the underlying pixel intensity. In addition, the analysis proves the existence of a set of thresholds around the optimal threshold that give asymptotically unbiased reconstructions. The asymptotic unbiasedness has a phase transition behavior. A new threshold update scheme based on this idea is proposed. Experimentally, the new method can provide good estimates of the thresholds with less computing budget compared to existing methods.Part 3 of the thesis extends QIS capabilities to color imaging by studying how a color filter array should be designed. Because of the small pixel pitch of QIS, crosstalk between neighboring pixels is inevitable and should be considered when designing the color filter arrays. However, optimizing the light efficiency while suppressing aliasing and crosstalk in a color filter array are conflicting tasks. A new optimization framework is proposed to solve the problem. The new framework unifies several mainstream design criteria while offering generality and flexibility. Extensive experimental comparisons demonstrate the effectiveness of the framework.

Entropy encoder for low-power low-resources high-quality CFA image compression

A novel scheme of scalable video coding (SVC) using super-resolution techniques is proposed in this paper. Utilizing the spatial/temporal scalability of H.264/AVC, we encode half of input high-resolution(HR) frames and their low-resolution (LR) counterparts in a video sequence and employ super-resolution (SR) method to reconstruct skipped frames during decoding. The payload saved from skipped frames can be used to improve the quality of encoded frames. This scheme provides a choice of SVC to improve the quality of HR frames while maintaining low bit rate transmission and reducing encoding complexity. Experiments show that our scheme performs well in both peak signal to noise ratio (PSNR) and subjective visual quality at low bit rate.

Most commercial digital cameras use a single electronic sensor overlaid with a color filter array (CFA) to capture imagery. Since only one primary color is sampled in each pixel, the missing color primaries must be reconstructed by interpolation. In this paper, an adaptive demosaicking scheme for CFA interpolation is proposed. The scheme uses intra-channel correlation, color difference correlation, constant hue, and luminance-color difference correlation is proposed. A rough interpolation is first implemented by bilinear interpolation. Then the color difference correlation and constant hue are successively used to update the missing color primaries. To obtain high quality color images, an adaptive algorithm using luminance-color difference correlation and the information of edge direction is iteratively applied to improve the image quality around the edges. Simulation results demonstrate that the image quality of the proposed algorithm is better than that of the approach using color difference correlation in terms of peak signal-to-noise ratio (PSNR).KeywordsColor filter arraydemosaickinghueluminance

Rate control algorithm is of great importance to video coding. Many existing schemes are based on the quadratic rate control scheme. The proposed method introduces a novel rate control algorithm with adaptive rounding offset, aiming at achieving bit-rate optimization and adaptation of the content of the video sequences. The rounding offset is chosen adaptively based on the minimum blocks’ discontinuities at the boundaries simply. The experimental results show that compared with a dead-zone adjustment which is H.264/AVC compliant, the proposed method can achieve considerable improvement on both objective peak signal noise ratio (PSNR) and subjective visual quality.

Most digital cameras capture one primary color at each pixel by a single sensor overlaid with a color filter array. To recover a full color image from incomplete color samples, one needs to restore the two missing color values for each pixel. This restoration process is known as color demosaicking. In this paper, we present a novel self-learning approach to this problem via support vector regression. Unlike prior learning-based demosaicking methods, our approach aims at extracting image-dependent information in constructing the learning model, and we do not require any additional training data. Experimental results show that our proposed method outperforms many state-of-the-art techniques in both subjective and objective image quality measures.

Common consumer RGB cameras use a single image sensor to capture RGB color images with a color filter array (CFA) placed in front of the image sensor. This system captures particularly one color sample ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> , or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$B$</tex-math></inline-formula> ) at each pixel location. This captured raw image is called a CFA image. The CFA demosaicking is used then to construct the complete color image. A similar approach of working with a single image sensor and extension of CFA to multispectral filter array (MSFA) enables us to develop a low-cost multispectral camera with the help of MSFA demosaicking. However, due to many spectral bands and their very sparse sampling, extending CFA demosaicking to MSFA demosaicking is not easy. This paper proposes an adaptive and progressive MSFA demosaicking method for various MSFA patterns for 5 to 15 bands multispectral images. Binary tree-based MSFA patterns are used to create MSFA images, and these MSFA patterns can be designed for any N-band MSFA image. The spectral bands are arranged specifically so that the middle band has the highest probability of appearance (PoA) in the MSFA pattern. The middle band is interpolated first using PoA based progressive adaptive interpolation, and the modified bilinear spectral difference method is used to estimate all other spectral bands with the interpolated middle band’s help. Experimental results and comparative analysis on two different datasets reveal that our proposed MSFA demosaicking method outperforms the existing generic MSFA demosaicking methods in terms of computational time and various image quality metrics considered.

As the size of the images being captured increases, there is a need for a robust algorithm for image compression which satiates the bandwidth limitation of the transmitted channels and preserves the image resolution without considerable loss in the image quality. Many conventional image compression algorithms use wavelet transform which can significantly reduce the number of bits needed to represent a pixel and the process of quantization and thresholding further increases the compression. In this paper the authors evolve two sets of wavelet filter coefficients using genetic algorithm (GA), one for the whole image portion except the edge areas and the other for the portions near the edges in the image (i.e., global and local filters). Images are initially separated into several groups based on their frequency content, edges, and textures and the wavelet filter coefficients are evolved separately for each group. As there is a possibility of the GA settling in local maximum, we introduce a new shuffling operator to prevent the GA from this effect. The GA used to evolve filter coefficients primarily focuses on maximizing the peak signal to noise ratio (PSNR). The evolved filter coefficients by the proposed method outperform the existing methods by a 0.31 dB improvement in the average PSNR and a 0.39 dB improvement in the maximum PSNR.

To compare subjective visual quality and patient-reported outcomes after bilateral implantation of three diffractive trifocal intraocular lenses (IOLs): AT LISA tri 839MP, AcrySof IQ PanOptix, and Tecnis Synergy. Prospective, randomized, comparative clinical trial. Forty-three patients with bilateral cataract were randomized to one of the three trifocal IOL models. A single surgeon used standardized techniques, and assessments were performed 3 months after second-eye surgery. Subjective visual quality was evaluated using three validated patient-reported outcome measures: Cataract TyPE Spec (TyPE), Near Activity Visual Questionnaire (NAVQ), and Catquest-9SF (Rasch-modeled). Outcomes included global and item-level scores, satisfaction, and photic phenomena. Nonparametric tests (Kruskal-Wallis; Mann-Whitney U with Bonferroni) were applied at the patient level. All IOLs yielded high visual function and satisfaction, with ceiling effects across questionnaires. The median TyPE global scores were 56 (AT LISA tri), 54 (PanOptix), and 59 (Synergy) ( P = 0.322). NAVQ indicated minimal difficulty with near/intermediate tasks ( P = 0.127). The Catquest-9SF Rasch person measures were -3.52 ± 0.77, -3.64 ± 0.35, and -3.79 ± 0.29, respectively ( P = 0.221). One TyPE intermediate-vision item favored Synergy over PanOptix ( P = 0.015). Photic phenomena were infrequent and comparable among groups. All three diffractive trifocal IOLs provided excellent subjective visual performance, satisfaction, and spectacle independence. Although Synergy showed a slight advantage for intermediate tasks, overall differences were minimal, supporting consistently high patient-reported outcomes with current trifocal optics.

Visually lossless compression for Bayer color filter array using optimized Vector Quantization

In general digital camera uses a single-chip image sensor, with spectrally shifted alternating patterns of three or more color filters applied to each pixel location called color filter array. Digital cameras with single sensor use demosaicing to fabricate full color images from the data captured by a color filter array (CFA). The goodness of demosaiced images is subject to influences from noise in the image sensor during the image capturing process. In order to reduce the noise the conventional solution like Joint Demosaicing and denoising may be adopted, that combines both demosaicing and denoising procedures in to a single operation. Another method, Total least square denoising method is also used to improve image quality. But, In PCA based spatially adaptive denoising of CFA, images uses the concept of denoising first and demosaicing later for reducing the noise. It has some advantages such as less noise-caused color artifacts and cost-effective implementation. When compared to Joint Demosaicing and denoising, PCA Based spatially adaptive denoising of CFA Images is more effective in reducing the noise in terms of both objective measurement and visual evaluation. This method effectively suppresses noise while preserving color edges and details. The direct CFA image denoising scheme, followed by a succeeding demosaicing scheme reduces extensively the noise-caused color artifacts in the demosaiced images. Such artifacts often appear in the output full-color images of many demosaicing first and denoising later schemes, as well as joint demosaicing-denoising schemes. The CPSNR ratio is much better, when compared to Joint demosaicing and denoising schemes by preserving the structures in the image which are often smoothed by other denoising schemes.

To investigate whether adding accelerated under-flap corneal cross-linking to hyperopic laser in situ keratomileusis (LASIK-ufCXL) affects postoperative stability and regression, visual and refractive outcomes, and subjective quality of vision. This prospective comparative contralateral eye study included 51 patients with hyperopia (102 eyes) who received LASIK-ufCXL in the eye with highest defocus equivalent (DEQ) or randomized when DEQ equal, with the contralateral control eye receiving LASIK alone. After excimer ablation, 0.25% riboflavin was instilled on the stromal bed for 3 minutes. The flap was repositioned, followed by a total irradiation dose of 3.24 J ultraviolet A (UV-A) light administered to the corneal surface, using 18 mW/cm2 UV-A for 3 minutes. Postoperative hyperopic regression (stability) was the primary outcome measure, defined by the difference in spherical equivalent (SEQ) at 1 week and 24 months postoperatively. Secondary measures reported uncorrected distance visual acuity, corrected distance visual acuity, cylinder vector analysis, subjective quality of vision, subjective night vision disturbances, and corneal haze. At 24 months, the SEQ stability (P = .4273) and the magnitude of hyperopic regression (P = .5613) did not significantly differ between groups, with a small trend showing hyper-opic regression of 0.50 diopters or greater being less frequent in LASIK-ufCXL eyes. There were no significant differences in accuracy, efficacy, and safety (P > .05), with a small trend of more residual refractive astigmatism in the LASIK-ufCXL group (P = .3216, Cohen's d: -0.29). Subjective quality of vision trended inferior in LASIK-ufCXL eyes (P = .2237, Cohen's d: -0.25), with a greater haze grading (P = .0466, Cohen's d: 0.41). Postoperative regression and stability were statistically equivalent between hyperopic LASIK vs LASIK-ufCXL, with identical safety. There were small clinical trends of lower efficacy, accuracy, and subjective quality of vision in LASIK-ufCXL eyes. [J Refract Surg. 2022;38(12):770-779.].

Temporal and spatial correlation-based reversible data hiding for RGB CFA videos

An efficient and novel imagery compression system for Synthetic Aperture Radar (SAR) which uses integer to integer wavelet transform and Modified Set Partitioning Embedded Block Coder (M-SPECK) has been presented in this paper. The presence of speckle noise, detailed texture, high dynamic range in SAR images, and even its vast data volume show the great differences of SAR imagery. Integer to integer wavelet transform is invertible in finite precision arithmetic, it maps integers to integers, and approximates linear wavelet transforms from which they are derived. Considering in terms of computational load, compression ratio and subjective visual quality metrics, several filter banks are compared together and some factors affecting the compression performance of the integer to integer wavelet transform are discussed in details. Then the optimal filter banks which are more appropriate for the SAR images compression are given. Information of high frequency has relatively larger proportion in SAR images compared with those of nature images. Measures to modify the quantizing thresholds in traditional SPECK are taken, which could be suitable to the contents of SAR imagery for the purpose of compression. Both the integer to integer wavelet transform and modified SPECK have the desirable feature of low computational complexity. Experimental results show its superiority over the traditional approaches in the condition of tradeoffs between compression efficiency and computational complexity.