A color constancy model for advanced television cameras

Invariance of surface color representations across illuminant changes in the human cortex

The red-green axis of colour vision evolved recently in primate evolutionary history. Signals serving red-green colour vision travel together with signals serving spatial vision, in the parvocellular (PC) division of the subcortical visual pathway. However, the question of whether receptive fields of PC pathway cells are specialized to transmit red-green colour signals remains unresolved. We addressed this question in single-cell recordings from the lateral geniculate nucleus of anaesthetized marmosets. Marmosets show a high proportion of dichromatic (red-green colour-blind) individuals, allowing spatial and colour tuning properties of PC cells to be directly compared in dichromatic and trichromatic visual systems. We measured spatial frequency tuning for sine gratings that provided selective stimulation of individual photoreceptor types. We found that in trichromatic marmosets, the foveal visual field representation is dominated by red-green colour-selective PC cells. Colour selectivity of PC cells is reduced at greater eccentricities, but cone inputs to centre and surround are biased to create more selectivity than predicted by a purely 'random wiring' model. Thus, one-to-one connections in the fovea are sufficient, but not necessary, to create colour-selective responses. The distribution of spatial tuning properties for achromatic stimuli shows almost complete overlap between PC cells recorded in dichromatic and trichromatic marmosets. These data indicate that transmission of red-green colour signals has been enabled by centre-surround receptive fields of PC cells, and has not altered the capacity of PC cells to serve high-acuity vision at high stimulus contrast.

Colour vision and colour signals are important to aquatic animals, but light scattering and absorption by water distorts spectral stimuli. To investigate the performance of colour vision in water, and to suggest how photoreceptor spectral sensitivities and body colours might evolve for visual communication, we model the effects of changes in viewing distance and depth on the appearance of fish colours for three teleosts: a barracuda, Sphyraena helleri, which is dichromatic and two damselfishes, Chromis verater and Chromis hanui, which are trichromatic. We assume that photoreceptors light-adapt to the background, thereby implementing the von Kries transformation, which can largely account for observed colour constancy in humans and other animals, including fish. This transformation does not, however, compensate for light scattering over variable viewing distances, which in less than a metre seriously impairs dichromatic colour vision, and makes judgement of colour saturation unreliable for trichromats. The von Kries transformation does substantially offset colour shifts caused by changing depth, so that from depths of 0 to 30 m modelled colour changes (i.e. failures of colour constancy) are sometimes negligible. However, the magnitudes and directions of remaining changes are complex, depending upon the specific spectral sensitivities of the receptors and the reflectance spectra. This predicts that when judgement of colour is important, the spectra of signalling colours and photoreceptor spectral sensitivities should be evolutionarily linked, with the colours dependent on photoreceptor spectral sensitivities, and vice versa.

Image acquisition devices inherently do not have color constancy mechanism like human visual system. Machine color constancy problem can be circumvented using a white balancing technique based upon accurate illumination estimation. Unfortunately, previous study can give satisfactory results for both accuracy and stability under various conditions. To overcome these problems, we suggest a new method: spatial and temporal illumination estimation. This method, an evolution of the Retinex and Color by Correlation method, predicts on initial illuminant point, and estimates scene-illumination between the point and sub-gamuts derived by from luminance levels. The method proposed can raise estimation probability by not only detecting motion of scene reflectance but also by finding valid scenes using different information from sequential scenes. This proposed method outperforms recently developed algorithms.

Neurons in the lateral geniculate nucleus cannot perform the spatial color calculations necessary for color contrast and color constancy. Under neutral-adapting conditions, we mapped the cone inputs (L, M, and S) to 83 cone-opponent cells representing the central visual field of the next stage of visual processing, primary visual cortex (V1), to determine how the color signals are spatially transformed. Cone-opponent cells, constituting approximately 10% of V1 cells, formed two populations, red-green (L vs M; 66 of 83) and blue-yellow (S vs L+M; 17 of 83). Many cone-opponent cells (48 of 83) were double-opponent, with circular receptive-field centers and crescent-shaped surrounds (0.63 degree offset) that had opposite chromatic tuning to the centers and a time-to-peak 11 ms later than the centers. The remaining cone-opponent cells were either spatially opponent in only one cone system (20 of 83) or lacked spatial opponency (15 of 83). Cells lacking spatial opponency had smaller receptive fields (0.5-0.7 degrees) than spatial-opponent cell centers (approximately 1 degree). We found that red-green cells received S-cone input, which aligned with M input, and, unlike blue-yellow cells, red-green cells gave push-pull responses: receptive-field centers of red-ON cells were excited by both L increments (bright red) and M decrements (dark red) and were suppressed by both L decrements (dark green) and M increments (bright green). Excitatory responses to decrements were slightly larger than to increments, which may account for the lower detection and discrimination thresholds of decrements shown psychophysically. By virtue of their specialized receptive fields, the neurons described here spatially transform the cone signals and represent the first stage in the visual system at which spatially opponent color calculations are made.

Different authors have studied the response of the human visual system (HVS) in the chromatic frequency domain and have obtained its spectral modulation sensitivity function (SMSF), either computationally or experimentally. The Fourier analysis applied to the colour signals (product of the spectral power distribution of the illuminant and the reflectance or transmittance function of an object) carries out the chromatic frequency domain. The SMSF defined in this domain is analogous to the CSF or the TMTF, which are defined in the spatial and temporal domains, respectively. We have studied how HVS processes chromatic information of daylight and natural objects. The analysis of the original and filtered colour signal with different cut-off frequencies reveals some degree of degradation of these colour signals. This is particularly important when the colour signal are characterised by peaked spectral power distributions.

We present a color-constancy algorithm that uses quantum-catch data from reflected lights to recover surface reflectance functions and illuminant spectral power distributions. The algorithm recovers both surface and light-source spectral properties simultaneously. The method works in all situations that were handled by the earlier two-stage algorithms of Maloney and Wandell [J. Opt. Soc. Am. A 3, 29 (1986)] and D'Zmura and Iverson [J. Opt. Soc. Am. A 9, 490 (1992); 10, 2148, 2166 (1993); 11, 1970 (1994)]. In addition, the method handles problems that lie outside the scope of earlier algorithms. Using this method, a trichromatic visual system can recover, when provided adequate information, spectral descriptions of arbitrarily high accuracy for lights and surfaces. We determine conditions under which bilinear models can be used to recover color properties uniquely with the new procedure, and we formulate an algorithm for checking whether a particular bilinear model provides perfect color constancy. This research extends our analysis of linear methods for color constancy begun earlier [J. Opt. Soc. Am. A 10, 2148, 2166 (1993)].

We describe an analysis method of the omnidirectional color signals in natural scenes. A multiband imaging system with six spectral channels is used for capturing high resolution images in the omnidirectional observations at three locations on campus. The spectral distributions of color signals are recovered using the Wiener estimator from the captured six-band images. The spectral compositions of omnidirectional color signals are investigated based on the PCA of each set of color signals acquired at three locations in different seasons and different times of the day. Three principal components are extracted from three sets of omnidirectional images observed in three different locations. The respective three principal component curves are invariant under seasonal and temporal changes. Moreover, we determine the unified principal components of color signals across all locations. High data compression of omnidirectional images can be achieved. The reliability of the proposed analysis method is confirmed using various experimental data.

It has been shown that the illuminant and object spectra could be approximated to a very high degree of accuracy by linearly combining a few principal components. As early as 1964, Judd, MacAdam, and Wyszecki reported that various daylight illuminations could be approximated by three components (see Section 1.5) and Cohen showed that the spectra of Munsell color chips could be accurately represented by a few principal components. Using principal component analysis (PCA), Cohen reported that the first basis vector alone accounted for 92.72% of the cumulative variance, and merely three or four vectors accounted for 99% variance or better. As a result, principal-component analysis became an extremely powerful tool for the computational color technology community. These findings led to the establishment of a finite-dimensional linear model. Combining with the rich contents of linear algebra and matrix theory, the model provides powerful applications in color science and technology. Many color scientists and researchers have contributed to the building and applying of this model to numerous color-image processing areas such as color transformation, white-point conversion, metameric pairing, indexing of metamerism, object spectrum reconstruction, illuminant spectrum reconstruction, color constancy, chromatic adaptation, and targetless scanner characterization. In this chapter, we present several methods for spectrum decomposition and reconstruction, including orthogonal projection, smoothing inverse, Wiener inverse, and principal component analysis. Principal components from several publications are compiled and evaluated by using a set of fourteen spectra employed for the color rendering index. Their similarities and differences are discussed. New methods of spectrum reconstruction directly from tristimulus values are developed and tested.

Color constancy is still one of the biggest challenges in camera color processing. Convolutional neural networks have been able to improve the situation but there are still problems in many conditions, especially in scenes where a single color is dominating. In this work, we approach the problem from a slightly different setting. What if we could have some other information than the raw RGB image data. What kind of information would help to bring significant improvements while still be feasible in a mobile device. These questions sparked an idea for a novel approach for computational color constancy. Instead of raw RGB images used by the existing algorithms to estimate the scene white points, our approach is based on the scene’s average color spectra-single pixel spectral measurement. We show that as few as 10–14 spectral channels are sufficient. Notably, the sensor output has five orders of magnitude less data than in raw RGB images of a 10MPix camera. The spectral sensor captures the “spectral fingerprints” of different light sources and the illuminant white point can be accurately estimated by a standard regressor. The regressor can be trained with generated measurements using the existing RGB color constancy datasets. For this purpose, we propose a spectral data generation pipeline that can be used if the dataset camera model is known and thus its spectral characterization can be obtained. To verify the results with real data, we collected a real spectral dataset with a commercial spectrometer. On all datasets the proposed Single Pixel Spectral Color Constancy obtains the highest accuracy in the both single and cross-dataset experiments. The method is particularly effective for the difficult scenes for which the average improvements are 40–70% compared to state-of-the-arts. The approach can be extended to multi-illuminant case for which the experimental results also provide promising results.

The White-Patch method, one of the very first colour constancy methods, estimates the illuminant colour from the maximum response of three colour channels. However, this simple method has been superseded by advanced physical, statistical and learning based colour constancy methods. Recently, a few research works have suggested that the simple idea of using maximum pixel values is not as limited an idea as it seems on first glance. These works show that in several situations some manipulations can indeed made it perform very well. Here, we extend the White-Patch assumption to include any of: white patch, highlights or light source; let us refer to these pixels in an image as the “bright” pixels areas. We propose that bright pixels are surprisingly helpful in the illumination estimation process.In this paper, we investigate the effects of bright pixels on several current colour constancy algorithms. Moreover, we describe a simple framework for an illumination estimation method based on bright pixels and compare its accuracy to well-known colour constancy algorithms applied to four standard datasets. We also investigate failure and success cases, using bright pixels, and propose desiderata on input images with regard to the proposed method.

The image color recorded in a computer vision system depends on three factors: the physical content of the scene, the illumination on the scene, and the characteristics of the camera. The goal of computational color correction is to reset the intensities of the color channels in digital images, and express the scene colors canonically. In the color correction method based on spectral reflectance reproduction, the finite-dimensional linear-model is an effective way to condense spectral reflectance. Since there are only three channels for each pixel in digital images, color correction based on spectral reflectance reproduction using the finite-dimensional model with three dimensions is needed. In the definite gamut, the spectral reflectance curves are analyzed using the finite-dimensional model. It is shown that reflectance generated by three basis functions which is capturing 99.13% of the overall variance, provides good approximations to the measured spectral reflectance. Together with the measured spectral power distribution of the illumination and the estimated spectral responses of the camera, the quantum catches are calculated. Errors caused by approximation of the finite-dimensional model are relatively small, which shows that color correction based on the finite-dimensional linear-model has feasibility to some extent.

Given the spectral mechanism of color constancy (Part I of this series), the remaining nonspectral mechanism is formulated here in Part II by the constraint of correlation with known spectral illuminant–invariant functions, i.e., invariant wavelength ratios between constant hues, which plot straight parallel lines in the plane of wavelength and reciprocal illuminant color temperature (MK−1). The same is assumed to apply to nonspectral constant hues in the same plane and dominant wavelength scale extended to cover the nonspectrals (see accompanying article “Relative wavelength metric for the complete hue cycle …”). To simplify analysis, stimuli are optimal aperture colors; their monochromatic stimuli lie between 442 and 613 nm, common boundaries with optimal compound stimuli (nonspectrals). It is shown that the wavelengths and invariant ratios of spectral constant hues can be formulated exactly (±0.5%) from the ratios of an harmonic period, which shifts wavelength with MK−1. The formula implies this color‐constant hue cycle is isomorphic across illuminants and allows prediction of nonspectral constant hues. To identify these colorimetrically, their spectral complementary wavelengths are specified for various illuminants. This completes theglobal color constancy mechanism for the illuminant color temperature range 2800 to 25,000 K. © 2010 Wiley Periodicals, Inc. Col Res Appl, 2010

This study assessed the relative effects of metameric (preserving illumination chromaticity) vs. chromatic illumination changes on the perceived quality of real fruits. 20 normal trichromats viewed four singly-presented fruits under three neutral metameric illuminations (differing in spectral power distribution with constant CCT of ∼6500K) and two broadband chromatic illuminations (∼2000K and ∼10000K) inside a white-walled lightroom illuminated by spectrally tuneable lamps. Participants’ ratings of nine fruit attributes were used to calculate Positive Attributes Scores (PAS) for each fruit-illumination pair. For two fruits (banana and pear), PAS varied significantly under metameric illumination changes, but not under chromatic illumination changes. PAS varied significantly under both metameric and chromatic illumination changes for orange, and under chromatic illumination changes only for red apple. The findings indicate that metameric illumination changes alter perceived food attributes via changes in fruit colour appearance, while changes in illumination chromaticity may influence perceived food quality independently of fruit colour appearance which is largely maintained via colour constancy.

We investigate the intrinsic color characteristics used in accomplishing color constancy. The surface reflectance of an object can be divided into two parts: interface reflectance and body reflectance. The body component is the physical basis for determining the surface color characteristics of objects, but it is influenced by geometrical factors in measurement. To eliminate the effect of illumination and geometrical factors we adopt the normalized deviation reflectance, which was suggested by Tominaga, to describe the body reflection characteristics. Then, according to a finite-dimensional linear model, the body reflection characteristics are expressed in terms of weighting coefficients of the constrained basis functions, so as to reduce its dimensionality and make it easy to use in color classification. We choose objects made of inhomogeneous dielectric material as samples and use a grating spectrometer to measure their surface spectral reflectance, from which the description of intrinsic color characteristics can be derived based on normalized deviation reflectance and a finite-dimensional linear model. The experimental results show that this description is very stable under different conditions and also compact, so that it can serve well for color classification.