A Dual-Mode 303-Megaframes-per-Second Charge-Domain Time-Compressive Computational CMOS Image Sensor.

Digital image sensors have been utilized in various applications including, but not limited to, consumer electronics, machine vision, and security surveillance. In the early days of digital cameras, charge-coupled-device (CCD) technology was dominant because of its superior imaging quality. In recent years, with the advancement of complementary-metal-oxide-semiconductor (CMOS) technology, CMOS image sensors have prevailed due to their lower power consumption and inexpensive fabrication cost. The design of commercial image sensors focuses on the improvement of imaging quality by increasing spatial resolution with smaller pixel size. When sensors without an on-chip computation capability are applied to the computer vision field, they produce massive amounts of raw image data. To extract valuable image features, such redundant raw data has to be transmitted and then processed by the vision processing system. This process wastes tremendous hardware resources and causes significant power consumption. Therefore, there is a growing demand for the smart image sensor that can perform on-chip image processing to directly export valuable image features such as spatial contrast or temporal motion. In many computer vision applications, motion features are widely used to identify active objects from stationary backgrounds. Once active regions with motion activities are extracted, more advanced processing such as image compression, object segmentation, and pattern recognition can be implemented on the acquired images. Visual motion detection can be achieved by measuring light intensity changes along with time on every pixel. Light intensities can be transformed into electrical signals in the form of currents, charges, or voltages. By detecting temporal variations in these signals with dedicated circuits, a motion-detection function can be implemented on image sensors. Various smart image sensors with on-chip motion feature-extraction capabilities have been reported in recent years. They can be classified into two main categories: “discrete mode” and “continuous mode”. In the first mode, an analog intensity image is acquired by integrating incident light within a certain period. A comparison of two consecutive frames can realize the motion-detection function. However, this algorithm suffers from a severe aliasing effect due to the discrete differentiation between two separate frames. In the “continuous mode”, incident light is directly converted into photo currents or intensity voltages without the integrating operation. By applying continuous differentiation on these signals, motion features can be extracted more efficiently. In order to report motion activities in real time, researchers innovatively developed an asynchronous address-event representation (AER) strategy to export motion address events from smart image sensors to post processors. Unfortunately, image processing on address events is different with the conventional frame-based strategy because of their asynchronous nature, and specific event-based algorithms have to be customized based on applications. The aim of this work is to design a smart CMOS image sensor that can extract and process motion features on the same chip so as to simplify the entire visual system with minimal computation burden. In order to achieve this ultimate goal, several smart motion-detection image sensors have been developed. The first image sensor was designed based on the temporal difference algorithm to on-chip export motion features, and this sensor is a compact visual system that integrates motion detection and feature processing on the same chip. In order to maximize photon sensitivity in every pixel, a feature-extraction image sensor based on an emerging 3D integration technology was developed. The sensor can extract either temporal motion or spatial contour in real time. Also, an event-clustering image sensor specialized for object tracking applications was proposed. This sensor continuously detects motion events and on-chip processes them in parallel to export specific “event-flow” features. Simulation results show that this image sensor significantly simplifies and accelerates visual object tracking systems.

One of the most dramatic demonstrations of the continuing advancement in VLSI has been the solid-state image sensor. Since the first array demonstration in 1967, the increase in pixel density in image sensors has matched the increase in cell density in dynamic memories. Today, image sensors with densities of 360,000 elements are available for consumer applications, and sensors with well over a million elements have been developed for government and scientific applications. Although electronic imaging has dominated the commercial and government market for many years, widespread application in consumer products has been limited by the high cost and low resolution of solid-state sensors as compared to vidicon-type tubes. Now, however, the cost of image sensors for television applications has fallen to a point at which it is attractive to design consumer products such as video cameras using solid-state image sensors. As a result, image sensors have entered the consumer marketplace as well. We discuss some of the advances made at Eastman Kodak Company in two areas: very high-speed image sensors and very high-density color image sensors. A brief introduction to image sensor architecture is presented, followed by descriptions of a 2000-frame/second sensor for high-frame-rate imaging and a 360,000-pixel color sensor for imaging photographic negatives.

An improved way of using the traditional Hartmann test has been evolved to measure spherical aberrations of photographic lenses. Instead of photographic plates, a solid-state linear image sensor was successfully used to achieve improvements in accuracy and in measuring efficiency. It was found possible to locate the intersection of a ray with the image plane in a few seconds, with an error of less than 0.3 microm. Two standard lenses of known data were measured to compare the results of the experiments and the calculations. The precision of the method is briefly discussed theoretically and experimentally.

The practical boundaries surrounding the design of very high resolution image sensors have been studied. The case study used to analyze these practical boundaries is a CMOS photodiode active pixel sensor (APS) image sensor with pixel array format of 3,072 (H) X 2,048 (V). The frame rate of the image sensor is variable up to 30 frames per second (fps), leading to a maximum image data throughput of 180 M pixels per second. The pixel size is 6.0 μm, resulting in a pixel fill factor of approximately 48% (implemented in a 0.25 μm CMOS fabrication process) and a 4/3 inch optical format. The resultant die fill factor is approximately 54%. The column-parallel approach, which works well for both the on-chip analog signal processing and analog-to-digital conversion, is adopted. The 10-bit successive approximation ADC was deemed suitable for on-chip integration. The projected total power consumption of the case study image sensor chip is below 200 mW at 3.3-V power supply and below 100 mW at 1.5-V power supply. These power estimates were made for operation at full resolution (6 M pixels per frame) and at maximum frame rate (30 fps), leading to a maximum digital image data throughput of 1.8 G bits per second.

In this paper, a high-speed CMOS image sensor having a new architecture and a new operating principle has been developed. The image sensor achieves both the continuous capturing and the burst capturing by a single chip, and has low power consumption, low heat generation, high sensitivity and high S/N ratio. This image sensor consist of mainly four blocks, two dimensional pixel array of 4-transister CMOS active pixel, analog memory arrays connected with each pixel output line independently to the pixel array, scanning circuits and multiple number of output amplifiers. A prototype image sensor was fabricated using a 0.18μm 2-Poly 3-Metal CMOS technology with the die size of 5550 μm^H x 4575 μm^V, the pixel size of 48 μm^H x 48 μm^V, the number of pixels of 72^H x 32^V, the number of analog memories of 104 memories per pixel and the 6 parallel horizontal output circuits and output amplifiers. The aperture ratio is 35% and the conversion gain is 60 μV/e-(input referred). It has been confirmed that this image sensor achieves 10,000,000 fps during burst capturing mode and 10,000 fps during the continuous capturing mode through the image capture experiments of high speed phenomena such as rotating object and discharge phenomenon.

In this paper, a CMOS image sensor using floating capacitor load readout operation has been discussed. The floating capacitor load readout operation is used during pixel signals readout. And this operation has two features: 1. in-pixel driver transistor drives load capacitor without current sources, 2. parasitic capacitor of pixel output vertical signal line is used as a sample/hold capacitor. This operation produces three advantages: a smaller chip size, a lower power consumption, and a lower output noise than conventional CMOS image sensors. The prototype CMOS image sensor has been produced using 0.18 μm 1-Poly 3-Metal CMOS process technology with pinned photodiodes. The chip size is 2.5 mmH x 2.5 mmV, the pixel size is 4.5 μmH x 4.5 μmV, and the number of pixels is 400H x 300V. This image sensor consists of only a pixel array, vertical and horizontal shift registers, column source followers of which height is as low as that of some pixels and output buffers. The size of peripheral circuit is reduced by 90.2 % of a conventional CMOS image sensor. The power consumption in pixel array is reduced by 96.9 %. Even if the power consumption of column source follower is included, it reduced by 39.0 %. With an introduction of buried channel transistors as in-pixel driver transistors, the dark random noise of pixels of the floating capacitor load readout operation CMOS image sensor is 168 μVrms. The noise of conventional image sensor is 466 μVrms; therefore, reduction of 63.8 % of noise was achieved. © (2013) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.

This paper presents a low noise CMOS image sensor using conventional 3T active pixel with Nwell/Psub diode as photo detector. Both fixed pattern noise (FPN) and temporal noise are suppressed by the proposed digital correlated multiple sampling (DCMS) technology. FPN and temporal noise from pixel, buffer circuit, and column-parallel ADC are analyzed in detail, and the total noise with DCMS is derived. General expression of 1/f noise with correlated multiple sampling is given, illustrating impact of delay time in DCMS. Output noise of image sensor, frame rate, power, and area are affected by order and oversampling ratio of sigma–delta ADC, which are discussed for practical design. A prototype CMOS image sensor with $800\times 600$ pixel array and second-order incremental sigma–delta ADCs is implemented with the 0.35- $\mu \text{m}$ standard CMOS process. Measurement results of the implemented image sensor show a column FPN of 0.009%, an input referred noise of 3.5 $\text{e}^{-}_{\mathrm {\mathbf {rms}}}$ , and a dynamic range of 84 dB with oversampling ratio of 255. This indicates that image sensor with low noise can be achieved by DCMS without the CIS process and column amplification.

This paper presents a modulation scheme in the time domain based on On-Off-Keying and proposes various compatible supports for different types of image sensors. The content of this article is a sub-proposal to the IEEE 802.15.7r1 Task Group (TG7r1) aimed at Optical Wireless Communication (OWC) using an image sensor as the receiver. The compatibility support is indispensable for Image Sensor Communications (ISC) because the rolling shutter image sensors currently available have different frame rates, shutter speeds, sampling rates, and resolutions. However, focusing on unidirectional communications (i.e., data broadcasting, beacons), an asynchronous communication prototype is also discussed in the paper. Due to the physical limitations associated with typical image sensors (including low and varying frame rates, long exposures, and low shutter speeds), the link speed performance is critically considered. Based on the practical measurement of camera response to modulated light, an operating frequency range is suggested along with the similar system architecture, decoding procedure, and algorithms. A significant feature of our novel data frame structure is that it can support both typical frame rate cameras (in the oversampling mode) as well as very low frame rate cameras (in the error detection mode for a camera whose frame rate is lower than the transmission packet rate). A high frame rate camera, i.e., no less than 20 fps, is supported in an oversampling mode in which a majority voting scheme for decoding data is applied. A low frame rate camera, i.e., when the frame rate drops to less than 20 fps at some certain time, is supported by an error detection mode in which any missing data sub-packet is detected in decoding and later corrected by external code. Numerical results and valuable analysis are also included to indicate the capability of the proposed schemes.

Image sensors are widely used in various applications. With the increasing requirement for high resolutions and frame rates, power consumption has become a critical issue, which limits the use of image sensors in mobile devices and IoT applications. Compressive sensing (CS) techniques can achieve a sub-Nyquist sampling rate to reduce the power consumption in hardware circuits. Currently, most compressive measurements are implemented in digital CMOS circuits, leading to high hardware complexity and power consumption, as well as the limited sampling speed. Furthermore, CS applications with large image sizes are usually based on block-wise methods, which require real-time rate controls during practical operations. In this article, we propose a memristor-based CS encoder that can be integrated with conventional image sensors to achieve high performance with low power consumption and hardware overheads. A self-adaptive compressing rate control mechanism is also devised to maximize the performance of the proposed technique. Simulation results of wireless video streaming demonstrate the advantages of the proposed technique.

The time resolution of charge modulation in CMOS image sensors has entered the sub-nano second regime and is still reducing toward tens of pico-second. The lateral electric field modulators (LEFM) invented at Shizuoka University has significantly contributed to the recent progress in the solid-state time-resolved imaging field. Based on the LEFM technology, we are developing ultra-high-speed CMOS image sensors whose frame rate or time resolution is determined only by the charge modulation speed. In this presentation, the concept, architecture, example of implementation, and demonstration of 200Mfps single-shot video capturing based on our scheme are shown.

Electronic imaging systems based on solid-state image sensors have received tremendous attention in the past few years. The many advantages of the solid-state image sensors have opened up a wide variety of new imaging possibilities. Applications, ranging from high resolution, spaceborne electronic camera to PC peripheral camera, were costly, it not impossible, using the earlier technology. This paper will describe the technologies of charge-coupled device (CCD) and CMOS image sensors. A review of the recent advancements and industry trends will be given. Challenges in sensor performance, device scaling, and fabrication processes will also be discussed. The paper will conclude with a description of some applications and opportunities for the solid-state image sensors.

Commercial high-resolution optical microscopes are essential for microscopy imaging; however, they are expensive and bulky, which limits their use in point-of-care devices, resource-limited areas, and real-time imaging of a sample in a large apparatus. In this study, we report a novel compact (10 cm × 5 cm × 5 cm, without the light source) lightweight (∼0.5 kg) submicron-resolution inverted optical microscope at low cost (∼$ 300). Our technique utilises the proximity of the image sensor to a commercial microscope objective lens for compactness of the microscope. The use of an image sensor with a small pixel size helps to reduce the information loss, which provides high-resolution images. Moreover, our technique offers a freedom to tailor the design of microscope according to the required resolution, cost, and portability for specific applications, which makes it a suitable candidate for affordable point-of-care devices. Images of several micron-to-submicron scale patterns and spherical beads are acquired to observe the resolution and quality of the images obtained using our microscope. In addition, we demonstrate the applications of our microscope in various fields such as recording of high-speed water microdroplet formation inside a microfluidic device, high-resolution live cell imaging inside an incubator, and real-time imaging of crack propagation in a sample under stretching by a material testing system (MTS). Therefore, this portable and inexpensive microscope provides the essential functionalities of a bulky expensive high-performance microscope at a lower cost. LAY DESCRIPTION: Microscope is an essential tool in research allowing for observation of microsized objects and life forms. Contemporary commercial high-resolution microscopes have long optical paths involving series of lenses and filters. Although this configuration precisely corrects for optical distortions and produces clear images, it makes modern microscopes very costly and bulky, restricting their usage to low-funded research laboratories and at remote places. We have developed a simple digital microscope with high-resolution but with much smaller size and lighter in weight at low cost by removing the long optical terrain. Our microscope consists of a commercial microscope objective lens for magnification and semiconductor image sensor with small pixels placed right after the lens, both of which are affordable and easily available. The small pixel size helps to translate the magnified analogue sample image to high-resolution digital image. In our paper, we show that our microscope can view micro and submicron-sized patterns and beads. Moreover, our fist-sized microscope can be placed inside an incubator for real-time imaging of cells or rotated sideways for recording submicron-sized crack generation due stretching of novel materials, both of which could not be accomplished with the 2 feet tall laboratory microscopes.

Reliable and stable visual perception systems are needed for humanoid robotic assistants to perform complex grasping and manipulation tasks. The recognition of the object and its precise 6D pose are required. This paper addresses the challenge of detecting and positioning a textureless known object, by estimating its complete 6D pose in cluttered scenes. A 3D perception system is proposed in this paper, which can robustly recognize CAD models in cluttered scenes for the purpose of grasping with a mobile manipulator. Our approach uses a powerful combination of two different camera technologies, Time-Of-Flight (TOF) and RGB, to segment the scene and extract objects. Combining the depth image and gray image to recognize instances of a 3D object in the world and estimate their 3D poses. The full pose estimation process is based on depth images segmentation and an efficient shape-based matching. At first, the depth image is used to separate the supporting plane of objects from the cluttered background. Thus, cluttered backgrounds are circumvented and the search space is extremely reduced. And a hierarchical model based on the geometry information of a priori CAD model of the object is generated in the offline stage. Then using the hierarchical model we perform a shape-based matching in 2D gray images. Finally, we validate the proposed method in a number of experiments. The results show that utilizing depth and gray images together can reach the demand of a time-critical application and reduce the error rate of object recognition significantly.

Time-of-flight (ToF) sensors using light pulses or continuous waves allow accurate distance measurements. Three-dimensional imagers can be based on an array of timing or time-gated integration pixels. Single-photon avalanche diodes (SPADs) have been increasingly chosen as the pixel's photodetector device to develop fast, long-range ToF sensors. Solid-state ToF cameras are replacing other alternatives, showing attractive characteristics and bringing up new potential applications. This paper presents the technical evolution of SPAD ToF 3D imaging sensors, and provides insight into their development over the last few decades. Starting with the first SPAD imagers reported in the early 2000's, various direct and indirect arrays up to present day state-of-the-art prototypes are referenced. The existing methods, options and possible implementations are described, addressing their advantages and drawbacks, and showing the progress yet to be made. The performance of the different presented approaches are given and compared.

Although solid-state image sensors are known to develop defects in the field, little information is available about the nature, quantity or development rate of these defects. We report on and algorithm and calibration tests, which confirmed the existence of significant quantities of in-field defects in 4 out of 5 high-end digital cameras. Standard hot pixels were identified in all 4 cameras. Stuck hot pixels, which have not been described previously, were identified in 2 cameras. Previously, hot-pixels were thought to have no impact at short exposure durations, but the large offset of stuck hot pixels will degrade almost any image and cannot be ignored. Fully-stuck and abnormal sensitivity defects were not found. Spatial investigation found no clustering. We tracked hot pixel growth over the lifetime of one camera, using only normal photographs. We show that defects develop continually over the lifetime of the sensor, starting within several months of first use, and do not heal over time. Our success in tracing the history of each defect confirms the feasibility of using automatic defect identification to analyze defect response and growth characteristics in a multitude of cameras already in the field, without performing additional experiments or requiring physical access to the cameras.