Compression-Efficient Feature Extraction Method for a CMOS Image Sensor

This article aims to improve the accuracy of real-time image recognition in the context of the Internet of Things (IoT), reduce the core network pressure of the IoT and the proportion of IoT broadband, and meet people’s demand for internet image transmission. An intelligent image fusion system based on mobile edge computing (MEC) and deep learning is proposed, which can extract the features of images and optimize the sum of intra-class distance and inter-class distance relying on the hierarchical mode of deep learning, and realize distributed computing with the edge server and base station. Through comparison with other algorithms and strategies on the text and character data sets, the effectiveness of the constructed system is verified in the performance of the algorithm and the IoT. The results reveal that the application of the unsupervised learning hierarchical discriminant analysis (HDA) has better accuracy and recall in various databases compared with conventional image recognition algorithms. When the sum intra-class and inter-class distance K is 2, the accuracy of the algorithm can be as high as 98%. The combination of MEC and layered algorithms effectively reduces the pressures of core network and shortens the response time, greatly reduces the broadband occupancy ratio. The performance of IoT is increased by 37.03% compared with the general extraction and common cloud computing. Image recognition based on the MEC architecture can reduce the amount of network transmission and reduce the response time under the premise of ensuring the recognition rate, which can provide a theoretical basis for the research and application of user image recognition under the IoT.

The integration of Internet of Things (IoT) technology into agriculture has revolutionized farming practices by using connected devices and sensors to optimize processes and facilitate sustainable execution. Because most IoT devices have limited resources, the vital requirement to efficiently manage data traffic while ensuring data security in agricultural IoT solutions creates several challenges. Therefore, it is important to study the data amount that IoT protocols generate for resource-constrained devices, as it has a direct impact on the device performance and overall usability of the IoT solution. In this paper, we present a comprehensive study that focuses on optimizing data transmission in agricultural IoT solutions with the use of compression algorithms and secure technologies. Through experimentation and analysis, we evaluate different approaches to minimize data traffic while protecting sensitive agricultural data. Our results highlight the effectiveness of compression algorithms, especially Huffman coding, in reducing data size and optimizing resource usage. In addition, the integration of encryption techniques, such as AES, provides the security of the transmitted data without incurring significant overhead. By assessing different communication scenarios, we identify the most efficient approach, a combination of Huffman encoding and AES encryption, to strike a balance between data security and transmission efficiency.

In order to explore the application of the image recognition model based on multi-stage convolutional neural network (MS-CNN) in the deep learning neural network in the intelligent recognition of commodity images and the recognition performance of the method, in the study, the features of color, shape, and texture of commodity images are first analyzed, and the basic structure of deep convolutional neural network (CNN) model is analyzed. Then, 50,000 pictures containing different commodities are constructed to verify the recognition effect of the model. Finally, the MS-CNN model is taken as the research object for improvement to explore the influence of label errors (p = 0.03, 0.05, 0.07, 0.09, 0.12) with different parameter settings and different probabilities (size of convolutional kernel, Dropout rate) on the recognition accuracy of MS-CNN model, at the same time, a CIR system platform based on MS-CNN model is built, and the recognition performance of salt and pepper noise images with different SNR (0, 0.03, 0.05, 0.07, 0.1) was compared, then the performance of the algorithm in the actual image recognition test was compared. The results show that the recognition accuracy is the highest (97.8%) when the convolution kernel size in the MS-CNN model is 2*2 and 3*3, and the average recognition accuracy is the highest (97.8%) when the dropout rate is 0.1; when the error probability of picture label is 12%, the recognition accuracy of the model constructed in this study is above 96%. Finally, the commodity image database constructed in this study is used to identify and verify the model. The recognition accuracy of the algorithm in this study is significantly higher than that of the Minitch stochastic gradient descent algorithm under different SNR conditions, and the recognition accuracy is the highest when SNR = 0 (99.3%). The test results show that the model proposed in this study has good recognition effect in the identification of commodity images in scenes of local occlusion, different perspectives, different backgrounds, and different light intensity, and the recognition accuracy is 97.1%. To sum up, the CIR platform based on MS-CNN model constructed in this study has high recognition accuracy and robustness, which can lay a foundation for the realization of subsequent intelligent commodity recognition technology.

This paper describes a two-stage recognition method that reduces the calculation load of correlation and improves recognition accuracy in statistical image recognition. It consists of an image screening and recognition stage. Image screening selects a candidate set of subimages that are similar to the object class using a lower dimensional feature vector. Since recognition is made for the selected subimages set using a higher dimensional feature vector, overall recognition efficiency is improved. The classifier in recognition designed from the selected subimages also improves recognition accuracy because selected subimages are less contaminated than the original ones. A screening criterion for measuring overall efficiency and accuracy of recognition is introduced to be exploited in designing the feature spaces of image screening and recognition. The results of experiments for the eye- and mouth-area detection in face images and text-area detection in document images show that the designed feature spaces improve recognition accuracy and more efficiency than does the conventional one-stage recognition method.

Modern cars are equipped with a wide variety of sensors generating continually growing amounts of data. This data is transmitted via bus systems such as Controller Area Network (CAN) inside of the vehicle to the microcontroller-based Electronic Control Units. By connecting the vehicle to its surroundings using wireless interfaces, this data becomes accessible to the vehicle manufacturer from a distance. Through the opening to the outside, cyber attacks can exploit these interfaces and introduce major risks to the privacy and safety of vehicle users. Hence, suitable methods for vehicle security monitoring such as intrusion detection and logging are needed. In this work, we focus on the logging of network data, since this data is useful for the development of security updates, countermeasures and incident signatures. On this account, we propose a new method to aggregate the data of the CAN bus. The method combines CAN frames into so-called flows. Each flow contains a set of packets that share a certain common attribute (e.g.: frame type and identifier). To integrate security monitoring of vehicle fleets seamlessly into backend server systems, the gathered CAN flow data is stored in an industry standard data format. Additionally, the payload data is included in the flow format using a compression algorithm to leverage deep-packet inspection. The evaluation results with realworld vehicle data indicate that in our case about 40 % reduction of the overall data size is possible with our method compared to industry-standard formats for storing CAN frames. On this account, we propose a new method to aggregate the data of the CAN bus. The method combines CAN frames into so-called flows. Each flow contains a set of packets that share a certain common attribute (e.g.: frame type and identifier). To integrate security monitoring of vehicle fleets seamlessly into backend server systems, the gathered CAN flow data is stored in an industry standard data format. Additionally, the payload data is included in the flow format using a compression algorithm to leverage deep-packet inspection. The evaluation results with realworld vehicle data indicate that in our case about 40 % reduction of the overall data size is possible with our method compared to industry-standard formats for storing CAN frames.

Transmitting massive amounts of image and audio data acquired by Internet-of-Everything (IoE) devices to data center servers for intelligent recognition processes is impractical for energy reasons, requiring in-situ processing of such data. However, algorithms accelerated by previous recognition processors [1, 2] are limited to specific applications, therefore, each IoE device may require an application-specific accelerator. On the other hand, deep convolutional neural networks (CNNs) [3] are a promising machine-learning approach, showing state-of-the-art recognition accuracy in a wide variety of applications, including both image and audio recognition. This makes CNNs a suitable candidate for a universal recognition platform for IoE devices, as described in Fig. 14.6.1. Due to the computational complexity and significant memory requirements of CNNs, a microcontroller unit (MCU) typically used for IoE devices is incapable of producing a meaningful recognition result in an energy-efficient way. Hence, the implementation of an energy-efficient CNN processor is desired to realize intelligent IoE systems.

International audience

Hyperspectral remote sensing sensors have the ability to capture a wide range of spectrum of ground objects with hundreds to thousands of bands. The obtained hyperspectral images contain more detailed spectral information than conventional panchromatic or color images. However, redundant and noisy data could also be introduced into the images and may impair further processing of the data. Removing the redundancy and noisy bands becomes one of the most important preprocessing steps of hyperspectral imaging. As a feature selection method, an attention mechanism has been successfully used in computer vision and natural language processing to enable algorithms to concentrate on the most significant data. In this article, we propose a band attention network for hyperspectral image classification. This network can automatically learn to attend to the desired band set that maximizes the classification accuracy. With careful design, the band attention framework can undertake both band weighting and band selection tasks. When working in the band weighting mode, the proposed band attention framework can learn a band weighting vector that models the relationship between all the bands. For band selection, our network can be adapted as two different types of band selection networks that select the significant bands and discard the useless bands. In both tasks, the attention and classification can be learned end to end. The experimental results prove the effectiveness and advantages of the proposed work.

This paper demonstrates, for the first time, TOMBO color imaging system that employs color-splitting filters on each lens. A red, green, or blue color filter is allocated to each microlens instead of each pixel in conventional single-sensor color imaging system. Thus the microlens array here also makes up a color filter array of Bayer geometry. For the imaging device, a CMOS image sensor with 1040 x 960 pixels whose size is 6.25 um square was used. 8 x 8 microlens array with 750 um pitch was employed as a taking lens. Excellent color images were obtained by rearrangement, interpolation, and postdigital processing.

High pixel density, efficient color splitting, a compact structure, superior quantum efficiency, and low power consumption are all important features for contemporary color-image sensors. In this study, we developed a surface plasmonics-based color-image sensor displaying a high photoelectric response, a microlens-free structure, and a zero-bias working voltage. Our compact sensor comprised only (i) a multifunctional electrode based on a single-layer structured aluminum (Al) film and (ii) an underlying silicon (Si) substrate. This approach significantly simplifies the device structure and fabrication processes; for example, the red, green, and blue color pixels can be prepared simultaneously in a single lithography step. Moreover, such Schottky-based plasmonic electrodes perform multiple functions, including color splitting, optical-to-electrical signal conversion, and photogenerated carrier collection for color-image detection. Our multifunctional, electrode-based device could also avoid the interference phenomenon that degrades the color-splitting spectra found in conventional color-image sensors. Furthermore, the device took advantage of the near-field surface plasmonic effect around the Al-Si junction to enhance the optical absorption of Si, resulting in a significant photoelectric current output even under low-light surroundings and zero bias voltage. These plasmonic Schottky-based color-image devices could convert a photocurrent directly into a photovoltage and provided sufficient voltage output for color-image detection even under a light intensity of only several femtowatts per square micrometer. Unlike conventional color image devices, using voltage as the output signal decreases the area of the periphery read-out circuit because it does not require a current-to-voltage conversion capacitor or its related circuit. Therefore, this strategy has great potential for direct integration with complementary metal-oxide-semiconductor (CMOS)-compatible circuit design, increasing the pixel density of imaging sensors developed using mature Si-based technology.

Fundus autofluorescence (FAF) has allowed in vivo mapping of retinal metabolic derangements and structural changes not possible with conventional color imaging. Incident light is absorbed by molecules in the fundus, which are excited and in turn emit photons of specific wavelengths that are captured and processed by a sensor to create a metabolic map of the fundus. Studies on the growing number of FAF platforms has shown each may be suited to certain clinical scenarios. Scanning laser ophthalmoscopes, fundus cameras, and modifications of these each have benefits and drawbacks that must be considered before and after imaging to properly interpret the images. Emerging clinical evidence has demonstrated the usefulness of FAF in diagnosis and management of an increasing number of chorioretinal conditions, such as age-related macular degeneration, central serous chorioretinopathy, retinal drug toxicities, and inherited retinal degenerations such as retinitis pigmentosa and Stargardt disease. This article reviews commercial imaging platforms, imaging techniques, and clinical applications of FAF.

New diagnostic techniques based on photodynamic medicine, such as near-infrared fluorescence using indocyanine green (NIR-ICG) and 5-aminolevulinic acid-mediated photodynamic diagnosis (ALA-PDD), are aiding navigation tasks across various fields of surgery. Specifically, NIR-ICG is being used for the intraoperative identification of sentinel lymph nodes or blood vessels in organ resection and for blood flow evaluation in surgery. These ICG-fluorescent imaging techniques could provide an additional and potentially valuable way to identify vascular and lymphatic structures in surrounding tissue. 5-Aminolevulinic acid is a precursor of a photosensitizing substance with affinity for tumors; thus, diagnostic laparoscopy using ALA-PDD in combination should improve the accuracy of detecting peritoneal dissemination in patients with advanced gastric cancer. The ability to overlay fluorescent imaging with conventional color images in real time using ALA-PDD and NIR with ICG would be of immense benefit to surgeons, providing good visualization and detection of target lesions not seen with the naked eye. A multi-center clinical study examining the safety and efficacy of ALA-PDD during laparoscopic examination for patients with advanced gastric cancer is currently underway in the form of doctor-initiated trials, and further verification studies will be conducted. Such imaging capability could have broad potential across cancer and vascular surgery.

The synthesis of holograms by computer requires the calculation of the complex amplitude emitted by an object in the hologram plane. Several approaches exist that decompose the object into primitives like points and lines. We assume that the object to be imaged can be approximately decomposed into congruent triangles. In a preprocessing step, we calculate wavefields for the triangle whose transformed copies build the object surface. The computed wavefields represent triangles rotated by different angles and positioned in different depths. The resulting wavefields are stored as conventional color images with alpha channel in a lookup table indexed by rotation angle and distance from the hologram plane. Each pixel in the texture codes a complex number. Every triangle of the input object has a corresponding entry in the lookup table. The rotation angles and the distance of the triangle determine the selection of the appropriate texture. The textures are rendered using special graphics hardware, and interference is simulated. The lookup table helps to react immediately to transformations of the input object. Texture swapping and repositioning according to the object movings lead to full- parallax hologram generation for small objects in real-time.© (2001) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Data compression involves a trade-off between delay time and data size. More the delay time, lesser the data size and vice versa. There have been many studies performed in the field of wireless sensor networks about increasing network life cycle durations that involve minimizing energy consumption by reducing data size; however, reducing data size results in increased delay time due to the added processing time required for data compression. Meanwhile, as energy generation occurs periodically in solar energy based wireless sensor networks, the redundant energy is often generated that is sufficient to run a node. In this study, the excess energy is used to reduce the delay time between nodes in a sensor network consisting of solar energy based nodes. The energy threshold value is determined by a formula based on the residual energy and charging speed. Nodes with residual energy less than the threshold, transfer data with compression in order to reduce energy consumption, and nodes with residual energy over the threshold transfer data without compression to reduce the delay time between nodes. Simulation based performance verifications show that the technique proposed in this study exhibits optimal performance in terms of both energy and delay time compared with traditional methods.

Anticipating the rise of the Internet of Things (IoT) era, we have proposed an object detection framework that employs a CMOS image sensor with binary feature extraction to reduce power requirements. Initially, we presented a lightweight deep neural network for the feature data based on the YOLOv7, comparable to the YOLOv7-tiny in the number of parameters and FLOPs, but it enhances large object recognition accuracy (APL50) by 6.6%. Moreover, our approach achieves a 48.8% reduction of GPU power consumption compared to the YOLOv7. Additionally, we introduce an on-chip signal processing method for the binary feature data. The proposed method achieves a compression rate of 64.1% and increases GPU power consumption by only 14.9% during the decoding process preceding object detection. Moreover, the size of 1-bit feature data is reduced by 96.0%, and object recognition accuracy is improved by 4.0% relative to 1-bit RGB color images.