Paper] Lightweight Object Detection Model for a CMOS Image Sensor with Binary Feature Extraction

To address the power constraints of the emerging Internet of Things (IoT) era, we propose a compression-efficient feature extraction method for a CMOS image sensor that can extract binary feature data. This sensor outputs six-channel binary feature data, comprising three channels of binarized luminance signals and three channels of horizontal edge signals, compressed via a run length encoding (RLE) method. This approach significantly reduces data transmission volume while maintaining image recognition accuracy. The simulation results obtained using a YOLOv7-based model designed for edge GPUs demonstrate that our approach achieves a large object recognition accuracy () of 60.7% on the COCO dataset while reducing the data size by 99.2% relative to conventional 8-bit RGB color images. Furthermore, the image classification results using MobileNetV3 tailored for mobile devices on the Visual Wake Words (VWW) dataset show that our approach reduces data size by 99.0% relative to conventional 8-bit RGB color images and achieves an image classification accuracy of 89.4%. These results are superior to the conventional trade-off between recognition accuracy and data size, thereby enabling the realization of low-power image recognition systems.

Metal Oxide Thin Film Transistors (MO TFTs) have garnered considerable interest in emerging Internet of Things (IoT) fields such as wearable electronics, displays, Radio Frequency Identification (RFID), and biomedical monitoring, owing to their flexibility and transparency. However, limitations in channel materials make MO TFT-based circuits unipolar. Unipolar circuits often exhibit elevated short-circuit power consumption, which restricts the development of MO TFTs in the IoT sector. This paper introduces a Capacitively Coupled Near-Threshold Biasing (CCNB) technique that leverages the unique Capacitance-Voltage (C-V) characteristics of MO TFTs to bias devices in the near-threshold region, achieving nearly a 95% reduction in power consumption compared to traditional designs with the device coupling ratio (channel capacitance/overlap capacitance) at 40. Furthermore, considering the significance of clock signals in IoT applications, we have also developed a low-power full-swing Ring Oscillator (RO) based on our CCNB technique, resulting in a 90% reduction in power consumption and a nearly 70% reduction in PDP compared to conventional low-power designs.

As the Internet of Things (IoT) continues to grow, efficient and reliable communication methods are essential for enabling widespread connectivity, particularly in remote and resource-constrained environments. LoRa, with its low power consumption and long-range capabilities, has become a cornerstone of IoT communication. The LoRaWAN protocol extends this functionality by enabling devices to transmit data globally through internet-connected gateways. These gateways aggregate data packets from nearby devices and forward them to centralized servers.However, traditional LoRaWAN gateways face significant limitations. They are often cost-prohibitive, require constant internet connectivity, and lack advanced data processing capabilities, making them unsuitable for deployment in areas with unreliable or no cellular coverage. These limitations hinder the adoption of LoRaWAN in scenarios such as environmental monitoring and rural IoT networks.To address these challenges, we developed a low-cost, energy-efficient edge-computing LoRaWAN gateway using embedded systems such as Raspberry Pi 0. Unlike conventional gateways, these edge-computing gateways locally process incoming data, enabling intelligent features such as optimized transmission, data buffering during network outages, and adaptive communication strategies. Additionally, these gateways can be configured to operate independently of cellular networks by utilizing satellite connectivity, further enhancing their usability in remote or off-grid applications.Preliminary testing demonstrates that these gateways consume up to one-third of the power required by traditional gateways while maintaining reliable data transmission. This substantial reduction in power consumption extends operational lifespans and reduces deployment costs. The ability to process and optimize data locally also improves network efficiency, ensuring timely and reliable communication even under challenging conditions.This innovation provides a scalable, cost-effective solution for IoT connectivity in remote and underserved regions. By addressing the limitations of conventional gateways, these edge-computing gateways enhance the feasibility of deploying IoT networks in scenarios where traditional infrastructure is impractical or unavailable. The broader implications of this work include improved access to IoT technologies for applications such as environmental monitoring and agriculture, ultimately expanding the reach and impact of IoT systems worldwide.

Object detection and recognition is an essential task in computer vision with numerous real-world applications such as surveillance, self-driving cars, and robotics. In recent years, deep learning-based approaches have significantly improved the accuracy and speed of object detection and recognition. The You Only Look Once version 3 (YOLOv3) algorithm is a popular deep learning-based approach that can detect and recognize objects in real-time. The Common Objects in Context (COCO) dataset is a large-scale dataset with over 330,000 labeled images and more than 2.5 million object instances, making it a popular choice for object detection and recognition tasks. In this paper, we propose a deep learning-based approach for real-time object detection and recognition using the YOLOv3 architecture and COCO dataset. We evaluate our approach based on several performance metrics, including mean average precision (mAP), frames per second (FPS), total object detection time, object detection accuracy, false positive rate, number of detected objects, and mean intersection over union (mIoU). Our results show that our approach achieves a mean average precision of 0.76 on the COCO dataset and a real-time performance of 40 frames per second on a single GPU. Additionally, our approach achieves an object detection accuracy of 93.5%, a false positive rate of 6.5%, and a mean intersection over union of 0.65. Our proposed approach shows promising results for real-time object detection and recognition and can be applied to various real-world applications.

Most mobile devices embed a CMOS image sensor (CIS) for capturing images. In addition, a variety of sensors such as proximity, ambient light, and fingerprint sensors are integrated for device control. The integration of multiple sensors in a device requires significant power consumption, area, and cost. In contrast to multiple sensors, an always-on CIS enables advanced smart sensing, including gesture sensing, face recognition, eye tracking, and so on. Smart sensing using a CIS offers a variety of user interfaces and experiences such as touch-less control, authentication, gaming, and object recognition for the Internet of Things (IOT). A major drawback of a CIS in mobile devices is that it consumes power greater than 50mW [1], and this is not feasible for always-on sensing that is required to function with the limited energy available from the device's battery. Moreover, power reduction in a CIS causes image degradation owing to reduced SNR, which is not acceptable for capturing high-quality images. Many low-power CISs have been reported [2-3]. However, they are inadequate for use as high-resolution sensors because of the requirement of additional in-pixel circuits for device operation at low supply voltages.

Internet of Things (IoT) devices face increasing security threats while operating under severe energy constraints. This research addresses the critical challenge of implementing effective cybersecurity measures within sustainable energy frameworks for resource-constrained IoT environments. The research presents Sustainable Micro-Neural Energy-Efficient Security Intelligence (SMEESI), a novel TinyML-enabled intrusion detection system (IDS) that significantly reduces energy consumption while maintaining robust security capabilities. The research innovation combines lightweight neural network architectures with energy-aware anomaly detection algorithms specifically optimized for microcontroller deployment. The SMEESI framework includes an adaptive power management module that dynamically adjusts computational intensity based on threat levels, achieving energy efficiency without compromising security posture. Performance evaluation demonstrates a 78% reduction in power consumption compared to traditional IDS implementations while maintaining 94.3% detection accuracy across multiple attack vectors. Memory footprint requirements decreased by 65%, enabling deployment on severely resource limited IoT sensors and actuators. The system has been successfully tested in smart buildings, industrial monitoring, and healthcare IoT applications, proving its versatility across critical infrastructure domains. This research contributes to green cybersecurity by enabling sustainable security monitoring in IoT ecosystems, extending device battery life, reducing electronic waste through prolonged hardware lifecycles, and minimizing the carbon footprint of security operations while maintaining essential protection against evolving cyber threats in the digital age.

Accurate identification of road object is crucial for intelligent traffic systems. However, due to the complexity of road traffic scenarios, developing efficient and accurate road object detection methods has been a challenging task. In this study, a new improved method for road object detection is proposed, named Enhanced YOLOv5 algorithm and Deep Schedule Object Detection (DSOD) algorithm. Real traffic scenes from the BDD100k dataset are used for training and testing the object detection model. The dataset consists of 9 different types of road objects in various traffic scenarios. The Mosaic data augmentation algorithm is applied to merge images in the dataset. Mean Average Precision (mAP), Precision (P), and Recall (R) are used as evaluation metrics to compare the enhanced YOLOv5 model with the most common models. Experimental results demonstrate that the DSOD algorithm achieves success in intelligent traffic object detection, significantly improving the accuracy and robustness of road object recognition. Additionally, the developed model shows significant performance improvement in accurately identifying objects in complex traffic scenes. These results suggest that the DSOD algorithm is a promising choice for intelligent road recognition and can easily adapt to different traffic scenarios. Furthermore, employing cloud computing for real-time detection meets the requirements of intelligent cooperative vehicles and enhances their visual perception capabilities.

With the advancement in artificial intelligence (AI) and microchips, we have been witnessing growing use of AI in object detection from video frames captured from Internet of thing (IoT) devices (e.g., IP cameras). From the video capturing sources to Cloud, generally, the packet forwarding delay could reach up to several hundreds of milliseconds, thereby making Cloud based objection detection as an impractical solution when the latency requirement for object detection from video frames is stringent. Embedded edge devices which are situated at the network access segment are considered to be a promising solution as they perform object detection locally and impose a negligible amount of packet forwarding delay. Nevertheless, due to lack of computation resources on such embedded edge devices, processing delay is likely to be increased as the frame rates of incoming images increase. To solve this issue, we design an adaptive queue management-based object detecting (QMOD) scheme for achieving low latency AI services with limited computation resources. We introduce a target delay requirement of AI services, and map such requirement when deciding optimal queue size so as to contain frames prior to object detection procedures. The optimal queue size, which represents a desired value at the embedded devices, is dynamically determined by the target delay requirement and weighted-average of processing time. Thus, maximum queue size is adaptively controlled, and some redundant frames are automatically dropped due to overflow of the queue. We demonstrate through experiments that the proposed QMOD scheme achieves 80%-90% reduction of latency for queuing and object detection in the commercial edge devices such as NVIDIA Xavier, Jetson TX2, and Nano, compared to the various benchmark schemes (i.e., static queue size).

Data converters significantly contribute to efficient and accurate data processing in Internet of Things (IoT) systems. As IoT expands into agriculture, industrial automation, and healthcare (AIH), precise and low-power data conversion has become crucial to support longer battery life and reliable performance in IoT devices. Efficient data converters are key to reducing energy use, especially in components like comparator circuits, which consume significant energy in successive approximation register analog-to-digital converters (SAR ADCs). This survey provides an in-depth review of recent developments in low-power data converter design, examining techniques that help reduce power consumption at various stages. It emphasizes advancements, such as energy scaling, dynamic voltage references, and architectural optimizations that enhance efficiency without compromising performance. A specific analysis of emerging technology trends, such as the application of machine learning in data converter design, is explored to stimulate further innovation. Machine learning (ML)-based optimization, including adaptive calibration, noise reduction, and real-time performance optimization, presents new opportunities for enhancing efficiency and accuracy while addressing critical design constraints in IoT applications. While quantum encryption offers promising advancements in securing IoT data transmission, a broader security perspective beyond encryption is necessary, including concerns, such as attack detection and data integrity, ensuring the robustness of IoT systems. This review also examines latency, signal integrity, and accuracy issues, offering a roadmap for next-generation converter designs and reducing power consumption in data converters, which are fundamental to enhancing the performance and lifespan of IoT devices.

A quantum efficiency model of complementary metal-oxide semiconductor image sensors based on Shockley–Read–Hall and Auger recombination is developed using the technology computer-aided design tool, and the quantum efficiency degradation after irradiation is analyzed. By simulating the surface recombination velocity and depletion region width of the photodiode, the decrease in the quantum efficiency of complementary metal-oxide semiconductor image sensors under short and long incident light wavelengths is found to be caused by the increase in the surface recombination velocity and capture of optical carriers by radiation-induced defects in the epitaxial layer, respectively. In addition, a method to reduce the quantum efficiency degradation behavior of an irradiated pixel is discussed.

Recent technology advances in CMOS image sensors (CIS) enable their utilization in the most demanding of surveillance fields, especially visual surveillance and intrusion detection in intelligent surveillance systems, aerial surveillance in war zones, Earth environmental surveillance by satellites in space monitoring, agricultural monitoring using wireless sensor networks and internet of things and driver assistance in automotive fields. This paper presents an overview of CMOS image sensor-based surveillance applications over the last decade by tabulating the design characteristics related to image quality such as resolution, frame rate, dynamic range, signal-to-noise ratio, and also processing technology. Different models of CMOS image sensors used in all applications have been surveyed and tabulated for every year and application.

In the era of Artificial Intelligence (AI), bio-inspired perceptual computing system design brings favorable opportunities, while still facing considerable challenges in the meantime. Especially for tasks of image recognition in power-limited vision-based Internet of Things (IoT) devices, energy constraints due to the end of Dennard scaling limit the performance of Neural Network (NN) algorithms on popular digital platforms, which would not reach the energy efficiency requirement for embedded AI applications. In this paper, a processing near sensor architecture in mixed-signal domain with CMOS Image Sensor (CIS) of convolutional-kernel-readout method is proposed. Visual data is collected from a smart CIS, which can realize maximum 5 × 5 kernel-readout with minimum one slide step for convolutional operations. The outputs of CIS are directly processed by analog processing units locating near CIS without the constraint of digital clock and bottleneck of Analog-to-Digital Converter (ADC). By analyzing the effects of analog noise on classification accuracy, we further evaluate the fault-tolerance of the system to circuit noise and the device imperfection, such as mismatch and process variation. A mixed-signal visual perception chip is fabricated with a 32 × 32 image sensor and a Binarized Neural Network (BNN) processing array integrated with SMIC 180nm standard CMOS mixed-signal process. Measurement results show up to 545.4 GOPS/W energy efficiency with 1.8mW power consumption taking the advantages of ADC-free processing architecture. This work provides a promising alternative for low-power vision-based IoT intelligent applications.

As the core technologies represented by Internet of Things (IoT) technology and computer vision technology booming, object detection algorithms based on deep learning have received high attention in various fields. Object detection is not only a core problem in the field of computer vision, but also a prerequisite and basis for many computer vision tasks, and has important applications in the fields of autonomous driving and video surveillance. The task of object detection is to find all the targets of interest in an image and determine their category and location. Object detection includes two tasks: object classification and object localization. However, object classification focuses on the most discriminative part of the feature map, and object localization requires a feature map focusing on the whole region of the object. Therefore, in order to improve the detection accuracy, we propose a CBAM-RetinaNet network model.CBAM (Convolutional Block Attention Module) is a lightweight convolutional attention module, which combines channel and spatial attention mechanism modules. The channel attention module focuses on the meaningful information in the input image, which would lead to an increase in the accuracy of object classification. The spatial attention module focuses on the location information of the object, which may improve the accuracy of object localization. The experiment results show that the proposed achieves better object detection performance on the PASCAL VOC2007 dataset, and the accuracy is 2% higher than that of the RetinaNet model, which has good experimental results.

Photodetectors are essential optoelectronic devices that play a critical role in modern technology by converting optical signals into electrical signals, which are one of the most important sensors of the informational devices in current ‘Internet of Things’ era. Two-dimensional (2D) material-based photodetectors have excellent performance, simple design and effortless fabrication processes, as well as enormous potential for fabricating highly integrated and efficient optoelectronic devices, which has attracted extensive research attention in recent years. The introduction of spontaneous polarization ferroelectric materials further enhances the performance of 2D photodetectors, moreover, companying with the reduction of power consumption. This article reviews the recent advances of materials, devices in ferroelectric-modulated photodetectors. This review starts with the introduce of the basic terms and concepts of the photodetector and various ferroelectric materials applied in 2D photodetectors, then presents a variety of typical device structures, fundamental mechanisms and potential applications under ferroelectric polarization modulation. Finally, we summarize the leading challenges currently confronting ferroelectric-modulated photodetectors and outline their future perspectives.

Digital image sensors have been utilized in various applications including, but not limited to, consumer electronics, machine vision, and security surveillance. In the early days of digital cameras, charge-coupled-device (CCD) technology was dominant because of its superior imaging quality. In recent years, with the advancement of complementary-metal-oxide-semiconductor (CMOS) technology, CMOS image sensors have prevailed due to their lower power consumption and inexpensive fabrication cost. The design of commercial image sensors focuses on the improvement of imaging quality by increasing spatial resolution with smaller pixel size. When sensors without an on-chip computation capability are applied to the computer vision field, they produce massive amounts of raw image data. To extract valuable image features, such redundant raw data has to be transmitted and then processed by the vision processing system. This process wastes tremendous hardware resources and causes significant power consumption. Therefore, there is a growing demand for the smart image sensor that can perform on-chip image processing to directly export valuable image features such as spatial contrast or temporal motion. In many computer vision applications, motion features are widely used to identify active objects from stationary backgrounds. Once active regions with motion activities are extracted, more advanced processing such as image compression, object segmentation, and pattern recognition can be implemented on the acquired images. Visual motion detection can be achieved by measuring light intensity changes along with time on every pixel. Light intensities can be transformed into electrical signals in the form of currents, charges, or voltages. By detecting temporal variations in these signals with dedicated circuits, a motion-detection function can be implemented on image sensors. Various smart image sensors with on-chip motion feature-extraction capabilities have been reported in recent years. They can be classified into two main categories: “discrete mode” and “continuous mode”. In the first mode, an analog intensity image is acquired by integrating incident light within a certain period. A comparison of two consecutive frames can realize the motion-detection function. However, this algorithm suffers from a severe aliasing effect due to the discrete differentiation between two separate frames. In the “continuous mode”, incident light is directly converted into photo currents or intensity voltages without the integrating operation. By applying continuous differentiation on these signals, motion features can be extracted more efficiently. In order to report motion activities in real time, researchers innovatively developed an asynchronous address-event representation (AER) strategy to export motion address events from smart image sensors to post processors. Unfortunately, image processing on address events is different with the conventional frame-based strategy because of their asynchronous nature, and specific event-based algorithms have to be customized based on applications. The aim of this work is to design a smart CMOS image sensor that can extract and process motion features on the same chip so as to simplify the entire visual system with minimal computation burden. In order to achieve this ultimate goal, several smart motion-detection image sensors have been developed. The first image sensor was designed based on the temporal difference algorithm to on-chip export motion features, and this sensor is a compact visual system that integrates motion detection and feature processing on the same chip. In order to maximize photon sensitivity in every pixel, a feature-extraction image sensor based on an emerging 3D integration technology was developed. The sensor can extract either temporal motion or spatial contour in real time. Also, an event-clustering image sensor specialized for object tracking applications was proposed. This sensor continuously detects motion events and on-chip processes them in parallel to export specific “event-flow” features. Simulation results show that this image sensor significantly simplifies and accelerates visual object tracking systems.