A passive infrared gesture recognition system

As the digital devices have been developed, hand gesture recognition plays an important role in Human-Computer interface (HCI) for direct interaction without contact. Among them, the non-contact methods do not require attaching sensors to human's body and mostly use visual technologies such as Kinect or camera sensors. Various wearable devices have been developed using contact sensors such as inertial sensor, magnetic sensor, gyro sensor, electromyography (EMG), FSRs, and others to identify gestures. However, various drawbacks remain for these sensor in detecting gestures. In this paper, we propose a new method using strain gauges-based Flexible Epidermal Tactile Sensor Array to overcome the drawbacks of the existing sensors. It acquires electric signals generated according to the movements of the wrist based on strain gauge estimating the degree of deformation such as tension, compression, and twist. The proposed sensor array shows robustness, stability, and repeatability through various experiments for testing the sensor array. Furthermore, the proposed method is verified by confirming gesture recognition accuracy with multiple subjects and has a greater performance than the existing gesture recognition devices in accuracy tests.

This paper describes a miniature, wearable wireless hand- gesture recognition (HGR) system called HANDIO. It obtains its input from not only traditional inertial sensors but also a muscle tension sensor (MTS). The addition of MTS enables recognition of a much broader range of intuitive hand gestures, particularly those involving the wrist, that would otherwise be difficult to distinguish by traditional inertial-only HGRs. Among MTSs, we choose an optical MTS over the conventional surface electromyography(sEMG) for the small size, low power consumption, wearing comfort, and good detection rate. This novel miniaturized design enables the whole system to be easily patched on the wrist area or integrated into a wearable device such as a wristband or watch without extra wiring. Experimental results show that a total of 8 hand gestures involving the wrist can be recognized with a detection rate over 93%. The average power consumption of the optical sensor is only around 258μW. This versatile system can also be used to detect other joint activities such as the elbow and knee joint.

A gesture recognition system using stereo vision and arm model fitting

Low-pixel time-of-flight (ToF) sensors offer advantages in low power consumption, small size, and low cost. For gesture recognition, there are two advantages. Firstly, they are robust to changes in lighting conditions. Secondly, they could protect privacy. This paper addresses the challenge of utilizing an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{8}\times \mathbf{8}$</tex> pixel ToF sensor with extremely low resolution by proposing a signal processing system. The system includes pre-processing the data through non-global mean filtering to calibrate outliers. Additionally, we design a frames difference sliding window to ensure the integrity of gesture sequences and use Lagrangian interpolation to norm the fixed length. Finally, we design a tiny parallel convolutional network (TPC-Net) to recognize dynamic gestures. Experimental results demonstrate that the proposed system achieves an average accuracy of 98.47% over 6 dynamic gestures. Furthermore, our system achieves better performance with less model complexity compared to existing methods using low-pixel ToF sensors. These results illustrate the effectiveness of our system.

Chapter 9 - Sensors and their application

Introduction: The introduction of Clinical Information Systems (CIS) to manage and store patient information has deeply modified the workflow of caregivers. As the information is no longer stored on a physical medium, caregivers depend on the presence of a computer to access patient data. This constraint has built up an invisible barrier between the source of information and the place of its use. The popularisation of mobile devices offers an oportunity to break this barrier by providing a ubiquitous access to information. The use of a mobile platform merges the advantages of a modern CIS and the mobility offered by paper. We have developed an application on a mobile device that nurses can carry along on all their visits. It allows the information gathered during their visits to be entered directly into the system. Method: The tool we present aims to manage the daily interventions of nurses. The interventions concern all kinds of treatments provided by nurses to patients. In the current organisation, nurses start their day by printing out all the daily interventions and use it as a procedure to follow. Every time a nurse performs an intervention, she takes a note indicating that the task has been done properly. When they have some spare time, nurses enter all the gathered information in the CIS. Once nurses are equipped with mobile devices, paper becomes useless. Consequently, the process is simplified. Replacing the paper by a mobile device application is not straightforward. In order to access patient information, the tool must be linked to the existing CIS of the hospital without creating too strong dependencies. Moreover, the mobile interface must be adapted to the specificities of the mobile device such as a smaller display size, a reduced computational power, a tactile-based interaction and a real-time usage. Results: In order to link the mobile application to the existing CIS, we relied on a Service Oriented Architecture (SOA). It was mandatory to think of a software architecture that would not create a dependency with any legacy system. Thus, we defined a gateway server providing centralised access for the mobile application to any required information to or from the CIS. Integrating any mobile application would only require integrating this bridge. The gateway server also clearly separates the services that are available remotely from the ones proposed as usual Web services. The developed application not only allows the validation of the interventions and the input of measures, but it also offers a simple way to define the context in which nurses are working. This context is necessary to identify the specific list of interventions for the patients under the caregiver’s responsibility. The current CIS of the HUG includes a module that provides information concerning the interventions. Its interface is adjusted for a personal computer screen. Since caregivers are accustomed to this organisation of the information, we aimed to develop an interface on a mobile device that guarantees a certain level of homogeneity with the existing PC application. However, a mobile device’s screen is of limited size, therefore the interface has been completely rebuilt to be adapted to the new display. To deal with the limited display size, we rely on hierarchy and indexicality principles. For instance, based on the hypothesis that all tasks belonging to one category and planned at the same time can be regrouped under a single item; we regroup the interventions of similar top level in a common item. Moreover, to minimise user manipulations, relevant items are automatically moved to the top. Conclusion: The current daily workflow of nurses cannot be managed consistently using a unique tool. The need to switch from paper to computer induces work overload and is a source of errors. Therefore, we have developed a program running on a mobile device that nurses can carry during their visits. This program allows to manage nurses' daily interventions in real-time. The development of such a tool has required a complete reconstruction of the classical personal computer interface. This new interface takes into account a smaller display size, an innovative interaction paradigm, and real-time constraints.

Mobile and handheld devices are becoming an integral part of people’s work, life and entertainment. These lightweight pocket-sized devices offer great mobility, acceptable computation power and friendly user interfaces. As people are making business transactions and managing their online bank accounts via handheld devices, they are concerned with the security level that mobile devices and systems provide. In this chapter we will discuss whether these devices, equipped with very limited computation power compared to full-sized computers, can make equivalent security services available to users. We focus on the security designs and technologies of hardware, operating systems and applications for mobile and handheld devices.

Optical particle detection in liquid suspensions with a hybrid integrated microsystem

This study presents a multimodal gesture recognition system based on the fusion of visual and strain sensor data, aiming to overcome the limitations of traditional gesture recognition methods in complex environments. A flexible strain sensor made from a CNTs-Graphene-TPU composite material was developed to create a smart glove with high sensitivity and stability, capable of accurately capturing finger flexion signals. Meanwhile, by integrating deep learning algorithms, a hybrid gesture recognition framework was constructed that combines visual and strain sensor data. Experimental results show that the system maintains high recognition accuracy (96.31%) across varying lighting conditions, significantly improving gesture recognition accuracy and system robustness. The system shows promising applications, particularly for gesture interaction and sign language translation in challenging lighting environments.

With the rapid development of human-computer interaction technology, gesture recognition becomes one of the key technologies of human-computer interaction. In this paper, we propose a new method of dynamic hand gestures recognition. The method adopts the hierarchical identification model for dynamic hand gestures recognition. First, we combine frame fusion with density distribution features for rough gesture recognition, second, we use the hausdorff distance or fingertip detection for accurate gesture recognition. The main innovation of this method lies in that we change the way of dynamic gestures recognition into the recognition of static image, improves the efficiency of gesture recognition effectively. Experimental results showed that our recognition rate is above 90%.

This article reports the findings of a compressive infrared sensing approach for arm gesture acquisition and recognition. The spatial-temporal changing motion information are intrinsical clues for determining the semantics of gesture, which can be considered as a sparse spatial distribution compared with the sensing region. We first built a database of dynamic arm gestures with writing Arabic numerals 0 – 9 in a constrained interaction region, and study its sparse property of spatial distribution. Then we design a pyroelectric infrared (PIR) sensor array with random visibility modulation for compressive gesture acquisition. The semantic recognition of gesture is executed directly on the sensors' low-dimensional sequence using vector quantization (VQ) technology. The experimental results demonstrate the effectiveness of the proposed sensing method.

To address the problems of the traditional human motion gesture tracking and recognition methods, such as poor tracking effect, low recognition accuracy, high frame loss rate, and long-time cost, a dynamic human motion gesture tracking and recognition algorithm using multimode deep learning was proposed. Firstly, the collected human motion images are repaired in the three-dimensional (3D) environment, and the multimodal 3D human motion model is reconstructed using the processed images. Secondly, according to the results of model reconstruction, the camera gesture and other parameters of the keyframe are used to construct the target tracking optimization function so as to achieve the purpose of accurate tracking of human motion. Finally, for multimodal human motion gesture learning, a convolutional neural network (CNN) is developed. The trained CNN is utilized to complete dynamic human motion recognition after convolutional and pooling calculations. The results demonstrate that the proposed algorithm is effective in tracking human motion gestures. The average recognition accuracy is 96%, the average frame loss rate is 8.8%, the time cost is low, and the proposed algorithm has a high F-measure and much lower power consumption than other algorithms, indicating that the proposed algorithm is effective.

Emerging markets are always driving demand for higher performance, higher bandwidth, lower power consumption as well as increasing functionality in mobile applications. Packaging technology has become more challenging and complicated than ever before, driving advanced silicon (Si) nodes, finer bump pitch as well as finer line width and spacing (LW/LS) substrate manufacturing capabilities to satisfy the increasing requirements in the semiconductor industry. As increasing input/output (I/O) counts in a package are needed in mobile devices, packaging solutions are migrating from traditional wire bond packages to flip chip interconnect to meet these requirements. Flip chip chip scale package (fcCSP) is viewed as an attractive solution for complicated and highly integrated systems with multiple functions and heterogeneous mobile applications. Although emerging markets are driving advanced technologies in high performance mobile devices, assembly cost is still the major issue to be addressed. In addition, due to the fast growth in emerging markets for mobile applications, advanced Si node technology development for mobile applications is moving to 10nm technology (and below) and pursuing the die size reduction, efficiency enhancement and lower power consumption now. For the sake of realizing the 10nm extremely low-k (ELK) performance in a flip chip package, this paper address the technology of Mediatek innovation package (MIP) design by using copper (Cu) pillar bond-on-lead (BOL) with embedded trace substrate (ETS) to achieve low cost, finer bump pitch as finer LW/LS requirements. The 10nm chip-package interaction (CPI) study in a 15×15mm fcCSP with finer Cu pillar bump pitch of 60μm and a 2-layer ETS with finer LW/LS design is illustrated. The utilizations of 60μm bump pitch with escaped trace and mass reflow (MR), thermo-compression bonding (TCB), TCB with non-conductive paste (TCNCP), and laser assist bonding (LAB) flip chip attach process are estimated. The quick temperature cycling (QTC) test with temperature range of -40°C to 60°C is performed to realize the ELK performance in this 10nm flip chip package. Moreover, the comparison of different under bump metallization (UBM) sizes, existence of polyimide (PI) layer and reflow profiles has been also studied. With the evaluated results, not only the significant factors that impact ELK performance can be obtained, but the optimized reflow profile can also be established to enhance the yield in flip chip attach processes. It is believed that the illustrated robust flip chip attach processes examined in this paper can guarantee 10nm fine bump pitch flip chip assembly yield with less ELK damage risk in the future.

As the demands of higher performance, higher bandwidth, lower power consumption as well as multiple functions increases in mobile applications, the mobile phone has evolved from a simple communication device to a complicated and highly integrated system with multiple functions and heterogeneous devices. Due to the fast growth in emerging markets for mobile applications, packaging technology has become more challenging than ever before, driving advanced Silicon (Si) nodes, finer bump pitch as well as finer line width and spacing substrate manufacturing capabilities to satisfy the increasing requirements in mobile devices. Flip chip chip scale package (fcCSP) is viewed as an attractive solution when higher input/output (I/O) counts in a package are needed. In order to enhance the performance, reduce the power consumption and increase transition rates, three-dimensional (3D) package-on-package solution with flip chip interconnect (fcPoP) has been widely utilized to successfully achieve these goals. With the ability to stack a logic processor and low power double data rate (LPDDR) memory device in a single package, the utilization of fcPoP is becoming a preferred solution in the mobile market segment for a better power and performance balance. However, as more and more functions are designed in a chip to target the high-end mobile market, the die is becoming larger and larger. With the larger die size, it is challenging for the regular fcPoP structure to support high bandwidth top memory I/O counts in a limited package size. In order to solve the constraints of larger die and/or package size limitations, top memory with wide I/O counts as well as customized mobile memory applications, high bandwidth fcPoP technology is proposed as an strong solution. This paper reports the development of a fine pitch high bandwidth fcPoP. The top interposer substrate with copper (Cu) post interconnections peripherally is connected to bottom package. The top interposer substrate can be designed in a different top and bottom pitch to connect top mobile memory and bottom packages, respectively. With this structure, die size limitations can be overcome by using finer interconnection pitch, providing the flexibility to allow any memory interface pitch application. In addition, through this developed result, not only the package warpage/coplanarity control and reliability characterization are illustrated, it also demonstrated this enabling technology of high bandwidth fcPoP as a highly integrated, miniaturized and low profile 3D packaging solution.

Gestures are a natural means of communication between humans, and also a natural modality for human-computer interaction. Automatic recognition of gestures using computer vision is an important task in many real-world applications, such as sign language recognition, computer games control, virtual reality, intelligent homes, and assistive environments. In order for a gesture recognition system to be robust and deployable in non-laboratory settings, the system needs to be able to operate in complex scenes, with complicated backgrounds and multiple moving and skin-colored objects. In this paper we propose an approach for improving gesture recognition performance in such complex environments. The key idea is to integrate a face detection module into the gesture recognition system, and use the face location and size to make gesture recognition invariant to scale and translation. Our experiments demonstrate the significant advantages of the proposed method over alternative computer vision methods for gesture recognition.