Rapid-Motion-Track: Markerless tracking of fast human motion with deep learning

Optical accuracy is a primary driver of parabolic trough concentrating solar power (CSP) plant performance, but can be damaged by wind loads, gravity, error during installation, and regular plant operation. Collecting and analyzing optical measurements over an entire operating parabolic trough plant is difficult, given the large scale of typical installations. Distant Observer, a software tool developed at the National Renewable Energy Laboratory, uses images of the absorber tube reflected in the collector mirror to measure both surface slope in the parabolic mirror and offset of the absorber tube from the ideal focal point. This technology has been adapted for fast data collection using low-cost commercial drones, but until recently still required substantial human labor to process large amounts of data. A new method leveraging advanced deep learning and computer vision tools can drastically reduce the time required to process images. This new method addresses the primary analysis bottleneck, identifying featureless, reflective mirror corner points to a high degree of accuracy. Recent work has shown promising results using computer vision methods. The combined deep learning and computer vision approach presented here proved highly effective and has the potential to further automate data collection and analysis, making the tool more robust. The method presented in this paper automatically identified 74.3% of mirror corners within 2 pixels of their manually marked counterparts and 91.9% within 3 pixels. This level of accuracy is sufficient for practical Distant Observer analysis within a target uncertainty. A commercial drone collected video of over 100 parabolic trough modules at an operating CSP plant to demonstrate the deep learning and computer vision method’s usefulness in processing large amounts of data. These troughs were successfully analyzed using Distant Observer, paired with the new deep learning and computer vision algorithm, and can provide plant operators and trough designers with valuable insight about plant performance, operating strategies, and plant-wide optical error trends.

Разработано программное обеспечение ImgOpinion, которое позволяет выполнять оптико-структурный анализ цифрового изображения объекта методами компьютерного зрения. Оно предназначено для использования в экспертных лабораториях в составе аппаратно-программного комплекса компьютерного зрения MS-Unit, который включает специализированный осветитель “Фотобокс 3138” с цифровой регистрирующей камерой. Выходные данные строятся в виде цифрового паспорта огнестрельного повреждения, в котором приводятся криминалистически значимые характеристики объекта исследования, а также выявленных на нем следов выстрела. The ImgOpinion software, capable to perform the optical and structural analysis of a digital image of the object by computer vision methods, was developed. It can be employed in expert laboratories as a part of the computer vision hardware-software complex MS-Unit together with specialized illuminator “Photobox 3138” with a digital recording camera. The output data is constructed in the form of a digital passport of the gunshot residue, which contains forensically significant characteristics of the object under investigation, as well as a gunshot residue detected on it. Purpose. The research has solved the problem of creating a specialised hardware and software complex for forensic examination, the use of which partially automates the processes of identification and quantitative characterisation of gunshot traces and products. Methodology. Non-destructive research methods are implemented in computer vision systems, which include two main stages: a digital image acquisition and its mathematical processing. The hardware and software complex of a computer vision usually consists of an illuminator, a digital recording video camera and a specialised program for processing the raw data. Findings. ImgOpinion software has been developed, which allows performing the optical-structural analyses of digital images using the computer vision methods and is intended for use in expert laboratories as part of the MS-Unit computer vision hardware and software complex, which includes a specialized illuminator “Photobox 3138” with a digital recording camera. Autonomous and portable device has a mobile design and provides white light with colour temperature 5000 K (CIE D50) and colour rendering index CRI = 97+. Six independently switched LED monochrome illuminators with narrow spectral bands from 365 to 880 nm (UV to IR spectral bands) provide the uniform illumination without extraneous stray light on the working field (light intensity drop below 2 % at the edges of the working field). Value. The ImgOpinion output is constructed in the form of a digital passport of a gunshot injury, which contains forensically relevant characteristics of the object of investigation. The digital passport is an adaptation of the forensically significant information on the firearm traces for the integration into specialized databases for further development of the automation of expert evaluation processes, conducting the automated trace comparison and the determination of incident circumstances by it.

Benchmark datasets used for testing computer vision (CV) methods often contain little variation in illumination. The methods that perform well on these datasets have been observed to fail under challenging illumination conditions encountered in the real world, in particular, when the dynamic range of a scene is high. The authors present a new dataset for evaluating CV methods in challenging illumination conditions such as low light, high dynamic range, and glare. The main feature of the dataset is that each scene has been captured in all the adversarial illuminations. Moreover, each scene includes an additional reference condition with uniform illumination, which can be used to automatically generate labels for the tested CV methods. We demonstrate the usefulness of the dataset in a preliminary study by evaluating the performance of popular face detection, optical flow, and object detection methods under adversarial illumination conditions. We further assess whether the performance of these applications can be improved if a different transfer function is used.

Traditional supervised learning methods have limitations in labeling speed, scene adaptability and accuracy.Unsupervised learning methods do not require labeling data and can automatically extract the laws, which provides a new idea for image segmentation, especially in medical diagnosis, automatic driving and other highprecision requirements of the scene has an important application value.This study explores unsupervised learningbased image segmentation methods in computer vision, focusing on the improved kernel fuzzy C-mean clustering algorithm (KFCM).The study constructs an image segmentation algorithm with noise robustness by introducing a kernel function instead of Euclidean distance and combining it with super-pixel segmentation technique.The experiments are validated on synthetic, natural and medical images and compared with various classical algorithms.The results show that when 30% Gaussian noise is added to the synthetic image, the segmentation accuracy of the KFCM algorithm reaches 99.8%, which is 12.6% higher than that of the traditional FCM; in the segmentation of the natural image with the addition of mixed noise, the average value of the segmentation coefficient of the KFCM reaches 96.45%, which is 17.47% higher than that of the FCM, and the segmentation entropy is reduced by 34.57%; and in the segmentation of the medical cell image, the KFCM algorithm shows good edge keeping ability in complex noise environment.The study shows that the improved KFCM algorithm significantly improves the image segmentation accuracy and anti-noise performance through the adaptive neighborhood information and kernel mapping, and provides an effective solution for unsupervised image segmentation, which is of practical application value for medical diagnosis, automatic driving and other fields.

Automated computer vision methods and tools offer new ways of analysing audio-visual material in the realm of the Digital Humanities (DH). While there are some promising results where these tools can be applied, there are basic challenges, such as algorithmic bias and the lack of sufficient transparency, one needs to carefully use these tools in a productive and responsible way. When it comes to the socio-technical understanding of computer vision tools and methods, a major unit of sociological analysis, attentiveness, and access for configuration (for both computer vision scientists and DH scholars) is what computer science calls “ground truth”. What is specified in the ground truth is the template or rule to follow, e.g. what an object looks like. This article aims at providing scholars in the DH with knowledge about how automated tools for image analysis work and how they are constructed. Based on these insights, the paper introduces an approach called “active learning” that can help to configure these tools in ways that fit the specific requirements and research questions of the DH in a more adaptive and user-centered way. We argue that both objectives need to be addressed, as this is, by all means, necessary for a successful implementation of computer vision tools in the DH and related fields.

Biological structures are defined by rigid elements, such as bones, and elastic elements, like muscles and membranes. Computer vision advances have enabled automatic tracking of moving animal skeletal poses. Such developments provide insights into complex time-varying dynamics of biological motion. Conversely, the elastic soft-tissues of organisms, like the nose of elephant seals, or the buccal sac of frogs, are poorly studied and no computer vision methods have been proposed. This leaves major gaps in different areas of biology. In primatology, most critically, the function of air sacs is widely debated; many open questions on the role of air sacs in the evolution of animal communication, including human speech, remain unanswered. To support the dynamic study of soft-tissue structures, we present a toolkit for the automated tracking of semi-circular elastic structures in biological video data. The toolkit contains unsupervised computer vision tools (using Hough transform) and supervised deep learning (by adapting DeepLabCut) methodology to track inflation of laryngeal air sacs or other biological spherical objects (e.g., gular cavities). Confirming the value of elastic kinematic analysis, we show that air sac inflation correlates with acoustic markers that likely inform about body size. Finally, we present a pre-processed audiovisual-kinematic dataset of 7+ hours of closeup audiovisual recordings of siamang (Symphalangus syndactylus) singing. This toolkit (https://github.com/WimPouw/AirSacTracker) aims to revitalize the study of non-skeletal morphological structures across multiple species.

Techniques for measuring the position and orientation of an object from corresponding images are based on the principles of epipolar geometry in the computer vision and photogrammetric fields. Contributing to their importance, many different approaches have been developed in computer vision, increasing the automation of the pure photogrammetric processes. The aim of this paper is to evaluate the main differences between photogrammetric and computer vision approaches for the pose estimation of an object from image sequences, and how these have to be considered in the choice of processing technique when using a single camera. The use of a single camera in consumer electronics has enormously increased, even though most 3D user interfaces require additional devices to sense 3D motion for their input. In this regard, using a monocular camera to determine 3D motion is unique. However, we argue that relative pose estimations from monocular image sequences have not been studied thoroughly by comparing both photogrammetry and computer vision methods. To estimate motion parameters characterized by 3D rotation and 3D translations, estimation methods developed in the computer vision and photogrammetric fields are implemented. This paper describes a mathematical motion model for the proposed approaches, by differentiating their geometric properties and estimations of the motion parameters. A precision analysis is conducted to investigate the main characteristics of the methods in both fields. The results of the comparison indicate the differences between the estimations in both fields, in terms of accuracy and the test dataset. We show that homography-based approaches are more accurate than essential-matrix or relative orientation–based approaches under noisy conditions.

There has been much research on computer vision in last three decades. Computer vision methods have been developed for different situations. One example is a detection of human face. For computers face detection is hard. Faces look different from different viewing directions. Facial expressions affect to the look of the face. Each individual person has a unique face. The lightning conditions can vary and so on.However, face detection is currently possible in limited conditions. In addition, there are some methods that can be used for gender recognition [3], face recognition [5] and facial expression recognition [2].Nonetheless, there has been very little research on how to combine these methods. There has also been quite little research on how to apply these methods in human-computer interaction (HCI).Finding sets of techniques that complement each other in a useful way is one research challenge. There are some applications that take advantage of one or two computer vision techniques. For example, Christian and Avery [1] have developed an information kiosk that uses computer vision to detect potential users from a distance. A similar kiosk has been developed by us in the University of Tampere [4]. There are also some games that use simple computer vision techniques for the interaction. However, there are very few applications that use several computer vision techniques together such as face detection, facial expression recognition and gender recognition. Overall, there has been very little effort in combining different techniques.In my research I develop computer vision methods and combine them, so that the combined method can detect face, recognize gender and facial expressions. After successfully combining the methods, it is easier to develop HCI applications that take advantage of computer vision. Applications that can be used by small group of people are my specific interest. These applications allow me to build adaptive user interfaces and analyze the use of computer vision techniques in improving human-computer interaction.

In this paper, we formulate human motion tracking as a high dimensional constrained dynamic optimization problem. A novel generative method, called Sequential Immune Genetic Algorithm, is proposed for human motion tracking. The main contribution is that we introduce immune genetic algorithm (IGA) for pose optimization in latent space of human motion. As the latent space is low-dimensional and contains the prior knowledge of human motion, it makes pose analysis more efficient and accurate. We apply IGA for pose optimization. Compared with GA and other evolutionary methods, its main advantage is the ability to use the prior knowledge of human motion. As motion tracking is a dynamic optimization problem, we incorporate the temporal continuity information into the traditional IGA and propose a sequential IGA (S-IGA) algorithm. We demonstrate our methods on different videos of different motion types. Experimental results show that the S-IGA motion tracking method can achieve accurate and stable tracking of 3D human motion.

Today, computer vision has shown a variety of roles in our lives, making people’s life more convenient. Also, a variety of artificial intelligence models and algorithms have emerged for computer vision. Image matching is an important technique in the field of computer vision to find the similarities between two images or multiple images with the help of matching algorithms to achieve scientific and accurate processing of images. Therefore, summarizing the effective approximate solution to this problem as well as the future research is the main part for current research. The paper firstly describes the basic elements of the image matching technique. Then, some representative traditional image matching algorithms proposed in the field of computer vision research in recent years are summarized and reviewed. Finally, the future research directions and research ideas of image matching are discussed, providing reliable guidance and reference for subsequent research.

ABSTRACTThe growing demand for skilled computer vision professionals requires effective educational approaches. This study explores a novel tool based on Virtual Programming Language principles designed to enhance computer vision education, specifically in video coding. The tool leverages a user‐friendly interface and a custom widget library for video exploration, enabling students to engage with video data manipulation, motion estimation simulations, and visualization of coding effects. A controlled experiment with computer vision students from a university master's program demonstrates that the tool significantly improves student motivation, knowledge acquisition, and overall learning outcomes. This study highlights the potential of such tools to revolutionize computer vision education, leading to better engagement, deeper understanding, and enhanced practical skills. Therefore, it paves the way for further exploration of similar tools in computer vision and other science, technology, engineering, and mathematics disciplines.

Lately unmanned aerial vehicles (UAV) became an unseparated part of modern life. These machines gained their particular popularity in the field of amateur or professional video production and small cargo deliveries. But they are not only limited by civilian lines of activity. Nowadays they are also used in military conflicts either as scout or as a deadly precise weapon. Due to UAV’s easy market access, these drones can be used by the hands of terrorists groups. It’s not a rocket science to attach an explosive device to UAV, guide it to strategic area and detonate it from far away. This is one of the main danger sources for such places like nuclear power plants. A downward trend also led to new type of thread – swarm attacks, when group of small UAVs strikes their victim. A timely detection of such attacks is not a trivial task. This question is being studied by many researches from different fields. The most promising methods comes from radiolocation and computer vision areas. Next in the paper, modern algorithms of detection, classification and tracking of UAVs by using optical flow will be reviewed. Usually, the common approach for this task comes from classic computer vision methods or, highly popular today, deep learning methods.

The automation of insect pest control activities implies the use of classifiers to monitor the temporal and spatial evolution of the population using computer vision algorithms. In this regard, the popularisation of supervised learning methods represents a breakthrough in this field. However, their claimed effectiveness is reduced regarding working in real-life conditions. In addition, the efficiency of the proposed models is usually measured in terms of their accuracy, without considering the actual context of the sensing platforms deployed at the edge, where image processing must occur. Hence, energy consumption is a key factor in embedded devices powered by renewable energy sources such as solar panels, particularly in energy harvesting platforms, which are increasingly popular in smart farming applications. In this work, we perform a two-fold performance analysis (accuracy and energy efficiency) of three commonly used methods in computer vision (e.g., HOG+SVM, LeNet-5 CNN, and PCA+Random Forest) for object classification, targeting the detection of the olive fly in chromatic traps. The training and testing of the models were carried out using pictures captured in various realistic conditions to obtain more reliable results. We conducted an exhaustive exploration of the solution space for each evaluated method, assessing the impact of the input dataset and configuration parameters on the learning process outcomes. To determine their suitability for deployment on edge embedded systems, we implemented a prototype on a Raspberry Pi 4 and measured the processing time, memory usage, and power consumption. The results show that the PCA-Random Forest method achieves the highest accuracy of 99%, with significantly lower processing time (approximately 6 and 48 times faster) and power consumption (approximately 10 and 44 times lower) compared with its competitors (LeNet-5-based CNN and HOG+SVM).

A computer vision method was developed and used to assign color score in salmon fillet according to SalmonFan™ card. The methodology was based on the transformation of RGB to L*a*b* color space. In the algorithm, RGB values assigned directly to each pixel by the camera in the salmon fillet image, were transformed to L*a*b* values, and then matched with other L*a*b* values that represent a SalmonFan score (between 20 and 34). Colors were measured by a computer vision system (CVS) and a sensorial panel (eight panelists) under the same illumination conditions in ten independent sets of experiments. Errors from transformation of RGB to L*a*b* values by the CVS were 2.7%, 1%, and 1.7%, respectively, with a general error range of 1.83%. The coefficient of correlation between the SalmonFan score assigned by computer vision and the sensory panel was 0.95. Statistical analysis using t test was performed and showed that there were no differences in the measurements of the SalmonFan score between both methods (tc = 1.65 ≤ t = 1.96 at α = 0.05%). The methodology presented in this paper is very versatile and can potentially be used by computer-based vision systems in order to qualify salmon fillets based on color according to the SalmonFan card.

n the last decade, there has been an active digitalization of industrial production based on rapidly developing information technologies, including artificial intelligence technologies. This is largely due to the development of deep learning methods and their applications in computer vision. Since the mid 2010s convolutional neural networks demonstrate exceptional efficiency in solving problems such as the detection, classification and segmentation of various objects. As a result, computer vision methods are beginning to be actively used in the problems of quality control of raw materials and finished products. All this applies to the mining industry. However, in the Russian scientific literature there are practically no systematic reviews of computer vision applications in this area. The present study aims to fill this gap. The paper provides a systematic review of the history of development and the current state of the methods and technologies of machine vision used in the mining industry for the analysis of solid materials, demonstrates the latest achievements in this area and examples of their application in the mining industry. The authors have analyzed 29 research papers in the field of application of computer vision in the mining industry and classified the stages of technology development from the mid-1980s, when computer vision was used without the use of machine learning, and ending with modern research based on the use of deep convolutional neural networks for solving problems of classification and segmentation. The effectiveness of the methods used is compared, their advantages and disadvantages are discussed, and forecasts are made for the development of computer vision methods in the mining industry in the near future. Examples are given showing that the use of convolutional neural networks made it possible to move to a qualitatively higher level of quality in solving problems of classification and segmentation as applied to the analysis of output volume, particle size distribution, including flakiness, angularity and roughness, dust and clay content, bulk density and emptiness, etc.