This work provides the fundament for a gesture-based interaction system between cargo-handling unmanned aerial vehicles (UAVs) and ground personnel. It enables novice operators to visually communicate commands with higher abstractions through a minimum number of necessary gestures. The interaction concept intends to transfer two goal-directed control techniques to a cargo-handling use case: Selecting objects via deictic pointing communicates intention and a single proxy manipulation gesture controls the UAV’s flight. A visual processing pipeline built around an RGB-D sensor is presented and its subordinate components like lightweight object detectors and human pose estimation methods are benchmarked on the UAV-Human dataset. The results provide an overview of suitable methods for 3D gesture-based human drone interaction. A first unoptimized model ensemble runs with 7 Hz on a Jetson Orin AGX Developer Kit.

Estimating the success of a song before its release is an important music industry task. This work uses audio descriptors to predict the success (popularity) of a song, where typical measures of success are chart measures such as peak position and streaming measures such as listener-count. Currently, a wide range of datasets is used for that purpose, but most of them are not publicly available; likewise, available datasets are restricted either in size, available features, or popularity measures. This substantially impedes the evaluation of the predictive power of a wide range of models. Therefore, we present two novel datasets called HSP-S and HSP-L based on data from AcousticBrainz, Billboard Hot 100, the Million Song Dataset, and last.fm. Both datasets contain audio features, mel-spectrograms as well as streaming listener- and play-counts. The larger HSP-L dataset contains 73,482 songs, whereas the smaller HSP-S dataset contains 7736 songs and additionally features Billboard Hot 100 chart measures. In contrast to the previous publicly available datasets, our datasets contain substantially more songs and richer and more diverse features. We solely utilize data from the public domain, allowing us to evaluate and compare a wide range of models on our datasets. To demonstrate the use of the datasets, we perform regression and classification (popular/unpopular) tasks on both datasets using a wide variety of models to predict song popularity for all provided target measures of success.

Artificial intelligence applications within the geosciences are becoming increasingly common, yet there are still many challenges involved in adapting established techniques to geoscience data sets. Applications in the realm of volcanic hazards assessment show great promise for addressing such challenges. Here, we describe a Jupyter Notebook we developed that ingests real-time Global Navigation Satellite System (GNSS) data streams from the EarthCube CHORDS (Cloud-Hosted Real-time Data Services for the geosciences) portal TZVOLCANO, applies unsupervised learning algorithms to perform automated data quality control (“noise reduction”), and explores autonomous detection of unusual volcanic activity using a neural network. The TZVOLCANO CHORDS portal streams real-time GNSS positioning data in 1[Formula: see text]s intervals from the TZVOLCANO network, which monitors the active volcano Ol Doinyo Lengai in Tanzania, through UNAVCO’s real-time GNSS data services. UNAVCO’s real-time data services provide near-real-time positions processed by the Trimble Pivot system. The positioning data (latitude, longitude and height) are imported into the Jupyter Notebook presented in this paper in user-defined time spans. The positioning data are then collected in sets by the Jupyter Notebook and processed to extract a useful calculated variable in preparation for the machine learning algorithms, of which we choose the vector magnitude for further processing. Unsupervised K-means and Gaussian Mixture machine learning algorithms are then utilized to locate and remove data points (“filter”) that are likely caused by noise and unrelated to volcanic signals. We find that both the K-means and Gaussian Mixture machine learning algorithms perform well at identifying regions of high noise within tested GNSS data sets. The filtered data are then used to train an artificial intelligence neural network that predicts volcanic deformation. Our Jupyter Notebook has promise to be used for detecting potentially hazardous volcanic activity in the form of rapid vertical or horizontal displacement of the Earth’s surface.

With the increasing availability of human motion data captured in the form of 2D or 3D skeleton sequences, more complex motion recordings need to be processed. In this paper, we focus on similarity-based indexing and efficient retrieval of motion episodes — medium-sized skeleton sequences that consist of multiple semantic actions and correspond to some logical motion unit (e.g. a figure skating performance). As a first step toward efficient retrieval, we apply the motion-word technique to transform spatio-temporal skeleton sequences into compact text-like documents. Based on these documents, we introduce a two-phase retrieval scheme that first finds a set of candidate query results and then re-ranks these candidates with more expensive application-specific methods. We further index the motion-word documents using inverted files, which allows us to retrieve the candidate documents in an efficient and scalable manner. We also propose additional query-reduction techniques that accelerate both the retrieval phases by removing semantically irrelevant parts of the motion query. Experimental evaluation is used to analyze the effects of the individual proposed techniques on the retrieval efficiency and effectiveness.

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder–decoder-based models without considering semantic information. To fill this gap, we present a novel encoder–decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-based encoder–decoder model. To enable semantic embeddings for the test audios, we introduce a Multilayer Perceptron classifier to predict the semantic embeddings of those clips. We also present exhaustive experiments to show the efficiency of different features and datasets for our proposed model the audio captioning task. To extract audio features, we use the log Mel energy features, VGGish embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive experiments on two audio captioning datasets Clotho and AudioCaps show that our proposed model outperforms state-of-the-art audio captioning models across different evaluation metrics and using the semantic information improves the captioning performance.

The rise of digital payments has caused consequential changes in the financial crime landscape. As a result, traditional fraud detection approaches such as rule-based systems have largely become ineffective. Artificial intelligence (AI) and machine learning solutions using graph computing principles have gained significant interest in recent years. Graph-based techniques provide unique solution opportunities for financial crime detection. However, implementing such solutions at industrial-scale in real-time financial transaction processing systems has brought numerous application challenges to light. In this paper, we discuss the implementation difficulties current and next-generation graph solutions face. Furthermore, financial crime and digital payments trends indicate emerging challenges in the continued effectiveness of the detection techniques. We analyze the threat landscape and argue that it provides key insights for developing graph-based solutions.

Hospital workers often experience burnout due to the demanding job responsibilities and long work hours. Data yielding from ambulatory monitoring combined with machine learning algorithms can afford us a better understanding of the naturalistic processes that contribute to this burnout. Motivated by the challenges related to the accurate tracking of well-being in real-life, prior work has investigated group-specific machine learning (GS-ML) models that are tailored to groups of participants. We examine a novel GS-ML for estimating well-being from real-life multimodal measures collected in situ from hospital workers. In contrast to the majority of prior work that uses pre-determined clustering criteria, we propose an iterative procedure that refines participant clusters based on the representations learned by the GS-ML models. Motivated by prior work that highlights the differential impact of job demands on well-being, we further explore the participant clusters in terms of demography and job-related attributes. Results indicate that the GS-ML models mostly outperform general models in estimating well-being constructs. The GS-ML models further depict different degrees of predictive power for each participant cluster, as distinguished upon age, education, occupational role, and number of supervisees. The observed discrepancies with respect to the GS-ML model decisions are discussed in association with algorithmic bias.

Customer reviews are an essential resource to reduce an online product’s uncertainty, which has been shown to be a critical factor for its purchase decision. Existing e-commerce platforms typically ask users to write free-form text reviews, which are sometimes augmented by a small set of predefined questions, e.g. “rate the product description’s accuracy from 1 to 5.” In this paper, we argue that this “passive” style of review solicitation is suboptimal in achieving low-uncertainty “review profiles” for products. Its key drawback is that some product aspects receive a very large number of reviews while other aspects do not have enough reviews to draw confident conclusions. Therefore, we hypothesize that we can achieve lower-uncertainty review profiles by carefully selecting which aspects users are asked to rate. To test this hypothesis, we propose various techniques to dynamically select which aspects to ask users to rate given the current review profile of a product. We use Bayesian inference principles to define reasonable review profile uncertainty measures; specifically, via an aspect’s rating variance. We compare our proposed aspect selection techniques to several baselines on several review profile uncertainty measures. Experimental results on two real-world datasets show that our methods lead to better review profile uncertainty compared to aspect selection baselines and traditional passive review solicitations. Moreover, we present and evaluate a hybrid solicitation method that combines the advantages of both active and passive review solicitations.

We study how mixed reality (MR) telepresence can enhance long-distance human interaction and how altering 3D representations of a remote person can be used to modulate stress and anxiety during social interactions. To do so, we developed an MR telepresence system employing commodity depth sensors and Microsoft’s Hololens. A textured, polygonal 3D model of a person was reconstructed in real time and transmitted over network for rendering in remote location using HoloLens. In this study, we used mock job interview paradigm to induce stress in human–subjects interacting with an interviewer presented as an MR hologram. Participants were exposed to three different types of real-time reconstructed virtual holograms of the interviewer, a natural-sized 3D reconstruction (NR), a miniature 3D reconstruction (SR) and a 2D-display representation (LCD). Participants reported their subjective experience through questionnaires, while their biophysical responses were recorded. We found that the size of 3D representation of a remote interviewer had a significant effect on participants’ stress levels and their sense of presence. The questionnaire data showed that NR condition induced more stress and presence than SR condition and was significantly different from LCD condition. We also found consistent patterns in the biophysical data.