Formal Models and Hybrid Approaches for Efficient Manual Image Annotation and Retrieval

This paper investigates new approaches to improve the efficiency of manual image annotation and help users to produce better annotation results in a given amount of time. Although important in practice, this issue has rarely been studied in a quantitative way before. To achieve this, we first propose two time models to analyze the annotation process for two popular manual annotation approaches, i.e., tagging and browsing. The complementary properties of these approaches have inspired us to merge them to develop a hybrid annotation algorithms called frequency-based annotation. Our experiments on large-scale multimedia collections have shown that the proposed algorithm can achieve an up to 40% annotation time reduction compared with the baseline methods. In other words, it can produce considerably better results using the same annotation time.

The tremendous advances in deep neural networks have demonstrated the superiority of deep learning techniques for applications such as object recognition or image classification. Nevertheless, deep learning-based methods usually require a large amount of training data, which mainly comes from manual annotation and is quite labor-intensive. In order to reduce the amount of manual work required for generating enough training data, we hereby propose to leverage existing labeled data to generate image annotations automatically. Specifically, the pixel labels are firstly transferred from one image modality to another image modality via geometric transformation to create initial image annotations, and then additional information (e.g., height measurements) is incorporated for Bayesian inference to update the labeling beliefs. Finally, the updated label assignments are optimized with a fully connected conditional random field (CRF), yielding refined labeling for all pixels in the image. The proposed approach is tested on two different scenarios, i.e., (1) label propagation from annotated aerial imagery to unmanned aerial vehicle (UAV) imagery and (2) label propagation from map database to aerial imagery. In each scenario, the refined image labels are used as pseudo-ground truth data for training a convolutional neural network (CNN). Results demonstrate that our model is able to produce accurate label assignments even around complex object boundaries; besides, the generated image labels can be effectively leveraged for training CNNs and achieve comparable classification accuracy as manual image annotations, more specifically, the per-class classification accuracy of the networks trained by the manual image annotations and the generated image labels have a difference within ± 5 % .

Since most computer vision approaches are now driven by machine learning, the current bottleneck is the annotation of images. This time-consuming task is usually performed manually after the acquisition of images. In this article, we assess the value of various egocentric vision approaches in regard to performing joint acquisition and automatic image annotation rather than the conventional two-step process of acquisition followed by manual annotation. This approach is illustrated with apple detection in challenging field conditions. We demonstrate the possibility of high performance in automatic apple segmentation (Dice 0.85), apple counting (88 percent of probability of good detection, and 0.09 true-negative rate), and apple localization (a shift error of fewer than 3 pixels) with eye-tracking systems. This is obtained by simply applying the areas of interest captured by the egocentric devices to standard, non-supervised image segmentation. We especially stress the importance in terms of time of using such eye-tracking devices on head-mounted systems to jointly perform image acquisition and automatic annotation. A gain of time of over 10-fold by comparison with classical image acquisition followed by manual image annotation is demonstrated.

Image set annotation is an important task in the supervised training of the deep neural network. Manual and data-driven dataset annotation methods are commonly used approaches. Both of them have shortcomings, especially in the case of the dataset requiring professional knowledge, which leads to high cost with manual annotation methods and poorly diversified annotation samples with data-driven annotation methods. Although the recommendation annotation method based on cosine similarity using deep neural network features takes advantages of manual annotation and data-driven method, there are still problems such as low accuracy and click-through rate. In order to improve the recommendation accuracy and click-through rate, we propose a confusion graph recommendation annotation method, which builds a confusion graph based on the Largest Margin Nearest Neighbor (LMNN) distance among deep neural network features, to recommend the most confusing images to annotators. In this paper, we made ablation studies on the self-built child face dataset in terms of Precision, mAP (mean Average Precision), and CTR (click-through-rate). The experimental results show that the proposed method achieves superior performance, compared with the cosine similarity recommendation annotation method and the manual annotation method.

Performance Comparison in Image Retrieval Using Geometric Histogram and K-Means Algorithm

Without textual descriptions or label information of images, searching semantic concepts in image databases is a very challenging task. Automatic annotation techniques for images are aimed to detect objects which are located visually inside images, like a tiger in grass. One challenge which remains to be solved is “Understanding”. Image understanding is something beyond automatic annotation scope. The second issue is manual annotation of images. In manual annotation, user contribution is important. In this demo, we have developed an online tool which simulates a collaborative environment to help users to generate several facts for selected images. Ontological thinking led us to devise a method for a simple user interface. We are also studying the construction of synergies out of generated facts for our future work.

Pixel-level Artery/Vein (A/V) classification of retinal blood vessels is important in diagnosing and understanding a wide spectrum of diseases. Deep learning-based methods have achieved promising results, but manual image annotation is a labor-intensive and time-consuming process and annotations between different datasets cannot be fully utilized. The model performance might be significantly decreased when training on one dataset but testing on another. Some unsupervised domain adaptation (UDA) methods can alleviate this problem. However, annotating a few target samples is usually very manageable and worthwhile especially if it improves the adaptation performance substantially. Consequently, we propose a novel semi-supervised retinal A/V classification method where a few labeled target samples and the labeled publicly-accessible source dataset are available. We first employ the teacher-student framework to achieve semi-supervised learning. Then, a new regional mixing method is proposed to reduce the domain gap at the region level, which takes into account the topological features of retinal vasculature. Furthermore, we combine spatial transformation with regional mixing to generate additional synthetic images, thus improving the generalization ability of the model. We evaluate the proposed method on a publicly accessible (DRIVE-AV) dataset and a private dataset, and the results show that the proposed method achieves state-of-the-art performance for A/V classification.

Automatic intracranial abnormality detection and localization in head CT scans by learning from free-text reports

The intricate process of coffee blossoming, pollination transfer, and successful development is crucial for creating every exquisite cup of coffee. During the flowering stage of coffee plants, delicate white flowers with a pleasant fragrance appear briefly, providing a limited opportunity for effective pollination. At this stage, the stigma of the flower, which is the female reproductive organ, becomes receptive and prepared to receive pollen. Existing research found methods such as machine learning and image analysis for monitoring crop pollination. Manual image annotation is conducted on pollen count disregarding spatial component of pollen collection which is essential for successful pollination. However, use of these strategies on coffee flowers by their complex structure and continuous changed in flowering stages. The paper represents novel methodologies by introducing methodological approach “IoU-AI” to monitor pollen transmission and success of pollination in coffee flower. IoU-AI utilize high resolution coffee flower image accurately track and detect floral organs by offering insight to pollen transfer. IoU-AI employs deep learning models to detect and observe floral components like stigma and anthers. Further computes overlap between structures and estimate pollen transmission. The flower detection accuracy was evaluated against ground truth measurements. The accuracy of coffee flower detection ranged from 94.77% to 85.34% for flower stages ranging from 20% to 100% blooming.

Endpoint impacts related to the transformation of land—including that related to energy infrastructure—have yet to be fully quantified and understood in life cycle assessment (LCA). Concentrated solar power (CSP) which generates electricity by using mirrors to concentrate incoming shortwave radiation onto a receiver, may serve as an alternate source of reliable baseload power in the coming years. As of 2019 (baseline year of the study), the United States (U.S.) had 1.7 GW of installed capacity across a total of eight CSP sites. In this study, we (1) develop an empirical, spatially explicit methodology to categorize physical elements embodied in energy infrastructure using a LCA approach and manual image annotation, (2) use this categorization scheme to quantify land- and ecosystem service-related endpoint impacts, notably potential losses in soil carbon, owing to energy infrastructure development and as a function of electricity generated (i.e., megawatt-hour, MWh); and (3) validate and apply this method to CSP power plants within the U.S. In the Western U.S., CSP projects are sited in Arizona, California, and Nevada. Project infrastructure can be disaggregated into the following physical elements: mirrors (“heliostats”), generators, internal roads, external roads, substations, and water bodies. Of these elements, results reveal that mirrors are the most land intensive element of CSP infrastructure (>90%). Median land transformation and capacity-based land-use efficiency are 0.4 (range of 0.3–6.8) m2/MWh and 40 (range of 11–48) W/m2, respectively. Soil grading and other site preparation disturbances may result in the release of both organic and inorganic carbon—the latter representing the majority stocks in deeper caliche layers—thus leading to potentially significant losses of stored carbon. We estimate three scenarios of soil carbon loss into the atmosphere across 30 years, based on land transformation in m2per megawatt-hour (m2/MWh) and carbon stock in kilograms of carbon per megawatt-hour (kg C/MWh). Results reveal that potential belowground CO2released may range from 7 to 137% of total life cycle CO2emissions. While this study takes a simplistic approach to estimating loss of carbon, the broad methodology provides a valuable baseline for improving comparative analyses of land-related endpoint impacts across energy technologies and other product systems.

Video annotation is an activity that aims to supplement this type of multimedia object with additional content or information about its context, nature, content, quality and other aspects. These annotations are the basis for building a variety of multimedia applications for various purposes ranging from entertainment to security. There are automatic methods for video annotation. However, these methods require specific conditions and features that are not even found in actual application scenarios. Manual annotation is a strategy that uses the intelligence and workforce of people in the annotation process and is an alternative to cases where automatic methods cannot be applied. However, manual video annotation can be a costly process because as the content to be annotated increases, so does the workload for annotating. Crowdsourcing appears as a viable solution strategy in this context because it relies on outsourcing the tasks to a multitude of workers, who perform specific parts of the work in a distributed way. However, as the complexity of required media annoyances increases, it becomes necessary to employ skilled labor, or willing to perform larger, more complicated, and more time-consuming tasks. This makes it challenging to use crowdsourcing, as experts demand higher pay, and recruiting tends to be a difficult activity. In order to overcome this problem, strategies based on the decomposition of the main problem into a set of simpler subtasks suitable for crowdsourcing processes have emerged. These smaller tasks are organized in a workflow so that the execution process can be formalized and controlled. In the literature, there are some different proposals for the construction of this type of workflow, but each of them presents limitations that still have to be overcome. In this sense, this thesis aims to present a new framework that allows the use of crowdsourcing to create applications that require complex video annotation tasks. The developed framework considers the whole process from the definition of the problem and the decomposition of the tasks, until the construction, execution, and management of the workflow. This framework, called CrowdWaterfall, contemplates the strengths of current proposals, incorporating new concepts, techniques, and resources to overcome some of its limitations.

BackgroundAutomatic functional annotation of proteins is an open research problem in bioinformatics. The growing number of protein entries in public databases, for example in UniProtKB, poses challenges in manual functional annotation. Manual annotation requires expert human curators to search and read related research articles, interpret the results, and assign the annotations to the proteins. Thus, it is a time-consuming and expensive process. Therefore, designing computational tools to perform automatic annotation leveraging the high quality manual annotations that already exist in UniProtKB/SwissProt is an important research problemResultsIn this paper, we extend and adapt the GrAPFI (graph-based automatic protein function inference) (Sarker et al. in BMC Bioinform 21, 2020; Sarker et al., in: Proceedings of 7th international conference on complex networks and their applications, Cambridge, 2018) method for automatic annotation of proteins with gene ontology (GO) terms renaming it as GrAPFI-GO. The original GrAPFI method uses label propagation in a similarity graph where proteins are linked through the domains, families, and superfamilies that they share. Here, we also explore various types of similarity measures based on common neighbors in the graph. Moreover, GO terms are arranged in a hierarchical manner according to semantic parent–child relations. Therefore, we propose an efficient pruning and post-processing technique that integrates both semantic similarity and hierarchical relations between the GO terms. We produce experimental results comparing the GrAPFI-GO method with and without considering common neighbors similarity. We also test the performance of GrAPFI-GO and other annotation tools for GO annotation on a benchmark of proteins with and without the proposed pruning and post-processing procedure.ConclusionOur results show that the proposed semantic hierarchical post-processing potentially improves the performance of GrAPFI-GO and of other annotation tools as well. Thus, GrAPFI-GO exposes an original efficient and reusable procedure, to exploit the semantic relations among the GO terms in order to improve the automatic annotation of protein functions

Content-based image retrieval (CBIR) has certain advantages over those pure keyword-based. CBIR indexes images by visual features that are extracted from the images. This may save the effort spent on the manual annotation. However, because low-level visual features, such as colour and texture, often carry no high-level concepts, images retrieved purely based on content may not match with the intention of the user. The work presented in this paper is an image retrieval system that bases both on text annotations and visual contents. It indexes and retrieves images by both keywords and visual features, with the purpose that the keywords may mend the gap between the semantic meaning an image carries and its visual content. Tests were made on the system that have demonstrated that such a hybrid approach did improve retrieval precisions over those pure content-based.

BackgroundSupervised learning can accelerate article screening in systematic reviews, but still requires labor-intensive manual annotation. While large language models (LLMs) like GPT-3.5 offer a rapid and convenient alternative, their reliability is challenging. This study aims to design an efficient and reliable annotation method for article screening.MethodsGiven that relevant articles are typically a small subset of those retrieved articles during screening, we propose a human-LLM collaborative annotation method that focuses on verifying positive annotations made by the LLM. Initially, we optimized the prompt using a manually annotated standard dataset, refining it iteratively to achieve near-perfect recall for the LLM. Subsequently, the LLM, guided by the optimized prompt, annotated the articles, followed by human verification of the LLM-identified positive samples. This method was applied to screen articles on precision oncology randomized controlled trials, evaluating both its efficiency and reliability.ResultsFor prompt optimization, a standard dataset of 200 manually annotated articles was equally divided into a tuning set and a validation set (1:1 ratio). Through iterative prompt optimization, the LLM achieved near-perfect recall in the tuning and validation sets, with 100% and 85.71%, respectively. Using the optimized prompt, we conducted collaborative annotation. To evaluate its performance, we manually reviewed a random sample of 300 articles that had been annotated using the collaborative annotation method. The results showed that the collaborative annotation achieved an F1 score of 0.9583, reducing the annotation workload by approximately 80% compared to manual annotation alone. Additionally, we trained a BioBERT-based supervised model on the collaborative annotation data, which outperformed the model trained on data annotated solely by the LLM, further validating the reliability of the collaborative annotation method.ConclusionsThe human-LLM collaborative annotation method demonstrates potential for enhancing the efficiency and reliability of article screening, offering valuable support for systematic reviews and meta-analyses.

In computer vision, data-driven convolutional neural networks could learn increasingly rich semantic features of images. However, manual annotation of images is an expensive and time-consuming task that hinders development. As a branch of unsupervised learning, self-supervised learning does not rely on labels, avoiding the work of labeling the data. This paper provides a comprehensive discussion of the development of self-supervised learning in computer vision. First, we briefly describes the motivation for proposing self-supervised learning and related concepts, introduces the self-supervised learning paradigm from three aspects, describes the applications of self-supervised learning in computer vision, and finally provides a summary and an outlook on its future development.KeywordsSelf-supervised learningDeep learningComputer vision