Semisupervised Cross-Media Retrieval by Distance-Preserving Correlation Learning and Multi-modal Manifold Regularization

Existing cross-media retrieval methods are usually conducted under the supervised setting, which need lots of annotated training data. Generally, it is extremely labor-consuming to annotate cross-media data. So unsupervised cross-media retrieval is highly demanded, which is very challenging because it has to handle heterogeneous distributions across different media types without any annotated information. To address the above challenge, this paper proposes Domain Adaptation with Scene Graph (DASG) approach, which transfers knowledge from the source domain to improve cross-media retrieval in the target domain. Our DASG approach takes Visual Genome as the source domain, which contains image knowledge in the form of scene graph. The main contributions of this paper are as follows: First, we propose to address unsupervised cross-media retrieval by domain adaptation. Instead of using the labor-consuming annotated information of cross-media data in the training stage, our DASG approach learns cross-media correlation knowledge from Visual Genome, and then transfers the knowledge to cross-media retrieval through media alignment and distribution alignment. Second, our DASG approach utilizes fine-grained information via scene graph representation to enhance generalization capability across domains. The generated scene graph representation builds (subject$\rightarrow $ relationship$\rightarrow $ object) triplets by exploiting objects and relationships within image and text, which makes the cross-media correlation more precise and promotes unsupervised cross-media retrieval. Third, we exploit the related tasks including object and relationship detection for learning more discriminative features across domains. Leveraging the semantic information of objects and relationships improves cross-media correlation learning for retrieval. Experiments on two widely-used cross-media retrieval datasets, namely Flickr-30K and MS-COCO, show the effectiveness of our DASG approach.

Cross-media retrieval aims to retrieve across different media types, such as image and text, whose key problem is to learn cross-media correlation from known training data. Existing methods indiscriminately take all data for model training, ignoring that there exist hard samples which lead to misleading and even noisy information, bringing negative effect especially in the early period of model training. Because cross-media training data is difficult to collect, the common challenge of small-scale training data makes this problem even severer to limit the robustness and accuracy of cross-media retrieval. For addressing the above problem, this paper proposes Progressive Cross-media Correlation Learning (PCCL) approach, which takes a large-scale cross-media dataset with general knowledge (reference data), to guide the correlation learning on another small-scale dataset (target data) via the progressive sample selection mechanism. Specifically, we first pre-train a hierarchical correlation learning network on reference data as reference model, which is used to assign samples in target data with different learning difficulties, via intra-media and inter-media relevance significance metric. Then, training samples in target data are selected with gradually ascending learning difficulties, so that the correlation learning process can progressively reduce the “heterogeneity gap” to enhance the model robustness and improve retrieval accuracy. We take our self-constructed large-scale XMediaNet dataset as the reference data, and the cross-media retrieval experiments on 2 widely-used datasets show PCCL outperforms 9 state-of-the-art methods.

Emerging multimedia content including images and texts are always jointly utilized to describe the same semantics. As a result, cross-media retrieval becomes increasingly important, which is able to retrieve the results of the same semantics with the query but with different media types. In this paper, we propose a novel heterogeneous similarity measure with nearest neighbors (HSNN). Unlike traditional similarity measures which are limited in homogeneous feature space, HSNN could compute the similarity between media objects with different media types. The heterogeneous similarity is obtained by computing the probability for two media objects belonging to the same semantic category. The probability is achieved by analyzing the homogeneous nearest neighbors of each media object. HSNN is flexible so that any traditional similarity measure could be incorporated, which is further regarded as the weak ranker. An effective ranking model is learned from multiple weak rankers through AdaRank for cross-media retrieval. Experiments on the wikipedia dataset show the effectiveness of the proposed approach, compared with state-of-the-art methods. The cross-media retrieval also shows to outperform image retrieval systems on a unimedia retrieval task.

With unstructured heterogeneous multimedia data such as texts, images being more and more widely used on the web, cross-media retrieval has become an increasingly important task. One of the key techniques in cross-media retrieval is how to compute distances or similarities among different types of media data. In this paper, we propose a novel heterogeneous metric learning method to compute distances between images and texts. We extend Locally Linear Embedding (LLE) to deal with heterogeneous data, so that we can not only preserve homogeneous local information but also capture heterogeneous constraints. In order to handle the out-of-sample problem, we learn two map functions from the embedding, and use them to transform heterogeneous data into a homogeneous space and do the retrieval in the new space. The experimental results on two real-world datasets show the effectiveness of our approach.

Portable media consoles are becoming extremely popular devices for viewing a number of different types of media content, both for entertainment and for educational purposes. Given the increasingly heavy use of portable consoles as an alternative to traditional television-style monitors, it is important to investigate how physiological and psychological effects of portable consoles may differ from those of television-based consoles, because such differences in physiological and psychological responses may precipitate differences in the delivered content's effectiveness. Because portable consoles are popular as a delivery system for multiple types of media content, such as movies and video games, it is also important to investigate whether differences between the effects of portable and television-based consoles are consistent across multiple types of media. This article reports a 2 x 2 (console: portable or television-based x medium: video game or movie) mixed factorial design experiment with physiological arousal and self-reported flow experience as dependent variables, designed to explore whether console type affects media experiences and whether these effects are consistent across different media. Results indicate that portable media consoles evoke lower levels of physiological arousal and flow experience and that this effect is consistent for both video games and movies. These findings suggest that even though portable media consoles are often convenient compared to television-based consoles, the convenience may come at a cost in terms of the user experience.

Cross-media retrieval is becoming increasingly important nowadays. To address this challenging problem, most existing approaches project heterogeneous features into a unified feature space to facilitate their similarity computation. However, this unified feature space usually has no explicit semantic meanings, which might ignore the hints contained in the original media content, and thus is not able to fully measure the similarities among different media types. By considering the above issues, we propose a new approach to cross-media retrieval via semantic entity projection SEP in this paper. Our approach consists of three main steps. Firstly, an entity level with fine-grained semantics between low-level features and high-level concepts are constructed, so as to help bridge the semantic gap to a certain extent. Then, an entity projection is learned by minimizing both cross-media correlation error and single-media reconstruction error from low-level features to the entity level, with which a unified feature space with explicit semantic meanings can be obtained from low-level features. Finally, the semantic abstraction of high-level concepts is generated by using logistic regression to conduct cross-media retrieval. Experimental results on the Wikipedia dataset show the effectiveness of the proposed approach.

Nowadays cross-media retrieval is an useful technology that helps people find expected information from the huge amount of multimodal data more efficiently. A common cross-media retrieval framework is first to map features of different modalities into an isomorphic semantic space so that the similarity between heterogeneous data can be measured. For most of semantic space based methods, the mapping mechanism from original to semantic space of each modality is optimized independently, yet the more discriminative characteristic of a certain modality is not taken into account. In this paper, we propose a deep framework which introduces a latent embedding layer to learn joint parameters to obtain semantically meaningful representations of images and texts. Specifically, the discriminative characteristic embedded in the textual modality can be transferred to images through the latent embedding layer and joint parameters to enhance the consistency between semantic representations. Extensive experiments on the three popular publicly available datasets well demonstrate the superiority of the proposed method, which achieves the new state-of-the-arts.

Simultaneously learning affinity matrix and data representations for machine fault diagnosis

This presentation explores the analysis of heterogeneous geospatial data from various sources through the application of artificial intelligence (AI) tools. Wastewater networks are used as a case study to address challenges such as data completion, multi-source integration, and managing diverse data formats, including Geographic Information Systems (GIS), analog maps, and pipe inspection videos, all derived from real-world data. We will review some solutions developed under the European project Starwars (STormwAteR and WastewAteR networkS heterogeneous data AI-driven management). These solutions are based on innovative models and tools that employ logical and graph-based representations of heterogeneous data. Specifically, we aim to represent different data types — such as GIS, ITV inspection videos, and maps — as annotated graphs, incorporating the uncertainty stemming from incomplete or inconsistent information.

The paper presents an extension of co-synthesis for data dominated applications to include reactive processes. The extension allows for rate constraints as used in data dominated applications as well as minimum and maximum time constraints for communication and I/O which is required to define reactive behavior of control tasks. A co-synthesis approach is proposed which differentiates global process and communication scheduling, which is non preemptive, and local scheduling which includes a restricted interrupt controlled process invocation to extend the design space. Several user parameters allow design space exploration. The approach includes buffering, process pipelining and parallelization for control as well as for data dominated tasks on different levels of granularity. It supports inter process time constraints which span processes with different periods. The target architectures are heterogeneous systems consisting of multiple processors, hardware components, memories and different types of communication media.

As the sole genus of the Thymelaeaceae in the continental United States, Dirca L. represents an important component of the diversity of the region. And while Dirca spp. have noted ornamental value these shrubs have not been utilized by the horticulture industry. Incorporating new species of conservational importance, such as Dirca spp., has multifaceted benefits, but is not without risks. The research contained herein is aimed at addressing selected risks associated with the utilization of Dirca spp.—that of supply, potential distribution in relation to cold hardiness, and possible pests. The first objective of this work was identifying how environmental conditions influence phenology of a species of Dirca and how that phenology relates to fruit set of these shrubs. Through tracking phenology and fruit set through multiple years, I demonstrate that low autumnal precipitation is associated with delayed growth resumption and flowering, which corresponds with increased fruit set of this rare species. The next objective was evaluating how pollinator presence affects the fruit set of D. mexicana in central Iowa. I introduced pollinators into cages with plants of D. mexicana to measure the change in fruit set compared to plants where pollinators were excluded and to open-pollinated plants. Compared to a pollinator-excluded treatment, the inclusion of pollinators or open-pollination increased probability of fruit set by more than ten times, but fruit set did not differ among pollinator types. The third objective involved developing a means to analyze pollen viability and longevity of Dirca and quantifying those traits. By testing multiple types of artificial media and other pollen-germination strategies, my results indicate that pollen longevity may be a contributing factor to the low fruit set observed in D. mexicana planted in temperate regions. Measuring the extent of damage that the leaf miner L. dircella causes in populations of D. occidentalis was the fourth objective. Damage was observed at multiple sites containing D. occidentalis, but the extent of damage varied among locations. The final objective of the dissertation was assessing the cold hardiness of Dirca spp. in central Iowa, including selections of particular interest for horticultural use. All the study plants acclimated quickly and deeply enough to endure local average and record low temperatures throughout the winter. In this work I have identified that if the barriers to seed production of Dirca spp. can be overcome, the multifaceted benefits of using these taxa far outweigh the few risks associated with the use of this genus.

Lake Bonney is one of numerous permanently ice-covered lakes located in the McMurdo Dry Valleys, Antarctica. The perennial ice cover maintains a chemically stratified water column and unlike other inland bodies of water, largely prevents external input of carbon and nutrients from streams. Biota are exposed to numerous environmental stresses, including year-round severe nutrient deficiency, low temperatures, extreme shade, hypersalinity, and 24-hour darkness during the winter 1. These extreme environmental conditions limit the biota in Lake Bonney almost exclusively to microorganisms 2. Single-celled microbial eukaryotes (called "protists") are important players in global biogeochemical cycling 3 and play important ecological roles in the cycling of carbon in the dry valley lakes, occupying both primary and tertiary roles in the aquatic food web. In the dry valley aquatic food web, protists that fix inorganic carbon (autotrophy) are the major producers of organic carbon for organotrophic organisms 4, 2. Phagotrophic or heterotrophic protists capable of ingesting bacteria and smaller protists act as the top predators in the food web 5. Last, an unknown proportion of the protist population is capable of combined mixotrophic metabolism 6, 7. Mixotrophy in protists involves the ability to combine photosynthetic capability with phagotrophic ingestion of prey microorganisms. This form of mixotrophy differs from mixotrophic metabolism in bacterial species, which generally involves uptake dissolved carbon molecules. There are currently very few protist isolates from permanently ice-capped polar lakes, and studies of protist diversity and ecology in this extreme environment have been limited 8, 4, 9, 10, 5. A better understanding of protist metabolic versatility in the simple dry valley lake food web will aid in the development of models for the role of protists in the global carbon cycle. We employed an enrichment culture approach to isolate potentially phototrophic and mixotrophic protists from Lake Bonney. Sampling depths in the water column were chosen based on the location of primary production maxima and protist phylogenetic diversity 4, 11, as well as variability in major abiotic factors affecting protist trophic modes: shallow sampling depths are limited for major nutrients, while deeper sampling depths are limited by light availability. In addition, lake water samples were supplemented with multiple types of growth media to promote the growth of a variety of phototrophic organisms. RubisCO catalyzes the rate limiting step in the Calvin Benson Bassham (CBB) cycle, the major pathway by which autotrophic organisms fix inorganic carbon and provide organic carbon for higher trophic levels in aquatic and terrestrial food webs 12. In this study, we applied a radioisotope assay modified for filtered samples 13 to monitor maximum carboxylase activity as a proxy for carbon fixation potential and metabolic versatility in the Lake Bonney enrichment cultures.

Computing surface-based Voronoi diagrams is a fundamental operation in geometry processing, typically relying on either geodesic or straight-line distances as solvers. However, when the input is a 3D model containing thin-plate structures, geodesic distances incur significant computational overhead,whereas straight-line distances can result in ownerless regions. To address this issue, we propose integrating biharmonic embedding distances into the SurfaceVoronoi framework. Specifically, mesh vertices can be embedded into a high-dimensional spectral space, ensuring that the embedding distance closely approximates the straight-line distance between sufficiently close points. In contrast, when points reside on opposite sides of a thin plate, the embedding distance significantly exceeds the straight-line distance, effectively preventing dominance from penetrating through to the opposite side. Our proposed framework offers several advantages: 1) It operates efficiently, as the embedding distance can be rapidly evaluated as a straight-line distance in high-dimensional space. 2) It guarantees the “one site, one region” property, even for models consisting of thinplate structures. 3) It enables high-quality triangulation through iterative repositioning of each site to the centroid of its dominant region. Extensive experiments conducted on benchmark meshes demonstrate these benefits.

The best-studied species to date are the human pathogens Candida albicans and C. glabrata. However, neither of these species is as yet known to possess a complete meiotic sexual cycle. C. lusitaniae, C. guilliermondii, and C. krusei are three members of this genus that are exciting prospects for further study. Of the three species discussed in this chapter, the sexual life cycle has been the best studied for this species. Multiple types of media are capable of inducing mating including dilute potato dextrose agar, 1% malt extract media, sodium acetate, yeast carbon base, V8, and SLAD. The anamorph has been cultured from a variety of ecological niches, including human clinical specimens, insects, fruit, and decaying matter. The sexual cycle of C. guilliermondii was first identified in 1952 by Wickerham and Burton, after recognizing that some yeasts previously classified to non-ascosporeforming genera actually represented the anamorphic form of a sexual species. That sexual Candida species exist forms the basis of the argument that loss or restriction of sexual reproduction is the evolved state, possibly due to the energy expenditure required to undergo meiosis or to limit genetic exchange in a pathogen highly evolved to its host niche. Thus, continued study of the Candida species complex will provide insight on interesting evolutionary questions regarding the evolution of signal transduction pathways, sexual reproduction, commensalism, and pathogenesis.

This commentary about alcohol marketing regulation in the UK draws on a conference held by the Institute of Alcohol Studies, highlighting a need for policy interventions to prevent harm and improve public health. Hazardous and harmful alcohol use is associated with many health conditions, wider social consequences, and harms to others. Following no improvement in alcohol mortality rates in the past decade, 2020 saw alcohol-specific deaths rise to record levels in the UK. Bans or comprehensive restrictions on alcohol advertising across multiple types of media are listed by the World Health Organization (WHO) as one of the ‘best buy’ policies to reduce alcohol harm. The UK’s current complaints-led self-regulatory approach fails to protect consumers and vulnerable groups from being exposed to influential alcohol marketing. There are few meaningful sanctions to deter brands and companies from violating existing codes, processes are retrospective, reactive and slow, and the codes fail in their stated aim of protecting young people. Other important impacts on heavier drinkers and those in recovery, as well as on gender and health equity, are also inadequately addressed. Innovation is also urgently needed to effectively regulate ever-evolving digital alcohol marketing. Addressing these issues through a combination of comprehensive restrictions, content controls, labelling, and replacing self-regulation with an independent body will benefit public health as well as protecting the vulnerable, including heavier drinkers, people in recovery, and children and young people.