Dynamic Bayesian framework for extracting temporal structure in video

In recent years we have seen the development of several novel content-based retrieval (CBR) systems that have had success by focusing on a specific domain and exploiting domain-speci c information. CBR systems allow users to query a media database using item content as opposed submitting a text-based query. In many CBR applications, the input to the search process is a complicated object that may be composed of several constituent parts. The proposed approach performs CBR queries by decomposing a complex query into several heterogeneous queries. We have developed a multi-index, multi-object CBR framework for geospatial imagery retrieval that extracts features specifically developed for high-resolution commercial satellite imagery. The results of these queries will be spatially summarized for a user based on both retrieval score and spatial distance. This allows results to be presented in a logical manner to allow for more efficient interpretation by the user. Further, we propose to develop an additional search capability that allows for multi-object searches by spatial configuration rather than simply by object-to-object correspondence. Additionally, to confront situations where a user has determined that certain search results are not relevant, we will provide online and memory-based relevance feedback algorithms for use with multi-index, multi-object CBR systems. The experimental results demonstrate the efficiency and accuracy of the proposed methods; moreover, through the fusion of multi-index and multi-object search techniques, we are able to construct new, sophisticated query mechanisms.

As the size of multimedia database grows, it becomes impractical to manually annotate all contents and attributes of the media, and the difficulty in finding desired information increases. To copy with these challenges, content-based multimedia retrieval systems have been developed for various applications. The chapter not only provides a conceptual architecture for the design of content-based retrieval system, but also discusses essential components of retrieval system and their research issues, including feature extraction and representation, dimension reduction of feature vector, indexing, and query specifications. As content-based multimedia retrieval is a young research field and there exists many challenging research problems, this chapter also addresses several research issues for the future research.

In this paper, the content-based retrieval with audio examples based on MP3 digital music archive for one singer is considered. In our proposed method, the Sub-Band Coefficients (SBC) in a MP3 frame is used for feature extraction. Both Quantization-Tree indexing (QT) approach and the Melody-Line Tracking (MLT) approach are proposed for indexing MP3 objects. Finally, a two-stage comparison method is used to measure the similarity between MP3 objects. We also employ the measurement of the similarity between query sample and database items for a singer. Evaluations on a content-based MP3 retrieval system are performed. Experimental results show that our proposed approach can achieve good performance and high accuracy.

Large image databases have been designed and used in various applications and in content-based retrieval systems. Medical Imaging modalities have been developed in the past each uniquely designed to specific imaging modality based on the image pathobiological medium. This paper presents an approach to improve the process of image storage and retrieval based on image data representation model that integrates both metadata and content description. The low level features and the labels associated with the salient objects extracted from the mages are stored in an ORDBMS (Object Relational Database Management systems) using Oracle. Our technique, which combines the image descriptors obtained by a unified approach phases of image segmentation and annotation, has a good time complexity and the experimental results on several image datasets.

In this paper, a prototype system is described for the management and content-based retrieval of defect images in huge image databases. This is a real problem in surface inspection applications, since modern inspection systems may produce up to thousands of defect images in a day. The retrieval is based on shape and internal structure characteristics of defects, so neither manual labeling nor annotation needs to be made. We are using a noncommercial, generic content-based image retrieval (CBIR) system called PicSOM that is modified to fit the special requirements of our application. The results of experiments show that the prototype system works fast with good retrieval results.

An on-line audio classification and segmentation system is presented in this research, where audio recordings are classified and segmented into speech, music, several types of environmental sounds and silence based on audio content analysis. This is the first step of our continuing work towards a general content-based audio classification and retrieval system. The extracted audio features include temporal curves of the energy function,the average zero- crossing rate, the fundamental frequency of audio signals, as well as statistical and morphological features of these curves. The classification result is achieved through a threshold-based heuristic procedure. The audio database that we have built, details of feature extraction, classification and segmentation procedures, and experimental results are described. It is shown that, with the proposed new system, audio recordings can be automatically segmented and classified into basic types in real time with an accuracy of over 90 percent. Outlines of further classification of audio into finer types and a query-by-example audio retrieval system on top of the coarse classification are also introduced.

How to find the image we need expediently in the tremendous database is one of the most important issues in content-based image retrieval (CBIR). In this paper, we presented an image retrieval system based on image content using fuzzy logic and proposed a new concept on partition the entire database based on content self-organized. In detail, we used modified fuzzy c-means (MFCM) clustering scheme to cluster the database before retrieval and defined a new method to determine the fuzzy weight. To improve the retrieval effectiveness more, this paper also used interactive query (IQ) and presented a new relevance feedback (RF) framework. On all accounts, we designed and implemented a content-based self-organized and interactive retrieval system. Experiments show that it contributes to improving retrieval effectiveness and will be applied in the E-government.

MiDiLiB is a six year research project on digital music libraries funded by the German Research Foundation (DFG) as a part of the Distributed Processing and Delivery of Digital Documents (V3D2) research initiative. MiDiLiB’s main focus is the development of content-based retrieval algorithms for both score- and waveform-based music. In this paper we give an overview of our research results, describe several prototypical systems for content-based music retrieval which have been developed during the project, and discuss applications of the presented techniques in the context of today’s and future digital music libraries.

Retrieving similar images is a challenging task for today's content-based retrieval systems. Aiming at high retrieval performance, these systems frequently capture the user's notion of similarity through expressive image models and adaptive similarity measures, which try to approximate the individual user-dependent notion of similarity as close as possible. As image models appearing on the query side can significantly differ in quality compared to those stored in the multimedia database, similarity measures have to be robust against these individual quality changes in order to maintain high retrieval performance. In order to evaluate the robustness of similarity measures, we introduce the general concept of the stability of a similarity measure with respect to query modifying transformations describing the change in quality on the query side. In addition, we include a comparison of the stability of the major state-of-the-art adaptive similarity measures based on different benchmark image databases.

Accurate figure-ground segregation of decoration designs allows for the computation of high-level design features for use in content-based retrieval and indexing systems, and may further serve as a basis for design understanding and various design manipulations. This paper outlines a method for figure-ground segregation that is particularly suitable for color design images generally encountered in the textile industry. We propose a strategy consisting of three main steps: (i) obtain initial candidates for the background by multi-scale detection of color texture regions; (ii) assess the appropriateness of the candidates by a classification algorithm based on visual cues such as relative size, connectedness and massiveness; (iii) integrate the results of the previous steps to produce a hierarchical description of the figure-ground structure of the design. The first step is performed by a novel algorithm for color texture detection based on an analysis of the local color structure of the design. The multi-scale nature of the algorithm allows for the detection of nested backgrounds. Results are presented for two databases of decorative design images; one of specialized type and one of general type.

In recent years, the rapid expansion of multimedia applications, partly due to the exponential growth of the Internet, has proliferated over the daily life of computer users (Yang & Hurson, 2006). The integration of wireless communication, pervasive computing, and ubiquitous data processing with multimedia database systems has enabled the connection and fusion of distributed multimedia data sources. In addition, the emerging applications, such as smart classroom, digital library, habitat/environment surveillance, traffic monitoring, and battlefield sensing, have provided increasing motivation for conducting research on multimedia content representation, data delivery and dissemination, data fusion and analysis, and contentbased retrieval. Consequently, research on multimedia technologies is of increasing importance in computer society. In contrast with traditional text-based systems, multimedia applications usually incorporate much more powerful descriptions of human thought—video, audio, and images (Karpouzis, Raouzaiou, Tzouveli, Iaonnou, & Kollias, 2003; Liu, Bao, Yu, & Xu, 2005; Yang & Hurson, 2005). Moreover, the large collections of data in multimedia systems make it possible to resolve more complex data operations such as imprecise query or content-based retrieval. For instance, the image database systems may accept an example picture and return the most similar images of the example (Cox, Miller, & Minka, 2000; Hsu, Chua, & Pung, 2000; Huang, Chang, & Huang, 2003). However, the conveniences of multimedia applications come with challenges to the existing data management schemes: • Efficiency: Multimedia applications generally require more resources; however, the storage space and processing power are limited in many practical systems, for example, mobile devices and wireless networks (Yang & Hurson, 2005). Due to the large data volume and complicated operations of multimedia applications, new methods are needed to facilitate efficient representation, retrieval, and processing of multimedia data while considering the technical constraints. • Semantic Gap: There is a gap between user perception of multimedia entities and physical representation/access mechanism of multimedia data. Users often browse and desire to access multimedia data at the object level (“entities” such as human beings, animals, or buildings). However, the existing multimedia retrieval systems tend to access multimedia data based on their lower-level features (“characteristics” such as color patterns and textures), with little regard to combining these features into data objects. This representation gap often leads to higher processing cost and unexpected retrieval results. The representation of multimedia data according to human’s perspective is one of the focuses in recent research activities; however, few existing systems provide automated identification or classification of objects from general multimedia collections. • Heterogeneity: The collections of multimedia data are often diverse and poorly indexed. In a distributed environment, because of the autonomy and heterogeneity of data sources, multimedia data objects are often represented in heterogeneous formats. The difference in data formats further leads to the difficulty of incorporating multimedia data objects under a unique indexing framework. • Semantic Unawareness: The present research on content-based multimedia retrieval is based on feature vectors—features are extracted from audio/video streams or image pixels, empirically or heuristically, and combined into vectors according to the application criteria. Because of the application-specific multimedia formats, the feature-based paradigm lacks scalability and accuracy.

The contribution of visual information to the perception of speech in noise with and without informative temporal fine structure

We introduce the problem of domain adaptation for content-based retrieval and propose a domain adaptation method based on relative aggregation points (RAPs). Content-based retrieval including image retrieval and spoken document retrieval enables a user to input examples as a query, and retrieves relevant data based on the similarity to the examples. However, input examples and relevant data can be dissimilar, especially when domains from which the user selects examples and from which the system retrieves data are different. In content-based geographic object retrieval, for example, suppose that a user who lives in Beijing visits Kyoto, Japan, and wants to search for relatively inexpensive restaurants serving popular local dishes by means of a content-based retrieval system. Since such restaurants in Beijing and Kyoto are dissimilar due to the difference in the average cost and areas' popular dishes, it is difficult to find relevant restaurants in Kyoto based on examples selected in Beijing. We propose a solution for this problem by assuming that RAPs in different domains correspond, which may be dissimilar but play the same role. A RAP is defined as the expectation of instances in a domain that are classified into a certain class, e.g. the most expensive restaurant, average restaurant, and restaurant serving the most popular dishes. Our proposed method constructs a new feature space based on RAPs estimated in each domain and bridges the domain difference for improving content-based retrieval in heterogeneous domains. To verify the effectiveness of our proposed method, we evaluated various methods with a test collection developed for content-based geographic object retrieval. Experimental results show that our proposed method achieved significant improvements over baseline methods. Moreover, we observed that the search performance of content-based retrieval in heterogeneous domains was significantly lower than that in homogeneous domains. This finding suggests that relevant data for the same search intent depend on the search context, that is, the location where the user searches and the domain from which the system retrieves data.

Content Based Retrieval (CBR) systems use Relevance Feedback (RF) to fill the semantic gap. RF can be short-term or long-term. The introduction of long-term learning methods address the memory problem in short-term learning methods. In this letter we propose a new method to enhance the gain of long-term relevance feedback. We have come up with a long term learning scheme in relevance feedback for CBR. The proposed system integrates the user feedback from all iterationations and instills memory into the feedback system of CBR without saving any log of earlier retrievals. In this letter, we have come up with a method to update the cluster parameters and weights assigned to features by accumulating the knowledge obtained from the user over iterations. The proposed update method is validated in the image retrieval context in terms of conventional recall-precision graph and retrieval accuracy.

This paper proposed a content-based video retrieval system based on object motion trajectory. An algorithm for tracking moving objects in MPEG-compressed domain is developed. It is to link individual macroblocks in the temporal domain first and then prune and merge the formed paths by considering spatial adjacency of MBs. In this way, the difficult spatial segmentation problem of traditional methods is avoided and tracking of multiple deformed objects can be achieved. Also, our system is capable of eliminating global motion so that camera motion is allowed. The extracted object motion trajectory is then converted into a form conformable to MPEG-7 motion descriptor (keypoints + interpolating functions). Both interfaces of query-by-example and query-by-sketch are provided and problems in descriptor matching (e.g., mismatch in keypoint interval and video time duration) are solved to achieve robustness and a high recall rate. We have tested our algorithm with real video clips, including fixed- or moving-camera, rigid or deformed, single or multiple objects, varying object size during motion, etc. Exeriments show that the tracking and retrieval results are satisfactory and suitable for further applications.