A framework for semantic annotation of digital evidence

Learning objects (LOs) are digital or non-digital entities used for learning, education or training commonly stored in repositories searchable by their associated metadata. Unfortunately, based on the current standards, such metadata is often missing or incorrectly entered making search difficult or impossible. In this paper, we investigate automating metadata generation for SCORM-complaint LOs based on user interactions with the LO and static information about LOs and users. Our framework, called the Intelligent Learning Object Guide (iLOG), involves real-time tracking of each user sessions (an LO Wrapper), offline data mining to identify key attributes or patterns on how the LOs have been used as well as characteristics of the users (MetaGen), and the selection of these findings as metadata. Mechanisms used in the data mining include data imputation via clustering, association rule mining, and feature selection ensemble. This paper describes the methodology of automatic annotation, presents the results on the evaluation and validation of the algorithms, and discusses the resulting metadata. We have deployed our iLOG implementation for five LOs in introductory computer science topics and collected data for over 1400 sessions. We demonstrate that iLOG successfully tracks user interactions that can be used to automate the generation of meaningful empirical usage metadata for different stakeholder groups including learners and instructors, LO developers, and researchers.

The authors present the XAR framework that allows for free text information extraction and semantic annotation. The language underpinning XAR, the authors argue, allows for the inclusion of probabilistic reasoning with the rule language, provides higher level predicates capturing text features and relationships, and defines and supports advanced features such as token consumption and stratified negotiation in the rule language and semantics. The XAR framework also allows the incorporation of semantic information as integrity constraints in the extraction and annotation process. The XAR framework aims to fill in a gap, the authors claim, in the Web based information extraction systems. XAR provides an extraction and annotation framework by permitting the integrated use of hand-crafted extraction rules, machine-learning based extractors, and semantic information about the particular domain of interest. The XAR system has been deployed in an emergency response scenario with civic agencies in North America and in a scenario with an IT department of a county level community clinic.

We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.

Learning commonsense causal and temporal relation between events is one of the major steps towards deeper language understanding. This is even more crucial for understanding stories and script learning. A prerequisite for learning scripts is a semantic framework which enables capturing rich event structures. In this paper we introduce a novel semantic annotation framework, called Causal and Temporal Relation Scheme (CaTeRS), which is unique in simultaneously capturing a comprehensive set of temporal and causal relations between events. By annotating a total of 1,600 sentences in the context of 320 five-sentence short stories sampled from ROCStories corpus, we demonstrate that these stories are indeed full of causal and temporal relations. Furthermore, we show that the CaTeRS annotation scheme enables high inter-annotator agreement for broad-coverage event entity annotation and moderate agreement on semantic link annotation.

In recent years, the ever-increasing quantities of entities in large knowledge bases on the Web, such as DBpedia, Freebase and YAGO, pose new challenges but at the same time open up new opportunities for intelligent information access. These knowledge bases (KBs) have become valuable resources in many research areas, such as natural language processing (NLP) and information retrieval (IR). Recently, almost every major commercial Web search engine has incorporated entities into their search process, including Google’s Knowledge Graph, Yahoo!’s Web of Objects and Microsoft’s Satori Graph/Bing Snapshots. The goal is to bridge the semantic gap between natural language text and formalized knowledge. Within the context of globalization, multilingual and cross-lingual access to information has emerged as an issue of major interest. Nowadays, more and more people from different countries are connecting to the Internet, in particular the Web, and many users can understand more than one language. While the diversity of languages on the Web has been growing, for most people there is still very little content in their native language. As a consequence of the ability to understand more than one language, users are also interested in Web content in other languages than their mother tongue. There is an impending need for technologies that can help in overcoming the language barrier for multilingual and cross-lingual information access. In this thesis, we face the overall research question of how to allow for semantic-aware and cross-lingual processing of Web documents and user queries by leveraging knowledge bases. With the goal of addressing this complex problem, we provide the following solutions: (1) semantic annotation for addressing the semantic gap between Web documents and knowledge; (2) semantic search for coping with the semantic gap between keyword queries and knowledge; (3) the exploitation of cross-lingual semantics for overcoming the language barrier between natural language expressions (i.e., keyword queries and Web documents) and knowledge for enabling cross-lingual semantic annotation and search. We evaluated these solutions and the results showed advances beyond the state-of-the-art. In addition, we implemented a framework of cross-lingual semantic annotation and search, which has been widely used for cross-lingual processing of media content in the context of our research projects.

Component-oriented software development has been attracting increasing attention for building complex distributed applications. A new infrastructure supporting this advanced concept is our prototype component framework based on the Grid component model. This paper provides an overview of the component framework and presents a case study where we utilise the component-oriented approach to develop a business process application for a biometric identification system. We then introduce the tools being developed as part of an integrated development environment to enable graphical component-based development of Grid applications. Finally, we report our initial findings and experiences of efficiently using the component framework and set of software tools.

In central fields of software engineering, there are two competing directions of research.Component frameworks and advanced programming languages both seek to improve fundamental software quality properties, most notably: modularity.While both directions have produced specific benefits, harvesting these benefits still requires a trade-off in selecting one of the two technologies.In this paper, we present the integration of the aspect-oriented programming language ObjectTeams/Java into the OSGi component framework.By combining these technologies, the design space for modular architectures of components and applications is enriched with fundamentally new options.As a result, the best of both worlds is available for re-using software modules at realistic scales and for evolving systems over significant periods of time.

Smart TV media content can be embedded with available information from the Internet and shared via the Social Web. An increasing number of social videos are now available as well as traditional digital videos such as TV programs and video on demand (VOD). However, it is difficult to find relevant content due to a lack of semantic content. Therefore, there is a great need for multimedia content analysis techniques. In this work, we propose a framework of collaborative semantic video annotation using consensus-based social network. The collaborative video annotation process is organized via social networking. The media content is shared with friends of friend who collaboratively annotate it. We use ontologies to semantically describe the media content and share the media content amongst the users. A consensus choice is applied to conciliate conflicts on annotation information among the participants. According to our experiments, a consensus method is an effective approach that can be used to solve conflicts in a collaborative annotation.

A Semantic-enhanced Augmented Reality Tool for OpenStreetMap POI Discovery

Purpose This study aimed to develop an implementation framework for remanufacturing indirect building components based on the key factors and job competencies of remanufacturing. The key factors constitute the challenges and opportunities that must be addressed, while the job competencies represent skills, characteristics, and behaviors required to execute tasks successfully. Design/methodology/approach This study proceeded in four stages. First, literature labeling identified the potential remanufacturing factors. Second, the Delphi method screened and validated these factors, producing a prioritized set of key factors, and those that required specific job competencies were determined. Third, these results were consolidated to develop a remanufacturing implementation framework for indirect building components. Fourth, case studies were done to evaluate the framework. The approach yielded a validated factor list, a competency model of hard and soft skills, and empirical evidence of the framework’s feasibility and practical utility. Findings The findings from the case studies showed that the implementation framework provides valuable guidance for companies. Its inclusive design accommodates organizations of varying sizes, enabling both large and small-to-medium companies to develop effective operating models. The concept of job competencies offers a valuable reference for corporate talent allocation while allowing for application-specific adjustments. The managers of the case companies praised the framework for its user-friendly interface, clear structural design, and insightful presentation. The framework’s combination of graphics, text, and directional arrows enhances concept comprehension, demonstrating accessibility and practicality for both specialists and general users. Originality/value This study has systematically identified 13 key factors for remanufacturing execution. In order to overcome complex technical aspects of remanufacturing, the concept of job competencies has been introduced. The combination of these two elements has created a novel and comprehensive remanufacturing implementation framework for indirect building components. The framework provides the foundation for achieving the sustainability goals of a built environment, transforming sustainability from a vision into actionable norms.

Dramatic expansion and eminence of the multimedia data from the last decades, culminates to a trouble in managing, accessing and annotating the data. The high level semantic annotation (HLS) of resources in general and multimedia resources in particular, is a resilient job. The Progression in automatic annotation mechanisms have not been able to comprehend with adequately accurate results. To outfit multimedia (e.g. image/video) retrieval capabilities, digital libraries have hung on manual annotation of images. Providing a track to enact high level semantic annotation automatically would be more worthwhile, efficient and scalable with magnifying image collections. This paper intent to equip the high level semantic annotation for images, and consequently, contributes to 1) calculating semantic intensity (SI) of each object in the image depicting the dominancy factor, (2) image similarity on the bases on metadata tag with the images, and (3) clustering approach based on the image similarity to tag set of images with a high level semantic description with their calculated similarity values. The experiment on a portion of randomly selected images from LabelMe database manifests stimulating outcomes.

Metadata extraction from natural history collection labels is a pivotal task for the online publication of digitized specimens. However, given the scale of these collections-which are estimated to host more than 2 billion specimens worldwide, including ca. 400 million herbarium specimens-manual metadata extraction is an extremely time-consuming task. Thus, automated data extraction from digital images of specimens and their labels therefore is a promising application of state-of-the-art computer vision techniques. Extracting information from herbarium specimen labels normally involves three main steps: text segmentation, multilingual and handwriting recognition, and data parsing. The primary bottleneck in this workflow lies in the limitations of Optical Character Recognition (OCR) systems. This study explores how the general knowledge embedded in multimodal Transformer models can be transferred to the specific task of herbarium specimen label digitization. The final goal is to develop an easy-to-use, end-to-end solution to mitigate the limitations of classic OCR approaches while offering greater flexibility to adapt to different label formats. Donut-base, a pre-trained visual document understanding (VDU) transformer, was the base model selected for fine-tuning. A dataset from the University of Pisa served as a test bed. The initial attempt achieved an accuracy of 85%, measured using the Tree Edit Distance (TED), demonstrating the feasibility of fine-tuning for this task. Cases with low accuracies were also investigated to identify limitations of the approach. In particular, specimens with multiple labels, especially if combining handwritten and typewritten text, proved to be the most challenging. Strategies aimed at addressing these weaknesses are discussed.

Social media provides a fertile ground for any user to find or share information about various events with others. At the same time, social media is not always used for benign purposes. With the availability of inexpensive and ubiquitous mass communication tools, disseminating false information and propaganda is both convenient and effective. In this research, we studied Online Deviant Groups (ODGs) that conduct cyber propaganda campaigns in order to achieve strategic and political goals, influence mass thinking, and steer behaviors or perspectives about an event. We provide case studies in which various disinformation and propaganda swamped social media during two NATO exercises in 2015. We demonstrate ODGs’ capability to spread anti-NATO propaganda using a highly sophisticated and well-coordinated social media campaign. In particular, blogs were used as virtual spaces where narratives are framed. And, to generate discourse, web traffic was driven to these virtual spaces via other social media platforms such as Twitter, Facebook, and VKontakte. By further examining the information flows within the social media networks, we identify sources of mis/disinformation and their reach, i.e., how far and how quickly the mis/disinformation could travel and consequently detect manipulation. The chapter presents an in-depth examination of the information networks using social network analysis (SNA) and social cyber forensics (SCF) based methodologies to identify prominent information brokers, leading coordinators, and information competitors who seek to further their own agenda. Through SCF tools, e.g., Maltego, we extract metadata associated with disinformation-riddled websites. The extracted metadata helps in uncovering the implicit relations among various ODGs. We further collected the social network of various ODGs (i.e., their friends and followers) and their communication network (i.e., network depicting the flow of information such as tweets, retweets, mentions, and hyperlinks). SNA helped us identify influential users and powerful groups responsible for coordinating the various disinformation campaigns. One of the key research findings is the vitality of the link between blogs and other social media platforms to examine disinformation campaigns.

The exploding number of computational models produced by Systems Biologists over the last years is an invitation to structure and exploit this new wealth of information. Researchers would like to trace models relevant to specific scientific questions, to explore their biological content, to align and combine them, and to match them with experimental data. To automate these processes, it is essential to consider semantic annotations, which describe their biological meaning. As a prerequisite for a wide range of computational methods, we propose general and flexible similarity measures for Systems Biology models computed from semantic annotations. By using these measures and a large extensible ontology, we implement a platform that can retrieve, cluster, and align Systems Biology models and experimental data sets. At present, its major application is the search for relevant models in the BioModels Database, starting from initial models, data sets, or lists of biological concepts. Beyond similarity searches, the representation of models by semantic feature vectors may pave the way for visualisation, exploration, and statistical analysis of large collections of models and corresponding data.

Tools and software that automate repetitive tasks, such as metadata extraction and deposition to data repositories, are essential for researchers to share Open Data, routinely. For research that generates microscopy image data, OMERO is an ideal platform for storage, annotation and publication according to open research principles. We present PyOmeroUpload, a Python toolkit for automatically extracting metadata from experiment logs and text files, processing images and uploading these payloads to OMERO servers to create fully annotated, multidimensional datasets. The toolkit comes packaged in portable, platform-independent Docker images that enable users to deploy and run the utilities easily, regardless of Operating System constraints. A selection of use cases is provided, illustrating the primary capabilities and flexibility offered with the toolkit, along with a discussion of limitations and potential future extensions. PyOmeroUpload is available from: https://github.com/SynthSys/pyOmeroUpload.