A Theory of Semantic Information

In recent years, semantic communication has received significant attention from both academia and industry, driven by the growing demands for ultra-low latency and high-throughput capabilities in emerging intelligent services. Nonetheless, a comprehensive and effective theoretical framework for semantic communication has yet to be established. In particular, finding the fundamental limits of semantic communication, exploring the capabilities of semantic-aware networks, or utilizing theoretical guidance for deep learning in semantic communication are very important yet still unresolved issues. In general, the mathematical theory of semantic communication and the mathematical representation of semantics are referred to as semantic information theory. In this paper, we introduce the pertinent advancements in semantic information theory. Grounded in the foundational work of Claude Shannon, we present the latest developments in semantic entropy, semantic rate-distortion, and semantic channel capacity. Additionally, we analyze some open problems in semantic information measurement and semantic coding, providing a theoretical basis for the design of a semantic communication system. Furthermore, we carefully review several mathematical theories and tools and evaluate their applicability in the context of semantic communication. Finally, we shed light on the challenges encountered in both semantic communication and semantic information theory.

Semantic Information Theory (SIT) is concerned with studies in Logic and Philosophy on the use of the term information, “in the sense in which it is used of whatever it is that meaningful sentences and other comparable combinations of symbols convey to one who understands them” (Hintikka, 1970). Notwithstanding the large scope of this description, SIT has primarily to do with the question of how to weigh sentences according to their informative content. The main difference with conventional information theory is that information is not conveyed by an ordered sequence of binary symbols, but by means of a formal language in which logical statements are defined and explained by a semantics. The investigation of SIT concerns two research directions: the axiomatisation of the logical principles for assigning probabilities or similar weighting functions to logical sentences and the relationship between information content of a sentence and its probability.

A Three-Level Information Architecture containing Syntactic Information, Semantic Information and Pragmatic Information is put forward in Comprehensive Information Theory (CIT). From this point of view, spatial data analysis is in cooperation with semantic information and Spatial Data Preprocessing (SDP) is corresponding to syntactic information. However, in many practical applications, SDP based only on syntactic information can not get a good effect. Semantics-based preprocessing may be an effective scheme. RS images mosaic is a typical SDP where optimal mosaic line extraction is the crux. Lots of researches based on syntactic information are effective just for orthophoto maps. In this paper, an overall optimal mosaic line extraction scheme has been addressed for non-Ortho RS images. It is argued that there is no projection error in the projection datum fitted by Ground Control Points (GCPs), or regional main height surface which can be recognized in medium resolution RS images. Based on above reasons, the method suggests that GCPs collecting for precise geometrical correction should be on the main height surface, as well as the mosaic line extracting for RS images mosaic. Three sheets of CBERS CCD images of Taiyuan are taken as the experimental data. According to the afore-mentioned method, by collecting GCPs in wide riverbeds, all three sheets are rectified to an existing ETM+ mosaic image. And then, the central lines of wide riverbeds in the overlapping areas are extracted as the mosaic line. The experimental result indicates that this method can extract an overall optimal mosaic line and eliminate the visual texture seam-line effectively, even for non-Ortho RS images. It concludes that SDP based on semantic information can play a good role in spatial data applications.

In this paper we explore the possibility of giving a justification of the “semantic information” content and measure, in the framework of the recent coalgebraic approach to quantum systems and quantum computation, extended to QFT systems. In QFT, indeed, any quantum system has to be considered as an “open” system, because it is always interacting with the background fluctuations of the quantum vacuum. Namely, the Hamiltonian in QFT always includes the quantum system and its inseparable thermal bath, formally “entangled” like an algebra with its coalgebra, according to the principle of the “doubling” of the degrees of freedom (DDF) between them. This is the core of the representation theory of cognitive neuroscience based on QFT. Moreover, in QFT, the probabilities of the quantum states follow a Wigner distribution, based on the notion and measure of quasiprobability, where regions integrated under given expectation values do not represent mutually exclusive states. This means that a computing agent, either natural or artificial, in QFT, against the quantum Turing machine paradigm, is able to change dynamically the representation space of its computations. This depends on the possibility of interpreting QFT system computations within the framework of category theory logic and its principle of duality between opposed categories, such as the algebra and coalgebra categories of QFT. This allows us to justify and not only to suppose, like in the “theory of strong semantic information” of L. Floridi, the definition of modal “local truth” and the notion of semantic information as a measure of it, despite both measures being defined on quasiprobability distributions.

Semantic information and syntactic information are two essential dimensions of language. In recent years, most researches focus on extracting semantic information or how to extract useful information from syntactic dependency tree. As the semantic representations are closely related to syntactic ones, how to effectively integrate semantic and syntactic information is still a subject worth studying. We propose attention mechanism based on graph convolutional networks. The model integrates syntactic dependence information into semantic features and automatically achieve a balance between semantic information and syntactic information. We conducted experiments on the task of sentence-level relation extraction, and the results show that our model performs better than some previous methods.

Rudolf Carnap (ed by A Shimony) 1978 London : University of California Press xxii + 115 pp price £9 As if entropy did not cause us enough trouble already, between the exponents of statistical mechanics, thermodynamics, physics and information theory, we have here the posthumous publication of some of the not so recent ideas of a mathematical philosopher who agreed with none of the previous work on entropy. The information theorist will look askance at the fact that these two 'essays' (they are by no means easy reading) were written between 1952 and 1954; and the definition of 'semantic information', which would not be accepted by linguists or psychologists, comes from a paper by Bar Hillel and Carnap which was presented to the 1952 symposium on applications of communication theory.

To apply information theory to more areas, the author proposed semantic information G theory, which is a natural generalization of Shannon’s information theory. This theory uses the P-T probability framework so that likelihood functions and truth functions (or membership functions), as well as sampling distributions, can be put into the semantic mutual information formula at the same time. Hence, we can connect statistics and (fuzzy) logic. Rate-distortion function R(D) becomes rate-verisimilitude function R(G) (G is the lower limit of the semantic mutual information) when the distortion function is replaced with the semantic information function. Seven difficult tasks are 1) clarifying the relationship between minimum information and maximum entropy in statistical mechanics, 2) compressing images according to visual discrimination, 3) multilabel learning for obtaining truth functions or membership functions from sampling distributions, 4) feature classifications with maximum mutual information criterion, 5) proving the convergence of the expectation-maximization algorithm for mixture models, 6) interpreting Popper’s verisimilitude and reconciling the contradiction between the content approach and the likeness approach, and 7) providing practical confirmation measures and clarifying the raven paradox. This paper simply introduces the mathematical methods for these tasks and the conclusions. The P-T probability framework and the semantic information G theory should have survived the tests. They should have broader applications. Further studies are needed for combining them with neural networks for machine learning.

This paper outlines a quantitative theory of strongly semantic information (TSSI) based on truth-values rather than probability distributions. The main hypothesis supported in the paper is that the classic quantitative theory of weakly semantic information (TWSI), based on probability distributions, assumes that truth-values supervene on factual semantic information, yet this principle is too weak and generates a well-known semantic paradox, whereas TSSI, according to which factual semantic information encapsulates truth, can avoid the paradox and is more in line with the standard conception of what generally counts as semantic information. After a brief introduction, section two outlines the semantic paradox implied by TWSI, analysing it in terms of an initial conflict between two requisites of a quantitative theory of semantic information. In section three, three criteria of semantic information equivalence are used to provide a taxonomy of quantitative approaches to semantic information and introduce TSSI. In section four, some further desiderata that should be fulfilled by a quantitative TSSI are explained. From section five to section seven, TSSI is developed on the basis of a calculus of truth-values and semantic discrepancy with respect to a given situation. In section eight, it is shown how TSSI succeeds in solving the paradox. Section nine summarises the main results of the paper and indicates some future developments.

ALSC (Aspect-level Sentiment Classification) is a fine-grained task in the field of NLP (Natural Language Processing) which aims to identify the sentiment toward a given aspect. In addition to exploiting the sentence semantics and syntax, current ALSC methods focus on introducing external knowledge as a supplementary to the sentence information. However, the integration of the three categories of information is still challenging. In this paper, a novel method is devised to effectively combine sufficient semantic and syntactic information as well as use of external knowledge. The proposed model contains a sentence encoder, a semantic learning module, a syntax learning module, a knowledge enhancement module, an information fusion module and a sentiment classifier. The semantic information and syntactic information are respectively extracted via a self-attention network and a graphical convolutional network. Specifically, the KGE (Knowledge Graph Embedding) is employed to enhance the feature representation of the aspect. Then, the attention-based gate mechanism is taken to fuse three types of information. We evaluated the proposed model on three benchmark datasets and the experimental results establish strong evidence of high accuracy.

Two theories of accounting accruals form the core of this study. The theory of the functions of accounting accruals is induced from observation of accounting accruals. One type of accounting accrual reflects incomplete transactions under a system of cash receipts and cash disbursements. The function of these accruals is to convey data about economic events associated with incomplete cash transactions. These accruals are referred to as syntactically redundant accruals. The second type of accounting accrual presents messages in a different format than their counterparts in a system of cash receipts and cash disbursements. The function of these accruals is to enable accounting data users to perceive messages that may be unclear when signaled under a system of cash receipts and cash disbursements. These accruals are referred to as semantically redundant accruals. The theory of the functions of accounting accruals states that accounting accruals are syntactically redundant or semantically redundant in function. The theory of the pragmatic information of accounting accruals states that accounting accruals contain pragmatic information because of their functions. A hypothesis associated with these theories asserts that the pragmatic information of accounting accruals decreases as the financial statement time frame increases. Cash-flow and accrual data were obtained from COMPUSTAT and provided to a neural network to generate forecasts of cash flows from operations. The forecasted cash flows were used to calculate an error metric that functioned as a measure of pragmatic information. A variant of the Bonferroni procedure was used to detect differences between means of error metrics and to test research hypotheses. The results indicated that syntactically and semantically redundant accounting accruals contain pragmatic information. Hypothesis tests, however, provided evidence that only annual accounting accruals contain pragmatic information. Hypothesis tests also indicated that no difference exists between the pragmatic information contained in syntactically redundant accounting accruals and the pragmatic information contained in semantically redundant accounting accruals. Finally, the results of hypothesis tests presented no evidence of a negative relationship between the pragmatic information of accounting accruals and the financial statement time frame.

After long arguments between positivism and falsificationism, the verification of universal hypotheses was replaced with the confirmation of uncertain major premises. Unfortunately, Hemple proposed the Raven Paradox. Then, Carnap used the increment of logical probability as the confirmation measure. So far, many confirmation measures have been proposed. Measure F proposed by Kemeny and Oppenheim among them possesses symmetries and asymmetries proposed by Elles and Fitelson, monotonicity proposed by Greco et al., and normalizing property suggested by many researchers. Based on the semantic information theory, a measure b* similar to F is derived from the medical test. Like the likelihood ratio, measures b* and F can only indicate the quality of channels or the testing means instead of the quality of probability predictions. Furthermore, it is still not easy to use b*, F, or another measure to clarify the Raven Paradox. For this reason, measure c* similar to the correct rate is derived. Measure c* supports the Nicod Criterion and undermines the Equivalence Condition, and hence, can be used to eliminate the Raven Paradox. An example indicates that measures F and b* are helpful for diagnosing the infection of Novel Coronavirus, whereas most popular confirmation measures are not. Another example reveals that all popular confirmation measures cannot be used to explain that a black raven can confirm “Ravens are black” more strongly than a piece of chalk. Measures F, b*, and c* indicate that the existence of fewer counterexamples is more important than more positive examples’ existence, and hence, are compatible with Popper’s falsification thought.

Electrical Impedance Tomography (EIT) is a noninvasive imaging technique that utilizes electrical voltage data to reconstruct cross-sectional images of the human body. EIT has diverse applications such as brain imaging and gesture recognition, making it a valuable tool for both medical diagnostics and human-computer interaction. One potential drawback of EIT is the power consumption and computational requirements for image reconstruction, which may limit real-time applications. This paper presents a novel approach that directly applies machine learning to raw EIT voltage data, bypassing the image reconstruction phase to achieve faster and highly accurate gesture classification. We introduce the Statistical Analysis, Information Theory, and Data-Driven (SID) pipeline, which analyzes raw EIT data using statistical techniques, information theory, and feature ranking, followed by classification with machine learning models. Two datasets were collected for this study: one for five gesture recognition (1193 samples) and another for distinguishing between arm flexion and extension (719 samples). The proposed SID pipeline was applied where the gesture recognition and arm flexion/extension feature set was reduced from 40 to 2 and 1, respectively, and XGBoost was used for classification and achieved an accuracy of 90.38% and 100.00%, respectively. The proposed method demonstrated high accuracy in both tasks while using a reduced feature set. This is in addition to by-passing the image reconstruction for EIT, further enhancing computational efficiency and reduced power consumption, highlighting the potential of this approach for real-time applications.

About the Authors

Sentence classification is a significant task in natural language processing (NLP) and is applied in many fields. The syntactic and semantic properties of words and phrases often determine the success of sentence classification. Previous approaches based on sequential modeling mainly ignored the explicit syntactic structures in a sentence. In this paper, we propose a Syntax-Aware Transformer (SA-Trans), which integrates syntactic information in the transformer and obtains sentence embeddings by combining syntactic and semantic information. We evaluate our SA-Trans on four benchmark classification datasets (i.e., AG’News, DBpedia, ARP, ARF), and the experimental results manifest that our SA-Trans model achieves competitive performance compared to the baseline models. Finally, the case study further demonstrates the importance of syntactic information for the classification task.KeywordsSyntaxTransformerSentence classification

Prior literature has debated whether syntax is separable from semantics in the brain. Using functional magnetic resonance imaging and multi-voxel pattern analysis, our previous studies investigated brain activity during morpho-syntactic versus semantic processing. These studies only detected semantic specialization in activation patterns and no syntactic specialization in 5- to 6-year-old and 7- to 8-year-old children. To examine if older children who have mastered morpho-syntactic skills would show specialization for syntax, the current study examined 64 9- to 10-year-old children using the same design and analyses. We observed that only the left IFG pars opercularis was sensitive to syntactic but not semantic information, supporting the hypothesis that this region serves as a core region for syntax. In addition, the left STG which has been implicated in the integration of semantics and syntax, as well as the left MTG and IFG pars triangularis which have been implicated in semantics, were sensitive to both semantic and syntactic information with no evidence of specialization. These findings suggest a lexicalized view of syntax, which argues that semantically sensitive regions are also critical regions for syntactic processing during language comprehension.