An ERP study of continuous speech processing: I. Segmentation, semantics, and syntax in native speakers

Semantic information and syntactic information are two essential dimensions of language. In recent years, most researches focus on extracting semantic information or how to extract useful information from syntactic dependency tree. As the semantic representations are closely related to syntactic ones, how to effectively integrate semantic and syntactic information is still a subject worth studying. We propose attention mechanism based on graph convolutional networks. The model integrates syntactic dependence information into semantic features and automatically achieve a balance between semantic information and syntactic information. We conducted experiments on the task of sentence-level relation extraction, and the results show that our model performs better than some previous methods.

Auditory event-related potentials index faster processing of natural speech but not synthetic speech over nonspeech analogs in children

Time effects on event-related brain potentials and vigilance performance

Listeners interpret cues in speech processing immediately rather than waiting until the end of a sentence. In particular, prosodic cues in auditory speech processing can aid listeners in building information structure and contrast sets. Native speakers even use this information in combination with syntactic and semantic information to build mental representations predictively. Research on second-language (L2) learners suggests that learners have difficulty integrating linguistic information across various domains, likely subject to L2 proficiency levels. The current study investigated eye-movement behavior of native speakers of English and Chinese learners of English in their use of contrastive intonational cues to restrict the set of upcoming referents in a visual world paradigm. Both native speakers and learners used contrastive pitch accent to restrict the set of referents. Whereas native speakers anticipated the upcoming set of referents, this was less clear in the L2 learners. This suggests that learners are able to integrate information across multiple domains to build information structure in the L2 but may not do so predictively. Prosodic processing was not affected by proficiency or working memory in the L2 speakers.

The main goal of monolingual models of sentence processing is to explain how the syntactic processor (or parser) assigns structure to an incoming string of words. The theoretical divide in the field has been about whether the architecture and mechanisms of the human sentence processor are modular—and computations are carried out serially—or whether it is interactive and computations are carried out in parallel. The debate about bilingual sentence processing has, instead, focused on whether bilingual speakers process their second language in a manner similar to monolingual speakers of the target language. Proposals rooted in generative approaches to language acquisition argue that adult second language (L2) learners lack access to the universal principles or the ability to reset parameters that guide language acquisition and language processing in their L2. Models grounded in neurocognitive approaches to memory hold that late bilinguals recruit different memory systems compared to native speakers of the target language. Other models have argued that differences in first and second language processing result largely from capacity differences, differences in susceptibility to interference, or lack of predictive ability. More recently, several studies have turned toward more experience-driven accounts, eschewing the earlier assumption that the first language is static and unchanging, and instead focusing on the interactive and interconnected nature of the bilingual linguistic system. These studies have revealed that not only does the first language (L1) affect L2 syntactic processing, but experience with the L2 can have ramifications for processing in the native language. A range of experimental techniques are employed to investigate how monolingual and bilingual speakers process language at the sentence level. Eye-tracking techniques allow measurement of responses, such as eye movements and pupil dilation, to study written and auditory language processing. Such measures permit insight into the cognitive processes that are engaged when individuals read written text or inspect visual scenes. Electrophysiological measures are particularly helpful for understanding the time course of neural activity associated with language and cognitive processes. Event-related potentials (ERPs) are obtained by recording and averaging across brain potentials associated with time-locked events (e.g., a word in a sentence). Electrophysiological measures are used to determine which stages of processing are affected by the experimental manipulation. Neuroimaging provides information about changes in brain structure and function. For example, the functional magnetic resonance imaging (fMRI) technique can be used to visualize which brain regions are engaged in processing a particular type of sentence. Although neuroimaging is a relatively-new methodology, it holds great promise for increasing our understanding of the dynamic processes in the brain related to language. The writing of this bibliography was supported in part by National Science Foundation (NSF) grant BCS-1535124 and OISE 1545900 to Paola E. Dussias.

Previous research using picture/word matching tasks has demonstrated a tendency to incorrectly interpret phrasally stressed strings as compounds. Using event-related potentials, we sought to determine whether this pattern stems from poor perceptual sensitivity to the compound/phrasal stress distinction, or from a post-perceptual bias in behavioral response selection. A secondary aim was to gain insight into the role played by contrastive stress patterns in online sentence comprehension. The behavioral results replicated previous findings of a preference for compounds, but the electrophysiological data suggested a robust sensitivity to both stress patterns. When incongruent with the context, both compound and phrasal stress elicited a sustained left-lateralized negativity. Moreover, incongruent compound stress elicited a centro-parietal negativity (N400), while incongruent phrasal stress elicited a late posterior positivity (P600). We conclude that the previous findings of a preference for compounds are due to response selection bias, and not a lack of perceptual sensitivity. The present results complement previous evidence for the immediate use of meter in semantic processing, as well as evidence for late interactions between prosodic and syntactic information.

In tonal languages, such as Mandarin Chinese, the pitch contour of vowels discriminates lexical meaning, which is not the case in non-tonal languages such as German. Recent data provide evidence that pitch processing is influenced by language experience. However, there are still many open questions concerning the representation of such phonological and language-related differences at the level of the auditory cortex (AC). Using magnetoencephalography (MEG), we recorded transient and sustained auditory evoked fields (AEF) in native Chinese and German speakers to investigate language related phonological and semantic aspects in the processing of acoustic stimuli. AEF were elicited by spoken meaningful and meaningless syllables, by vowels, and by a French horn tone. Speech sounds were recorded from a native speaker and showed frequency-modulations according to the pitch-contours of Mandarin. The sustained field (SF) evoked by natural speech signals was significantly larger for Chinese than for German listeners. In contrast, the SF elicited by a horn tone was not significantly different between groups. Furthermore, the SF of Chinese subjects was larger when evoked by meaningful syllables compared to meaningless ones, but there was no significant difference regarding whether vowels were part of the Chinese phonological system or not. Moreover, the N100m gave subtle but clear evidence that for Chinese listeners other factors than purely physical properties play a role in processing meaningful signals. These findings show that the N100 and the SF generated in Heschl’s gyrus are influenced by language experience, which suggests that AC activity related to specific pitch contours of vowels is influenced in a top-down fashion by higher, language related areas. Such interactions are in line with anatomical findings and neuroimaging data, as well as with the dual-stream model of language of Hickok and Poeppel that highlights the close and reciprocal interaction between superior temporal gyrus and sulcus.

Electrophysiological estimates of semantic and syntactic information access during tacit picture naming and listening to words.

Recent studies have shown that certain brain event-related potentials (ERPs) are sensitive to auditory perceptual categorical boundaries. This study investigates brain responses to lexical tone categorization for three groups of adult listeners: (1) native English speakers who had no exposure to Mandarin before age 17, but took advanced Mandarin courses as adults; (2) naïve English speakers; and (3) native Mandarin speakers. Two tonal continua were derived from natural speech through interpolation within two tonal contrasts (Tone 1/Tone 4; Tone 2/Tone 3). First, category boundaries were examined through classic identification and discrimination tasks. Secondly, high-density electroencephalography (EEG) was used to record brain responses while participants listened to tones in two oddball paradigms: across-category and within-category. If perception of lexical tones is categorical, cross-category deviants are expected to elicit larger ERP responses (specifically, mismatch negativity (MMN) and P300) than within-category deviants. Both behavioral and ERP results indicate that lexical tones are perceived categorically by native Chinese speakers but not by inexperienced English speakers. Although English learners of Chinese demonstrated categorical perception in behavioral tasks, their ERP response amplitudes were attenuated, and did not differ between within- and across-category conditions. Acoustic cues and characteristics of L2 phonological learning in adulthood are discussed.

ALSC (Aspect-level Sentiment Classification) is a fine-grained task in the field of NLP (Natural Language Processing) which aims to identify the sentiment toward a given aspect. In addition to exploiting the sentence semantics and syntax, current ALSC methods focus on introducing external knowledge as a supplementary to the sentence information. However, the integration of the three categories of information is still challenging. In this paper, a novel method is devised to effectively combine sufficient semantic and syntactic information as well as use of external knowledge. The proposed model contains a sentence encoder, a semantic learning module, a syntax learning module, a knowledge enhancement module, an information fusion module and a sentiment classifier. The semantic information and syntactic information are respectively extracted via a self-attention network and a graphical convolutional network. Specifically, the KGE (Knowledge Graph Embedding) is employed to enhance the feature representation of the aspect. Then, the attention-based gate mechanism is taken to fuse three types of information. We evaluated the proposed model on three benchmark datasets and the experimental results establish strong evidence of high accuracy.

International audience

Sentence classification is a significant task in natural language processing (NLP) and is applied in many fields. The syntactic and semantic properties of words and phrases often determine the success of sentence classification. Previous approaches based on sequential modeling mainly ignored the explicit syntactic structures in a sentence. In this paper, we propose a Syntax-Aware Transformer (SA-Trans), which integrates syntactic information in the transformer and obtains sentence embeddings by combining syntactic and semantic information. We evaluate our SA-Trans on four benchmark classification datasets (i.e., AG’News, DBpedia, ARP, ARF), and the experimental results manifest that our SA-Trans model achieves competitive performance compared to the baseline models. Finally, the case study further demonstrates the importance of syntactic information for the classification task.KeywordsSyntaxTransformerSentence classification

Most parts of speech are voiced, exhibiting a degree of periodicity with a fundamental frequency and many higher harmonics. Some neural populations respond to this temporal fine structure, in particular at the fundamental frequency. This frequency-following response to speech consists of both subcortical and cortical contributions and can be measured through EEG as well as through magnetoencephalography (MEG), although both differ in the aspects of neural activity that they capture: EEG is sensitive to both radial and tangential sources as well as to deep sources, whereas MEG is more restrained to the measurement of tangential and superficial neural activity. EEG responses to continuous speech have shown an early subcortical contribution, at a latency of around 9 msec, in agreement with MEG measurements in response to short speech tokens, whereas MEG responses to continuous speech have not yet revealed such an early component. Here, we analyze MEG responses to long segments of continuous speech. We find an early subcortical response at latencies of 4-11 msec, followed by later right-lateralized cortical activities at delays of 20-58 msec as well as potential subcortical activities. Our results show that the early subcortical component of the FFR to continuous speech can be measured from MEG in populations of participants and that its latency agrees with that measured with EEG. They furthermore show that the early subcortical component is temporally well separated from later cortical contributions, enabling an independent assessment of both components toward further aspects of speech processing.

The goal of this study was to examine the degree to which age-related differences in early or automatic levels of auditory processing and attention-related processes explain age-related differences in auditory temporal processing. We hypothesized that age-related differences in attention and cognition compound age-related differences at automatic levels of processing, contributing to the robust age effects observed during challenging listening tasks. We examined age-related and individual differences in cortical event-related potential (ERP) amplitudes and latencies, processing speed, and gap detection from 25 younger and 25 older adults with normal hearing. ERPs were elicited by brief silent periods (gaps) in an otherwise continuous broadband noise and were measured under two listening conditions, passive and active. During passive listening, participants ignored the stimulus and read quietly. During active listening, participants button pressed each time they detected a gap. Gap detection (percent detected) was calculated for each gap duration during active listening (3, 6, 9, 12, and 15 msec). Processing speed was assessed using the Purdue Pegboard Test and the Connections Test. Repeated measures analyses of variance assessed effects of age on gap detection, processing speed, and ERP amplitudes and latencies. An "attention modulation" construct was created using linear regression to examine the effects of attention while controlling for age-related differences in auditory processing. Pearson correlation analyses assessed the extent to which attention modulation, ERPs, and processing speed predicted behavioral gap detection. Older adults had significantly poorer gap detection and slower processing speed than younger adults. Even after adjusting for poorer gap detection, the neurophysiological response to gap onset was atypical in older adults with reduced P2 amplitudes and virtually absent N2 responses. Moreover, individual differences in attention modulation of P2 response latencies and N2 amplitudes predicted gap detection and processing speed in older adults. That is, older adults with P2 latencies that decreased and N2 amplitudes that increased with active listening had faster processing speed and better gap detection than those older adults whose P2 latencies increased and N2 amplitudes decreased with attention. The results from the present study are broadly consistent with previous findings that older adults exhibit significantly poorer gap detection than younger adults in challenging tasks. Even after adjusting for poorer gap detection, older and younger adults showed robust differences in their electrophysiological responses to sound offset. Furthermore, the degree to which attention modulated the ERP was associated with individual variation in measures of processing speed and gap detection. Taken together, these results suggest an age-related deficit in early or automatic levels of auditory temporal processing and that some older adults may be less able to compensate for declines in processing by attending to the stimulus. These results extend our previous findings and support the hypothesis that age-related differences in cognitive or attention-related processing, including processing speed, contribute to an age-related decrease in gap detection.

Neuropsychological studies in brain-injured patients with aphasia and children with specific language-learning deficits have shown the dependence of language comprehension on auditory processing abilities, i.e. the detection of temporal order. An impairment of temporal-order perception can be simulated by time reversing segments of the speech signal. In our study, we investigated how different lengths of time-reversed segments in speech influenced comprehension in ten native German speakers and ten participants who had acquired German as a second language. Results show that native speakers were still able to understand the distorted speech at segment lengths of 50 ms, whereas non-native speakers only could identify sentences with reversed intervals of 32 ms duration. These differences in performance can be interpreted by different levels of semantic and lexical proficiency. Our method of temporally-distorted speech offers a new approach to assess language skills that indirectly taps into lexical and semantic competence of non-native speakers.