Multi-grained Spatio-Temporal Features Perceived Network for Event-based Lip-Reading

User performance in virtual environments with degraded visual conditions due to low frame rates is an interesting area of inquiry. Visual content shown in a low frame rate simulation has the quality of the original image, but persists for an extended period until the next frame is displayed (so-called high persistence-HP). An alternative, called low persistence (LP), involves displaying the rendered frame for a single display frame and blanking the screen while waiting for the next frame to be generated. Previous research has evaluated the usefulness of the LP technique in low frame rate simulations during a static target acquisition task. To gain greater knowledge about the LP technique, we have conducted a user study to evaluate user performance and learning during a dynamic target acquisition task. The acquisition task was evaluated under a high frame rate, (60 fps) condition, a traditional low frame rate HP condition (10 fps), and the experimental low frame rate LP technique. The task involved the acquisition of targets moving along several different trajectories, modeled after a shotgun trap shooting task. The results of our study indicate the LP condition approaches high frame rate performance within certain classes of target trajectories. Interestingly we also see that learning is consistent across conditions, indicating that it may not always be necessary to train under a visually high frame rate system to learn a particular task. We discuss implications of using the LP technique to mitigate low frame rate issues as well as its potential usefulness for training in low frame rate virtual environments.

Cameras are appealing sensors for mobile robots because they are small, passive, inexpensive and provide rich information about the environment. While cameras have been used successfully on a plenitude of robots, such as autonomous cars or drones, serious challenges remain: power consumption, latency, dynamic range, and frame rate, among others. The sequences of images acquired by a camera are highly redundant (both in space and time), and both acquiring and processing such an amount of data consumes significant power. This limits the operation time of mobile robots and, moreover, defines a fundamental power-latency tradeoff. Specialized cameras designed for high-speed or high-dynamic-range scenarios are expensive, heavy, and require additional power, which prevents their use in agile mobile robots. In this thesis, we investigate event cameras as a biologically-inspired alternative to overcome the limitations of standard cameras. These neuromorphic vision sensors work in a completely different way: instead of providing a sequence of images (i.e., frames) at a constant rate, event cameras transmit only information from those pixels that undergo a significant brightness change. These pixel-level brightness changes, called events, are timestamped with micro-second resolution and transmitted asynchronously at the time they occur. Hence, event cameras are power efficient because they convey only non-redundant information, and are able to capture very high-speed motions, thus they directly address the power-latency tradeoff. Additionally, event cameras achieve a dynamic range of more than 140dB, compared to about 60dB of standard cameras, because each pixel is autonomous and operates at its own set-point. However, since the output of an event camera is fundamentally different from that of standard cameras for which computer-vision algorithms have been developed during the past fifty years, new algorithms that can deal with the asynchronous nature of the sensor and exploit its high temporal resolution are required to unlock its potential. This thesis presents algorithms for using event cameras in the context of robotics. Since event cameras are novel sensors that are being intensively prototyped and have been commercially available only recently (ca. 2008), the literature on event-based algorithms is scarce. This poses some operational challenges as well as uncountable opportunities to explore in research. This thesis focuses on exploring the possibilities that event cameras bring to some fundamental problems in robotics and computer vision, such as localization and actuation. Amongst others, this thesis provides contributions to solving the localization problem, i.e., for a robot equipped with an event camera to be able to infer its location with respect to a given map of the environment. Classical approaches for robot localization build upon lower-level vision algorithms, and so, this thesis also presents contributions in the topics of detection, extraction, and tracking of salient visual features with an event camera, whose applicability expands far beyond the localization problem. This thesis also presents contributions in the use of event cameras for actuation and closed-loop control, i.e., in endowing the robot with the capabilities to interact with the environment to fulfill a given task. Additionally, this thesis also presents the infrastructure developed to work with event cameras in a de-facto standard robotics platform. The following is a list of contributions: * Software infrastructure, consisting of publicly available drivers, calibration tools, sensor delay characterization, and the first event camera dataset and simulator tailored for 6-DOF (degrees of freedom) camera pose estimation and SLAM (Simultaneous localization and mapping). * We introduce the concept of event and provide an algorithm to compute it. The lifetime endows the events with a finite temporal extent for a proper continuous representation of events in time. * The first method to extract FAST-like visual features (i.e., interest points or corners) from the output of an event camera. The detector operates an order of magnitude faster than previous corner detectors. * The first method to extract and track features from the output of a DAVIS camera (an event camera that also outputs standard frames from the same pixel array). Using these feature tracks, we developed the first sparse, feature-based visual-odometry pipeline. * The first two methods to track the 6-DOF pose of an event camera in a known map.While the first method minimizes the reprojection error of the events and only works on black-and-white scenes consisting of line segments, the second method uses a probabilistic filtering framework that allows tracking at high speeds on natural scenes. * The first application of a continuous-time framework to estimate the trajectory of an event camera, possibly incorporating inertial measurements, showing superior performance over pose-tracking-only methods. * An application of event cameras to collision avoidance of a quadrotor, showing how event cameras can be used to control a robot with very low latency. * An application of the use of an event camera for human-vs-machine slot-car racing, showing that event-driven algorithms are power efficient and can outperform human control.

The frame rate (FR) of a video plays an important role in affecting the perceptual video quality. Most studies about the effect of FR on the video quality mainly focused on low frame rate, e.g. less than 30 frames per second (fps), at low resolutions like CIF or QCIF. As the video frame rate and resolution advance, we reconsider this issue and investigate the relationship between frame rate and the perceptual video quality under high frame rate and high resolution. In this paper, we discuss the impact of frame rate on the perceptual quality of High Definition (HD) video with high frame rates (up to 120 fps) considered. Firstly, we design and conduct subjective experiment to construct the video dataset, which includes video sequences at different frame rates and the corresponding mean opinion scores (MOS) which represent the perceptual video quality. Based on the MOS results, we analyze how perceptual video quality changes as frame rate varies among different video sequences and propose some meaningful findings. The video dataset will be made publicly available. We deem that this study will enrich video quality assessment and benefit the development of high frame rate and high definition video business.KeywordsVideo quality assessmentSubjective experienceHigh frame rateHigh definition

The analysis of knee kinematics and kinematics during high-risk movements associated with anterior cruciate ligament (ACL) injury is essential to studying the mechanism of ACL injury. Motion capture at low frame rates may not always detect the actual peak values during high-speed movements. However, the knee kinetic differences between various frame rate measurements during high-risk movements have not been reported. The purpose of this study was to investigate whether 3D knee kinetics would be measured differently between frame rates such as a high frame rate (1200 Hz), 400 Hz, 240 Hz and the popular low frame rate (120 Hz). Knee kinetics during a single-leg drop landing and side-step cutting under different frame rates were repeatedly measured and statistically compared. Peak knee valgus and tibial internal rotation moments measured at a popular low frame rate were significantly lower than those measured at the other frame rates of 400 Hz and 1200 Hz. The peak anterior and superior forces measured at the highest (1200 Hz) frame rate were significantly higher than those measured at any other frame rates. In addition, the variations in the peak kinetic values were significantly larger at the lowest frame rate capture and trended toward being smaller at the higher frame rates. In conclusion, significant differences in the knee kinetics between the frame rates suggest that the high frequency capturing increases the accuracy of the knee kinetics measurement for high risk maneuvers for the study of ACL injury.

We present a new approach to the source separation problem in the case of multiple speech signals. The method is based on the use of automatic lip reading: the objective is to extract an acoustic speech signal from other acoustic signals by exploiting its coherence with the speaker's lip movements. For this aim, a statistical model is used to quantify this coherence. The results, while very preliminary, are encouraging. They show that this method can achieve a good separation of a speech source in the case of simple 2/spl times/2 additive mixtures. Moreover, it presents some interesting complementarity with traditional pure audio techniques.

Light field videos captured in RGB frames (RGB-LFV) can provide users with a 6 degree-of-freedom immersive video experience by capturing dense multi-subview video. Despite its potential benefits, the processing of dense multi-subview video is extremely resource-intensive, which currently limits the frame rate of RGB-LFV (i.e., lower than 30 fps) and results in blurred frames when capturing fast motion. To address this issue, we propose leveraging event cameras, which provide high temporal resolution for capturing fast motion. However, the cost of current event camera models makes it prohibitive to use multiple event cameras for RGB-LFV platforms. Therefore, we propose EV-LFV, an event synthesis framework that generates full multi-subview event-based RGB-LFV with only one event camera and multiple traditional RGB cameras. EV-LFV utilizes spatial-angular convolution, ConvLSTM, and Transformer to model RGB-LFV's angular features, temporal features, and long-range dependency, respectively, to effectively synthesize event streams for RGB-LFV. To train EV-LFV, we construct the first event-to-LFV dataset consisting of 200 RGB-LFV sequences with ground-truth event streams. Experimental results demonstrate that EV-LFV outperforms state-of-the-art event synthesis methods for generating event-based RGB-LFV, effectively alleviating motion blur in the reconstructed RGB-LFV.

Most consumer cameras use rolling shutter (RS) exposure, the captured videos often suffer from distortions (e.g., skew and jelly effect). Also, these videos are impeded by the limited bandwidth and frame rate, which inevitably affect the video streaming experience. In this paper, we excavate the potential of event cameras as they enjoy high temporal resolution. Accordingly, we propose a framework to recover the global shutter (GS) high frame rate (i.e., slow motion) video without RS distortion from an RS camera and event camera. One challenge is the lack of real-world datasets for supervised training. Therefore, we explore self-supervised learning with the key idea of estimating the displacement field-a non-linear and dense 3D spatiotemporal representation of all pixels during the exposure time. This allows for a mutual reconstruction between RS and GS frames and facilitates slow-motion video recovery. We then combine the input RS frames with the DF to map them to the GS frames (RS-to-GS). Given the under-constrained nature of this mapping, we integrate it with the inverse mapping (GS-to-RS) and RS frame warping (RS-to-RS) for self-supervision. We evaluate our framework via objective analysis (i.e., quantitative and qualitative comparisons on four datasets) and subjective studies (i.e., user study). The results show that our framework can recover slow-motion videos without distortion, with much lower bandwidth (94% drop) and higher inference speed ($ 16\; {\rm ms}/{\rm frame}$16 ms / frame ) under $32 \times$32× frame interpolation.

Conventional transcranial ultrasound perfusion imaging (UPI) depends on bolus injection and is limited to triggered imaging. To improve our set-ups, we compared two imaging modalities with two different frame rates (FR) and mechanical indices (MI), intending to approach conditions more similar to real time imaging in order to increase parameter precision. Fifteen healthy volunteers were investigated twice with UPI after i. v. administration of 1 ml of SonoVue(R): first, with a high MI (1.6) and a low FR (0.67 Hz)) and second, with a low MI (1.0) and a high FR (5 Hz). Apart from visual analysis, time-intensity curves were calculated from three regions of interest (ROI) and peak intensity (PI), time to PI (TP), and area-under-the-curve (AUC) were compared between the two imaging modalities. Visually, only scarce contrast enhancement was observed in 10/15 probands, and penetration depth was markedly lower at the low MI/high FR setting, while the high MI/low FR setting lead to very intense enhancement in 13/15 individuals. Signal-to-noise-ratio was higher at the low MI/high FR setting. TP was not significantly different between the two set-ups (p > 0.05). PI and AUC were significantly lower at the low MI/high FR setting (p</= 0.001), and both parameters correlated well in corresponding ROIs (p < 0.05 or < 0.01 in 8/9 ROIs). Parameter images reflected the differences of these two semi-quantitative parameters. In patients with excellent bone windows, the new set-up seems to offer highly precise kinetic analysis with an excellent signal-to-noise ratio. In the majority of patients, however, conventional transcranial UPI is limited to high MI and a resulting rather low FR to allow sufficient penetration depth and contrast enhancement.

In accordance with advances in camera technology, the requirement of high-quality video has greatly increased. Among the factors required for high-quality video are a high-resolution and a high frame rate. However, the limitation of the pixel transfer rate restricts compatibility of a high-resolution and a high frame rate in commercial camera. We propose a dual-sensor camera that consists of two cameras: one with a high-resolution and a low frame rate and the other with a low-resolution and a high frame rate. The system is capable of capturing two different image sequences: high-resolution images and high-frame-rate images. A sensor calibration for the dual-sensor camera is also proposed.

Our objective is an audio-visual model for separating a single speaker from a mixture of sounds such as other speakers and background noise. Moreover, we wish to hear the speaker even when the visual cues are temporarily absent due to occlusion. To this end we introduce a deep audio-visual speech enhancement network that is able to separate a speaker's voice by conditioning on both the speaker's lip movements and/or a representation of their voice. The voice representation can be obtained by either (i) enrollment, or (ii) by self-enrollment -- learning the representation on-the-fly given sufficient unobstructed visual input. The model is trained by blending audios, and by introducing artificial occlusions around the mouth region that prevent the visual modality from dominating. The method is speaker-independent, and we demonstrate it on real examples of speakers unheard (and unseen) during training. The method also improves over previous models in particular for cases of occlusion in the visual modality.

In small animal models, two-dimensional (2D) and Doppler echocardiography should provide more information than M-mode, especially in animals with infarcted and distorted left ventricles, but has been limited by low frame rates and poor near field resolution. New, high frame rate echo-Doppler equipment with digital processing was tested for accuracy of measurements. Fourteen normal Wistar rats (232-328 g) were examined under halothane anaesthesia. Pulsed Doppler recordings from both left ventricular outflow tract(LVOT) and right ventricular outflow tract (RVOT) cor-responded well with simultaneous ultrasound transit time measurements of aortic flow (LVOT: v=0.99x+4.8, min R=0.93. Standard error of estimate (SEE)=8.3 ml x min(-1), and RVOT: v=0.97x -4.3. R=0.93. SEE =8.4 ml x min(-1). No systematic differences were observed over a flow range of 20-90 ml x min(-1). Left ventricular (LV) dimensions assessed by 2D parasternal long-axis and short-axis views were equal to M-mode measurements with LV diameter 6.6 + 0.44 mm, anterior wall 1.8 +/- 0.18 mm, and posterior wall 1.5 + 0.56 mm. Mean absolute difference 4.4-8.5%. Intra- and interobserver variability was 4.6 +/- 4.1% and 6.7 +/- 7.0% for Doppler measurements, and 4.3 +/- 3.8% and 3.8 +/- 4.6% for dimensions, respectively. High frame rate Doppler echocardiography provides accurate non-invasive measurements of cardiac structure and function in the rat.

Sir, It was with great interest that we read the article titled “Feasibility study to assess clinical applications of 3T cine MRI coupled with synchronous audio recording during speech in evaluation of velopharyngeal insufficiency in children” by Sagar and Nimkin [1]. We believe dynamic real-time MRI will become a prevalent tool in studying velopharyngeal closure and, when associated with detailed anatomical scans, should provide added clinical information compared to the current imaging modalities, videofluoroscopy and nasendoscopy. In this regard, the authors must be commended for their detailed comparison of real-time MRI with those techniques. However, it is worth highlighting that contrary to what is stated in the article, many studies of velopharyngeal closure using real-time MRI have been published in the imaging literature, most of them with synchronised audio recording. Studies focussing specifically on velopharyngeal closure include but are not limited to those by Beer et al. [2], which compared the technique to X-ray videofluoroscopy, Bae et al. [3] at 3 T, and Scott et al. [4], which included data at both 1.5 and 3 T. The last two studies were carried out with simultaneous audio recording. There are far too many speech studies using real-time MRI to mention in a short letter, although the work conducted at the University of Southern California both on acquisition and analysis [5, 6] is worth highlighting. We strongly encourage the interested reader to refer to the recent review of the field [7]. When studying velopharyngeal closure, the dynamic frame rate is a key issue, and we are concerned that only two frames per second (fps) were used in the study by Sagar and Nimkin [1]. Although there is no doubt this is sufficient to study sustained phonation, it appears to be insufficient for speech studies. All the previously mentioned studies [2–6] were acquired at substantially faster rates (5–25 fps). In our clinical experience, for various speech samples in normal phonation all closure events are detected at rates around 10 fps [8]; however, some are already missed at 5 fps. A lower frame rate will increase both blurring and the number of missed closure events; it could potentially lead to an incorrect diagnosis of velopharyngeal insufficiency if closure is short and not sampled. Higher frame rates (e.g., 30 fps [9] or more than 100 fps [10]) are achievable and can be required for linguistics and co-articulatory events studies. However, they are obtained using non-cartesian acquisitions, which have been recently developed and are not necessarily available on standard scanners. Furthermore, they usually rely on delayed-reconstruction, which is a limiting factor to conduct an interactive clinical speech study as would be desirable for velopharyngeal closure assessment. Real-time MRI of speech is a very active field of research, and new publications in this area are a welcome addition to the body of knowledge. However, a consensus on best practise for acquisition methodology is still to emerge, and until such time, we would recommend newcomers to err on the side of caution and try to obtain the best spatial-temporal resolution compromise that can be achieved with their chosen acquisition method.

Spatiotemporal feature learning for no-reference gaming content video quality assessment

Charge coupled device(CCD) is one kind of photoelectric solid-state imaging device with excellent performance. It has the features of large array and good image quality. Working under very complex driving timing, it usually has low frame frequency and can not meet the demand of high imaging frame rate. The method of equivalent high frame rate imaging technique is introduced based on normal CCD in this paper. Some CCDs can be drived via electrodes separately. To meet the imaging demands of high frame rate, high resolution and high quality of two frame imaging technique for pulse light images, the right kind of CCDs should be adopted. The most important is that the CCD electrodes must be controlled directly and respectively. The charge transfer period is a key point while designing CCD driving timing for separate pulse light integration, charge package transfer and readout for two pulse light images. The validation experiments for the principle of equivalent high frame rate exposure of two pulse light images has been accomplished. It is shown that two pulse light images with interval of about 2 μs can be distinguished while all CCD’s inherent excellent performance are retained. A kind of two framing camera system with simple structure can be developed based on this principle.

The use of flat panel detectors in computed tomography (CT) systems can improve resolution, reduce system cost, and add operational flexibility by combining fluoroscopy and radiography applications within CT systems. However, some prior studies have suggested that flat panel detectors would not perform well in CT applications due to their lack of high dynamic range, lag artifacts, and inadequate frame rate. The purpose of this study was to perform a physical evaluation of a prototype flat panel detector capable of high frame rates and extended dynamic range. The flat panel detector used had a pixel size of 194 microns and a matrix size of 2048x1536. The detector could be configured for several combinations of frame rate and matrix size up to 750 frames per second for a 512x16 matrix size with 4x4 binning. The evaluation was performed in terms of the MTF and DQE as a function of frame rate and exposure at the IEC RQA5 (~75 kVp, 21 mm Al) beam quality. The image lag was evaluated in terms of temporal-frequency dependent transfer function. Offset shift were also evaluated. Preliminary results indicate 0.1 MTF at 0.92 cycles/mm and DQE(0) of approximately 0.8, 0.6, 0.4, and 0.22 at 0.144, 0.065, 0.035, and 0.008 mR per frame exposures. The temporal MTF exhibited a low-frequency drop and a value of 0.5 at the Nyquist frequency. Offset shift was negligible. Considering high frame rate capabilities of the new detector, the results suggest that the detector has potential for use in real-time CT applications including CT angiography.