A Spatial Prediction-Based Motion-Compensated Frame Rate Up-Conversion

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

Internet of Things (IoT) has progressively grown in its use cases ranging from the day-to-day lives of both individuals to industries. The growing rate on the Internet of Multimedia Things (IoMT) applications has increased efficiency, improved communication, and greater productivity, to mention a few. Still, it comes with higher accretion of security threats. IoMTs are vulnerable due to the various resource constraint (i.e., Low CPU power, limited memory, and low power source). The resource constraints in IoMT prevent highly-ended traditional security systems from being implemented. Edge computing platform has emerged as a solution to mitigate these constraints by enabling computing capabilities and IT service proximal to the IoMT network’s edge. To protect the IoMT network from attacks, various lightweight security systems have emerged as a viable approach for detecting and preventing cyber-attacks. Machine learning-based Network Intrusion Detection System NIDS, lightweight end-to-end encryption and new network security standards are promising for detecting known and unknown attacks. Many security systems proposed are inefficient to be implemented in real application due to their performance overheads and evaluation processes of the underlying methods. This chapter presents different realistic centralized and distributed security approaches utilizing an edge computing platform to secure the IoMT network. Furthermore, a deployment architecture between the cloud, the IoMT edge computing platform, and the IoMT end-device is proposed. The first layer (IoMT end-device) runs a lightweight security system. The second level deployed in the edge computing platform performs extensive attack detection and protection based on the responses from the IoMT. The last layer (cloud) holds all the performance logic and orchestrates security updates and commands. Finally, we demonstrate our proposed deployment strategies in a real-world IoMT system to evaluate its performance. The Long Short Term Memory (LSTM) model deploy to the edge computing platform had an accuracy of 80%, and the lightweight One-Class support vector machine also produced 82% accuracy in the IoMT end-device.

This paper proposes a fast motion estimation algorithm for variable block-sizes by utilizing motion vector bottom-up procedure for H.264. The refined motion vectors of adjacent small blocks are merged to predict the motion vectors of larger blocks for reducing the computation. Experimental results show that our proposed method has lower computational complexity than full search, fast full search and fast motion estimation of the H.264 reference software JM93 with slight quality decrease and little bit-rate increase.

Motion estimation (ME) has the highest computational complexity in motion-compensated frame rate up-conversion (MC-FRUC). For the real-time implementation of FRUC, a fast ME algorithm is required. In this paper, a new hierarchical ME algorithm for MC-FRUC is proposed. It constructs an image pyramid by dividing the frame into several sub-images according to resolution, and performs ME at the top level to reduce complexity while improving accuracy by selecting multiple motion vector candidates. These candidates are refined at the lower levels, and the final motion vector is selected at the bottom level. Thus, the proposed algorithm obtains an average peak signal-to-noise ratio gain of upto 0.85 dB compared to conventional algorithms with lower computational complexity and yields interpolated images with better visual quality than other methods.

With the immersive growth of the Internet of Things (IoT) and real-time adaptability, quality of life for people is improving. IoT applications are diverse in nature and one crucial aspect of it is multimedia sensors and devices. These IoT multimedia devices form the Internet of Multimedia Things (IoMT). It generates a massive volume of data with different characteristics and requirements than the IoT. The real-time deployment scenarios vary from smart traffic monitoring to smart hospitals. Hence, Timely delivery of IoMT data and decision making is critical as it directly involves the safety of human beings. In this paper, we present a brief overview of IoMT and future research directions. Afterward, we provide an overview of the accepted articles in our special issue on the IoMT: Opportunities, Challenges, and Solutions.

Internet of Things (IoT) has widely been accepted as a promising paradigm for connecting a large number of resource-constrained miniature sensor nodes that have the ability to sense the deployed environment. They have found their applications in various aspect of our daily lives. However, these nodes are mostly restricted to sense only the scalar data. Nowadays, multimedia sensor nodes are gaining significant attention due to their ability to collect scalar as well as multi-dimensional data. These nodes are considered as the foundation of Internet of Multimedia Things (IoMT) and are shaping the perception of IoT. Multimedia data have stringent requirements in terms of reliability, latency, storage, bandwidth, and Quality of Service (QoS). To provide seamless and interoperable communication in IoMT, the underlying protocol stacks need to fulfill these stringent requirements. However, the heterogeneous nature of multimedia sensors makes interoperability a challenging task to fulfill. To understand the challenges faced by seamless and interoperable communication in IoMT, we provide a comprehensive review of the existing protocol stacks of IoMT and analyze their feasibility for multimedia streaming applications. Data storage of multimedia applications is another area that requires immediate attention of the research community. For this purpose, we study cloud as an entity to facilitate multimedia applications of IoMT. The instances of multimedia cloud are analyzed and a number of shortcomings are identified that pave the way for edge computing in IoMT. Finally, we present a case study that shows the significance of our work. The case study portrays an in-home patient monitoring system with an interoperable communication among the connected multimedia streaming devices at home, and healthcare practitioners at hospital. The case study also highlights the importance of uninterrupted data storage and retrieval at the network edge and multimedia sensor nodes.

Liquid crystal displays (LCDs), which serve as receivers of high visual quality video, suffer from motion blur issues. One of the methods in terminating motion blur is motion compensated frame rate upconversion, which is widely adopted in LCDs. In the previous work, i.e., particle-based frame rate upconversion, the computation complexity is high while repeated operations and some improper cost evaluation setup are observed. Therefore, in this paper, hybrid frame rate upconversion is proposed with two features. First, the cost evaluation for particle-based motion trajectory calibration is modified based on the possible noise sources and video resolution variations. Second, repeated operations in particle-based motion trajectory calibration are observed. Therefore, original particle-based motion trajectory calibration is replaced by initial motion vector assignment and subsequent motion vector mapping to achieve computation complexity reduction, while the effective search range is relatively expanded. According to the experiment results, the visual quality is enhanced by 1.87 dB on average, compared with state-of-the-art unidirectional-based frame rate upconversion approaches. On the other hand, the computation complexity of the proposed design is reduced by 25-90% based on target video resolution, concluding a high visual quality and low computation complexity frame rate upconversion design.

This paper presents a motion compensated frame rate up-conversion algorithm that uses the expanded range of the motion trajectory to enhance the accuracy of motion estimation. In the proposed algorithm, an adaptive motion vector refinement is also proposed to correct false motion vectors. In addition, a recursive motion vector smoothing is proposed to smooth motion vectors considering true neighboring motion vectors only. Finally, we propose the weighted index-based bidirectional motion compensated interpolation to reduce artifacts. In the experiments using benchmark test sequences, the proposed algorithm improves the average PSNR of interpolated frames by up to 4.84 dB, when compared to the conventional algorithm.

The Internet of Things (IoT) is growing fast and is gaining significant adoption in areas such as smart cities and manufacturing. The variety and low-cost of IoT devices with excellent audio/visual sensors is fueling the growth of multimedia IoT applications, many of which are bandwidth-hungry and time-sensitive (i.e., must produce their results within an application specific time-sensitive requirement). While a large body of related work has studied distribution of such Time Sensitive Multimedia IoT (TS-MIoT) applications in simulated environments, there is lack of a platform that can be used to experiment with techniques for meeting their time-sensitive and computing resource requirements on real-world IoT infrastructure, i.e., a combination of IoT devices, close by computers and a cloud data centre connected by a variety of networks. This paper proposes APOLLO, a platform for experimental analysis of TS-MIoT applications. APOLLO provides mechanisms to load TS-MIoT application execution plans and execute the plans on available IoT infrastructure. We describe a proof-of-concept implementation using Orleans and present experimental evaluations to validate APOLLO's ability to support the experimental analysis of TS-MIoT applications.

In this paper, a novel camera motion descriptor is proposed for video shot classification. In the proposed method, raw motion information of consecutive video frames are extracted by computing the motion vector of each macroblock to form motion vector fields (MVFs). Next, a motion consistency analysis is applied on MVFs to eliminate the inconsistent motion vectors. Then, MVFs are divided into nine (3 × 3) local regions and the singular value decomposition (SVD) technique is applied on the motion vectors extracted from each local region in the temporal direction. Consistent motion vectors of a number of MVFs are compactly represented at a time to characterize temporal camera motion. Accordingly, each local region of the whole video shot is represented using a sequence of compactly represented vectors. Finally, the sequence of vectors is converted into a histogram to describe the camera motions of each local region. Combination of all the local histograms is considered as the camera motion descriptor of a video shot. The shot descriptors are used in a classifier to classify video shots. In this work, we use support vector machine (SVM) for performing classification tasks. The experimental results show that the proposed camera motion descriptor has strong discriminative capability to classify different camera motion patterns in professionally captured video shots effectively. We also show that our proposed approach outperforms two state-of-the-art video shot classification methods.

This paper addresses the issue of evaluating the Quality of Experience (QoE) for Internet of Things (IoT) applications, with particular attention to the case where multimedia content is involved. A layered IoT architecture is firstly analyzed to understand which QoE influence factors have to be considered in relevant application scenarios. We then introduce the concept of Multimedia IoT (MIoT) and define a layered QoE model aimed at evaluating and combining the contributions of each influence factor to estimate the overall QoE in MIoT applications. Finally, we present a use case related to the remote monitoring of vehicles during driving practices, which is used to validate the proposed layered model, and we discuss a second use case for smart surveillance, to emphasize the generality of the proposed framework. The effectiveness in evaluating classes of influence factors separately is demonstrated.

Abstract. This paper presents a comprehensive method for the determination of glacier surface motion vector fields at high spatial and temporal resolution. These vector fields can be derived from monocular terrestrial camera image sequences and are a valuable data source for glaciological analysis of the motion behaviour of glaciers. The measurement concepts for the acquisition of image sequences are presented, and an automated monoscopic image sequence processing chain is developed. Motion vector fields can be derived with high precision by applying automatic subpixel-accuracy image matching techniques on grey value patterns in the image sequences. Well-established matching techniques have been adapted to the special characteristics of the glacier data in order to achieve high reliability in automatic image sequence processing, including the handling of moving shadows as well as motion effects induced by small instabilities in the camera set-up. Suitable geo-referencing techniques were developed to transform image measurements into a reference coordinate system.The result of monoscopic image sequence analysis is a dense raster of glacier surface point trajectories for each image sequence. Each translation vector component in these trajectories can be determined with an accuracy of a few centimetres for points at a distance of several kilometres from the camera. Extensive practical validation experiments have shown that motion vector and trajectory fields derived from monocular image sequences can be used for the determination of high-resolution velocity fields of glaciers, including the analysis of tidal effects on glacier movement, the investigation of a glacier's motion behaviour during calving events, the determination of the position and migration of the grounding line and the detection of subglacial channels during glacier lake outburst floods.

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

This paper investigates a novel motion vector outlier rejection method based on using mean shift clustering on block motion vectors. The accuracy of compressed domain global motion estimation techniques is largely influenced by its ability to counter the outlier motion vectors. These outliers occur in the block motion vector field due to moving objects, noise or due to large matching errors as a result of the encoders priority on rate distortion optimization. In the present work it is shown that by using mean shift clustering on block motion vectors, those clusters which correspond to outlier motion vectors can be identified. Once detected these clusters are kept out of the global motion estimation process thereby increasing the robustness of estimated camera parameters. The proposed method is compared with existing state-of-the-art outlier removal methods using synthetic and real video sequences to establish and validate its superiority.

This paper goes beyond classic Internet of Things (IoT) and focuses on the concept of Multimedia-based Internet of Things (M-IoT). Smart heterogeneous multimedia things can interact and cooperate with each another and with other things connected to the Internet to facilitate multimedia-based services and applications that are globally available to the users. Nowadays, the access network selection for heterogeneous multimedia devices is not managed taking into account the reputation that a given network has at a given instant. In order to solve these issues ReMIoT, a Reputation-based M-IoT architecture is introduced in this paper and presented alongside a novel solution to improve multimedia content delivery quality. The solution includes three distinct operational stages: (i) multimedia service detection, (ii) network reputation detection, and (iii) M-IoT network selection. The performance of the proposed solution is compared against that of a traffic type-based differentiated reputation algorithm for radio resource allocation (TYDER). This performance was evaluated in terms of QoS parameters such as throughput, packet delay and packet loss ratio.