Video Summarization Using Fully Convolutional Residual Dense Network

In urethra-sparing radiation therapy, prostatic urinary tract visualization is important in decreasing the urinary side effect. A methodology has been developed to visualize the prostatic urinary tract using post-urination magnetic resonance imaging (PU-MRI) without a urethral catheter. This study investigated whether the combination of PU-MRI and super-resolution (SR) deep learning models improves the visibility of the prostatic urinary tract. We enrolled 30 patients who had previously undergone real-time-image-gated spot scanning proton therapy by insertion of fiducial markers. PU-MRI was performed using a non-contrast high-resolution two-dimensional T2-weighted turbo spin-echo imaging sequence. Four different SR deep learning models were used: the enhanced deep SR network (EDSR), widely activated SR network (WDSR), SR generative adversarial network (SRGAN), and residual dense network (RDN). The complex wavelet structural similarity index measure (CW-SSIM) was used to quantitatively assess the performance of the proposed SR images compared to PU-MRI. Two radiation oncologists used a 1-to-5 scale to subjectively evaluate the visibility of the prostatic urinary tract. Cohen's weighted kappa (k) was used as a measure of agreement of inter-operator reliability. The mean CW-SSIM in EDSR, WDSR, SRGAN, and RDN was 99.86%, 99.89%, 99.30%, and 99.67%, respectively. The mean prostatic urinary tract visibility scores of the radiation oncologists were 3.70 and 3.53 for PU-MRI (k = 0.93), 3.67 and 2.70 for EDSR (k = 0.89), 3.70 and 2.73 for WDSR (k = 0.88), 3.67 and 2.73 for SRGAN (k = 0.88), and 4.37 and 3.73 for RDN (k = 0.93), respectively. The results suggest that SR images using RDN are similar to the original images, and the SR deep learning models subjectively improve the visibility of the prostatic urinary tract.

This paper addresses the video summarization problem. For the given video goal is to find the subset of frames that capture the important events of the input video and produce a small concise summary. We formulate video summarization as a sequence labeling problem, where for a given input video a subset of frames are selected as a summary video. Based on the principle of semantic segmentation, here each pixel within a frame is assigned to one of the labels, where each frame is assigned a binary label indicating whether it will be included in the summary video or not. We propose a SegNet sequence network (SegNetSN) for video summarization and further extend the work by applying various feature fusion techniques to enhance the input. We performed experiments on the benchmark dataset TVSum.

With the development of convolutional neural network, video super-resolution algorithm has achieved remarkable success. Because the dependence between frames is complex, traditional methods lack the ability to model the complex dependence, and it is difficult to estimate and compensate the motion accurately in the process of video super-resolution reconstruction. Therefore, a reconstruction network based on optical flow residuals is proposed. In low resolution space, the dense residual network is used to obtain the complementary information of adjacent video frames, and then the optical flow of high-resolution video frames is predicted through the pyramid structure, and then the low resolution video frames are transformed into high-resolution video frames through the sub-pixel convolution layer, The high-resolution video frame is compensated with the predicted high-resolution optical flow. Finally, it is input into the super-resolution fusion network to get better effect. A new loss function training network is proposed to better constrain the network. Experimental results on public data sets show that the reconstruction effect is improved in PSNR, structural similarity and subjective visual effect.

As COVID-19 spread worldwide, many major grain-producing countries have adopted measures to restrict their grain exports; food security has aroused great concern from various parties. How to improve grain production has become one of the most important issues facing all countries. However, crop diseases are a difficult problem for many farmers so it is important to master the severity of crop diseases timely and accurately to help staff take further intervention measures to minimize plants being further infected. In this paper, a restructured residual dense network was proposed for tomato leaf disease identification; this hybrid deep learning model combines the advantages of deep residual networks and dense networks, which can reduce the number of training process parameters to improve calculation accuracy as well as enhance the flow of information and gradients. The original RDN model was first used in image super resolution, so we need to restructure the network architecture for classification tasks through adjusted input image features and hyper parameters. Experimental results show that this model can achieve a top-1 average identification accuracy of 95% on the Tomato test dataset in AI Challenger 2018 datasets, which verifies its satisfactory performance. The restructured residual dense network model can obtain significant improvements over most of the state-of-the-art models in crop leaf identification, as well as requiring less computation to achieve high performance.

In this paper, we propose a residual dense networks (RDN)-based image super-resolution method using Meta-learning. Specifically, deep extraction of global features is performed on the external dataset through an RDN, meta-learning to obtain an initial parameter definitely for internal learning, so we can utilize both external and internal data. Our method achieves good results with only one gradient update. And it can be appropriate for image super-resolution under the action of different blur kernels, with a wider application range and high flexibility.

Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.

This paper addresses the problem of unsupervised video summarization, formulated as selecting a sparse subset of video frames that optimally represent the input video. Our key idea is to learn a deep summarizer network to minimize distance between training videos and a distribution of their summarizations, in an unsupervised way. Such a summarizer can then be applied on a new video for estimating its optimal summarization. For learning, we specify a novel generative adversarial framework, consisting of the summarizer and discriminator. The summarizer is the autoencoder long short-term memory network (LSTM) aimed at, first, selecting video frames, and then decoding the obtained summarization for reconstructing the input video. The discriminator is another LSTM aimed at distinguishing between the original video and its reconstruction from the summarizer. The summarizer LSTM is cast as an adversary of the discriminator, i.e., trained so as to maximally confuse the discriminator. This learning is also regularized for sparsity. Evaluation on four benchmark datasets, consisting of videos showing diverse events in first-and third-person views, demonstrates our competitive performance in comparison to fully supervised state-of-the-art approaches.

An efficient video summarization for surveillance system using normalized k-means and quick sort method

With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from original lengthy videos. Though supervised video summarization approaches have demonstrated the state-of-the-art performance, unsupervised methods are still highly demanded due to resourcefully expensive human annotations and the subjectiveness of video summarization tasks. In this paper, a novel unsupervised-based Deep Self-attention Recurrent summarization network with Reinforcement Learning (DSR-RL) for video summarization is proposed. The model can learn the input video sequence and suggest the key-shot summary without additional human annotations by integrating self-attention, BRNN, and reinforcement learning mechanisms. The DSR-RL improves not only importance score through the attention map vector of self-attention network but also the diversity of summaries via the reward function of reinforcement learning. Our method outperforms the state-of-the-art unsupervised video summarization methods on both SumMe and TVSum datasets. The source code is available at https://github.com/phaphuang/DSR-RL.

Natural scene text images captured by handheld devices usually cause low resolution (LR) problems, thus making sub-sequent detection and recognition tasks more challenging. To address this problem, LR text images are generally super-resolution (SR) processed first. In this paper, we propose a novel low-resolution text image super-resolution method. This method adopts the residual-in-residual dense network (RRDN) to extract deeper high-frequency features than the residual dense network (RDN). Then, enhances the spatial and channel features with an attention mechanism. According to the characteristics of the text, we added gradient loss to adversarial learning. Experiments show that our method performs well in both qualitative and quantitative aspects of the latest public text image super-resolution dataset. Similarly, the proposed super-resolution method for text images of natural scenes also achieves the latest results.

Advancements in technology resulted in a sharp growth in the number of digital cameras at people's disposal all across the world. Consequently, the huge storage space consumed by the videos from these devices on video repositories make the job of video processing and analysis to be time-consuming. Furthermore, this also slows down the video browsing and retrieval. Video summarization plays a very crucial role in solving these issues. Despite the number of video summarization approaches proposed up to the present time, the goal is to take a long video and generate a video summary in form of a short video skim without losing the meaning or the message transmitted by the original lengthy video. This is done by selecting the important frames called key-frames. The approach proposed by this work performs automatic summarization of digital videos based on detected objects' deep features. To this end, we apply sparse subspace clustering with an automatically estimated number of clusters to the objects' deep features. The summary generated from our scheme will store the meta-data for each short video inferred from the clustering results. In this paper, we also suggest a new video dataset for video summarization. We evaluate the performance of our work using the TVSum dataset and our video summarization dataset.

Video summarisation is the process of creating a meaningful and self-explanatory summary of a given video by automatically selecting keyframes and thus creating a short and concise video summary of the original video clip. Video captioning refers to the task of automatically generating natural language description of a given video to provide some additional or interpretive information. Video summarisation and video captioning are often considered as two different tasks, each with a diverse application field. Thus, we propose an approach to jointly combine these two tasks and present a model which generates a short video summary along with a relevant caption. We make use of self-attention based transformer network combined with Multi Layer Perceptron (MLP) with multiple hidden layers for video captioning and video summarisation. We aim to demonstrate that the joint model can attain better performance than many of the previous approaches in both of the individual tasks. Our proposed model is based on the fact that both of the above mentioned tasks of captioning and summarisation will improve the performance of the other one as both the tasks are complementary to each other

In general, video-based learning contains rich media information, but displaying an entire video linearly is time-consuming. As an alternative, video summarization techniques extract important content provides short but informative fragments. In this paper, a video learning platform (KVSUM: Keyword-based Video Summarization) is presented that integrates image processing, text summarization, and keyword extraction techniques. Without human annotators, the learning platform can process an input video and transform it into online learning materials automatically. The video frames are first split from a given video while the transcription is used to generate the text summary and keywords. In the current study, the video surrogates are composed of extracted keywords, text and video summaries, and video frames. In other words, KVSUM is able to provide both visual and verbal surrogates. In order to validate the effect of surrogates in the KVSUM, a comparison with another video surrogate, the fast forward (FF), to evaluate learners' comprehension to video contents. Sixty undergraduate students took part in examining two different video surrogates. The experimental results show that KVSUM had a more positive effect than FF in comprehension to videos. In terms of system usage and satisfaction, KVSUM is significantly more attractive than FF to learners.

Short videos on the Internet are growing exponentially, and the number of videos uploaded every day is huge; people also involve a lot of video data in real life. People can retrieve and view all kinds of videos, but it also brings a lot of problems. On the one hand, the accumulation of a large number of videos makes people unable to find the videos they want quickly, and the repeated scenes in the videos will also waste people's time and energy; on the other hand, a large amount of video data also brings enormous pressure to storage. Aiming at the problems of inaccurate selection of key frames and how to select video frame features in existing video summarization models, this paper proposes a multi-feature-based video summarization generation model (DME-VSNet), which extracts multiple features of video frames. Including importance score, image memory strength and image entropy. Aiming at the problem of inaccurate video shot segmentation, this model proposes a video shot segmentation algorithm based on TransNet network, which divides the original video into several short shots through shot boundaries; the model inputs the above three features into the proposed The video frame score is obtained in the MLP architecture, and the key frame is selected by the score to generate a video summary. The effectiveness of the video shot segmentation method based on TransNet network and the overall model based on convolutional neural network is verified by comparative experiments. The experimental results show that the evaluation results of the video summaries generated by the three features are better.

Chinese calligraphy, as a writing art of Chinese characters, plays an important role in the inheritance of traditional culture. Compared with simple English fonts, the generation of Chinese calligraphy fonts is more challenging. This paper proposes Chinese calligraphy generation based on residual dense network. The generator network combines residual network structure and dense network structure, and uses local residual learning and global feature fusion to enhance feature reuse and information continuous transmission. The discriminator network adopts an autoencoder structure, and restores the similarity between the distributions by an approximate convergence strategy to achieve fast and stable training. At the same time, the structural similarity loss is introduced to quantify the similarity between the generated font image and the real font image. Experiments show that the proposed method achieves good calligraphy font generation effect, which effectively improves the quality of the generated font images and the rate of training.