Online clustering and classification for real-time event detection in Twitter

With the advent of social networks and the rise of mobile technologies, users have become ubiquitous sensors capable of monitoring various real-world events in a crowd-sourced manner. Location-based social networks have proven to be faster than traditional media channels in reporting and geo-locating breaking news, i.e. Osama Bin Laden’s death was first confirmed on Twitter even before the announcement from the communication department at the White House. However, the deluge of user-generated data on these networks requires intelligent systems capable of identifying and characterizing such events in a comprehensive manner. The data mining community coined the term, event detection , to refer to the task of uncovering emerging patterns in data streams . Nonetheless, most data mining techniques do not reproduce the underlying data generation process, hampering to self-adapt in fast-changing scenarios. Because of this, we propose a probabilistic machine learning approach to event detection which explicitly models the data generation process and enables reasoning about the discovered events. With the aim to set forth the differences between both approaches, we present two techniques for the problem of event detection in Twitter : a data mining technique called Tweet-SCAN and a machine learning technique called Warble. We assess and compare both techniques in a dataset of tweets geo-located in the city of Barcelona during its annual festivities. Last but not least, we present the algorithmic changes and data processing frameworks to scale up the proposed techniques to big data workloads.

Event detection in twitter is the process of discovering popular events using messages generated by the twitter users. Event detection is an interesting research topic. Tweets are focused and may contain short forms. The tweets are noisy because there may be personal messages by the user also. In this paper we propose an algorithm to find top k popular events using keywords contained in the tweets. This paper classifies the popular events into different categories and the timeline is provided for every event. The timeline is useful to check when the event was popular. Geotagging is also done to find where the event was popular. We have implemented the algorithm using 14,558 users and 5, 27,548 tweets over a period of 10 months (22 June, 2015 to 25 April, 2016). The results are quite promising.

Event detection in Twitter is an attractive and hard task. Existing methods mainly consider words co-occurrence or topic distribution of tweets to detect the event. Few of them consider the time-series information in the text stream. In this paper, for event detection in twitter, we propose a novel multi-view clustering model which can consider both topic information and time-series information. First, we build a topic similarity matrix and a time-series similarity matrix by using the topic model and the wavelet analysis, respectively. Then, the multi-view clustering algorithms are used to group keywords. Each cluster of keywords is finally represented as an event. The experiments show that our method achieves better performance than other related works.

With the increased popularity of online social networks, spammers find these platforms easily accessible to trap users in malicious activities by posting spam messages. In this work, we have taken Twitter platform and performed spam tweets detection. To stop spammers, Google SafeBrowsing and Twitter's BotMaker tools detect and block spam tweets. These tools can block malicious links, however they cannot protect the user in real-time as early as possible. Thus, industries and researchers have applied different approaches to make spam free social network platform. Some of them are only based on user-based features while others are based on tweet based features only. However, there is no comprehensive solution that can consolidate tweet's text information along with the user based features. To solve this issue, we propose a framework which takes the user and tweet based features along with the tweet text feature to classify the tweets. The benefit of using tweet text feature is that we can identify the spam tweets even if the spammer creates a new account which was not possible only with the user and tweet based features. We have evaluated our solution with four different machine learning algorithms namely - Support Vector Machine, Neural Network, Random Forest and Gradient Boosting. With Neural Network, we are able to achieve an accuracy of 91.65% and surpassed the existing solution [1] by approximately 18%.

Twitter First Story Detection (FSD) task refers to the detection of first tweet about the new event in tweet stream, which is a hard but important task in twitter event detection. However, the number of comparisons is too large in traditional online detection methods and there is a lack of global information of each event during detection process. To deal with the shortcomings above, we propose a novel FSD method based on nugget, which can describe the event concisely. Our approach generates and updates dynamically a nugget for each detected event in the process of detection. When a new tweet arrives, it is first compared with the nugget of each event, to be clustered into the event when it hits the nugget. Otherwise it is compared with individual tweets in the event. Our method improves the detection accuracy and reduces the number of comparisons. The experimental results on two public data sets show that our system has reached the state-of-the-art. Besides, we prove theoretically that our method possesses advantages in efficiency.

Event detection is an interesting area of study that has taken a great deal of attention throughout the last years due to the extensive social media data availability. Problems with event detection have been explored in various social media sources such as Twitter, Flickr, YouTube, and Facebook. The event Detection process includes many challenges, including processing huge volumes of information and high noise levels. The event depends on many basic determinants of which are what, where, and when. Tweets that express a real-world event include semantic terms that match these determinants. The paper presents an incremental approach to clustering based on these determinants and semantic terms that express an event to detect social media events, especially from Twitter. The event detection framework includes three main components: pre-processing, on-line clustering, and event search and detection modules. By implementing some experiments on the proposed framework using a set of large and real-world data, it becomes clear to us that the results are positive. The framework detects events effectively from Twitter compared to other approaches.

Acoustic sensors are widely used for monitoring urbanized water distribution networks (WDNs) to detect and localize pipe leakages. Since their inception, few research studies have focused on developing a generic, effective, and practical methodology to analyse complex acoustics signals for leakage detection and localization in large-scale WDNs. In collaboration with PUB, Singapore’s National Water Agency, a generic acoustic data analysis approach has been developed to facilitate PUB’s present Smart Water Grid (SWG) management. The proposed approach encompasses multi-stage systematic analyses, namely: (1) data quality assessment; (2) data pre-processing; (3) near real-time leakage event detection and classification; and finally (4) near real-time leakage localization. Our proposed approach is then tested in major WDNs in Singapore having more than 1100km of underground water pipelines and 82 permanently installed hydrophone acoustic sensors between 1 Aug 2019 and 31 Aug 2020, where multiple historical leakage events were reported to within 600m, or less, from neighbouring hydrophones across the large complex networks. By emulating the near real-time detection and localization analyses daily, our proposed methodology could localize reported leakage events to an error range of around 150m on average, while demonstrating significant and stable acoustic leakage power rate over the temporal size of the leakage event cluster(s).

A real-time digital seismic event detection and recording system has been developed for the MIT Seismic Network. The system has been designed specifically for an environment of low natural seismic activity and for surface stations which are often influenced by weather conditions and cultural noise. The system runs on an HP-1000 computer and can handle up to 16 channels of short- and long-period data. The structure of the system centers around the event detectors, one for short-period data and one for long-period data. These detectors base their decisions on a metric computed from the Walsh transform of the data. This allows them to detect changes in the amplitude of the waveform as well as frequency shifts. Detections at several stations are correlated to prevent glitches from triggering the detector. Present operation successfully saves those events that are large enough for analysis and leaves 23 of the computer available for general timesharing use.

This research introduces an adaptive control algorithm designed to determine gait phase in real-time using an inertial measurement unit (IMU) affixed to the shank. Focusing on detecting specific gait events, primarily initial contact (IC) and toe-off (TO), the algorithm utilizes dynamic thresholds and ratios that facilitate accurate event determination adaptively across a range of walking speeds. Built-in safety checks further ensure precision and minimize false detections. We validated the algorithm with eight participants walking at varying speeds. The algorithm demonstrated promising results in detecting IC and TO events with mean lead of 8.95 ms and 4.42 ms and detection success rate of 100% and 99.72%, respectively. These results are consistent with benchmarks from established algorithms (Hanlon and Anderson, 2009, "Real-Time Gait Event Detection Using Wearable Sensors," Gait Posture, 30(4), pp. 523-527; Maqbool et al., 2017, "A Real-Time Gait Event Detection for Lower Limb Prosthesis Control and Evaluation," IEEE Trans. Neural Syst. Rehabil. Eng.: Publ. IEEE Eng. Med. Biol. Soc., 25(9), pp. 1500-1509). Moreover, the algorithm's self-adaptive nature ensures it can be used in scenarios of varying movement, offering a promising solution for real-time gait phase detection.

Recent advancements in computer vision led to the development of a real-time surveillance system which ensures the safety and security of the people in public places. An aerial surveillance system will be advantageous in this scenario using a platform like Unmanned Aerial Vehicle (UAV) will be very reliable and can be considered as a cost-effective option for this task. To make the system fully autonomous, we require real-time abnormal event detection. But, this is computationally complex and time-consuming due to the heavy load on the UAV, which affords limited processing and payload capacity. In this paper, we propose a cost-effective approach for aerial surveillance in which we move the large computation tasks to the cloud while keeping limited computation on-board UAV device using edge computing technique. Further, our proposed system will maintain the minimum communication between UAV and cloud. Thus it not only reduces the network traffic but also reduces the end-to-end delay. The proposed method is based on the state-of-the-art YOLO (You Only Look Once) technique for real-time object detection deployed on edge computing device using Intel neural compute stick Movidius VPU (Vision Processing Unit), and we applied abnormal event detection using motion influence map on the cloud. Experimental results demonstrate that the proposed system reduces the end-to-end delay. Further, Tiny YOLO is six times faster while processing the frames per second (fps) when compared to other state-of-the-art methods.

In recent years, real-time direct detection of events by surveillance systems has attracted a great deal of attention. In this paper, we propose a new video-based surveillance system that can perform real-time event detection. In the background modeling phase, we adopt a mixture of Gaussian approach to determine the background. Meanwhile, we use color blob-based tracking to track foreground objects. Due to the self-occlusion problem, the tracking module is designed as a multi-blob tracking process to obtain similar multiple trajectories. We devise an algorithm to merge these trajectories into a representative one. After applying the Douglas-Peucker algorithm to approximate a trajectory, we can compare two arbitrary trajectories. The above mechanism enables us to conduct real-time event detection if a number of wanted trajectories are pre-stored in a video surveillance system.

Although real-time event detection during video EEG recording is required to ensure patients’ safety, it is limited by the technologists’ availability. We sought to explore the efficiency of real-time event detection by the EEG technologists in a single tertiary academic center. We retrospectively reviewed events from continuous inpatient video EEGs (cEEGs) and epilepsy monitoring unit (EMU) recordings in January 2017, when real-time surveillance was only available during the night shift, and June 2017, when a dedicated neurodiagnostic EEG technologist was available for real-time monitoring during all shifts. The events were categorized into those detected immediately (eyes-on), later in the same shift (delayed) or identified on the subsequent shift (missed). Chi-square and Fisher’s exact tests were used for statistical comparisons. In January 2017, there were 25 patients (117 days of monitoring) in the EMU and 54 inpatients (146 days of monitoring) on cEEG with 92 total events, (39% seizures). In June 2017, there were 30 patients (133 days of monitoring) in the EMU and 47 additional inpatients (80 days of monitoring) on cEEG with 110 total events, (39% seizures). The number of events identified in real time was low and did not significantly differ among shifts regardless of the availability of the monitoring technologist. Most events were identified at the time of subsequent EEG scanning by the EEG technologist. Partial staffing for continuous video EEG surveillance is insufficient to identify events in real time. EEG technologists are able to identify events during regular EEG scanning.

The emergence of social media opens tremendous research opportunities. Many individuals, mostly teens and young adults around the world, share their daily lives and opinions about a wide variety of topics (e.g., crime, sports, and politics) on social media sites. Thus, social media becomes a valuable repository for data of different types, which could provide insights of social events happening around the world. However, it is still a challenge to identify the bursty and disruptive events from the massive and noisy user-generated content on social media sites. In this paper, we present a novel event detection framework for identifying surrounding real-world events that can support decision making and emergency management. Our proposed framework consists of four main components, including data pre-processing, event-related tweets classifying, online clustering, and bursty event recognition. We conducted a series of experiments on the real-world social media dataset collected from Twitter. The experimental results demonstrated the effectiveness of our proposed method.

In this paper, we present PAN, a generic middleware for distributed real-time complex event detection CED which is able to analyze multiple distributed data streams. In PAN, CED applications are defined as workflows and are executed by dedicated workers in a distributed way in a P2P network. In consequence, PAN is scalable in terms of the number of data streams and the complexity of the analyses. Evaluations based on an extended version of the ACM DEBS 2013 Grand Challenge scenario show the effectiveness and efficiency of PAN.

A software framework is described which was designed to process EEG, video of one eye, and head movement in real time, towards achieving early detection of microsleeps for prevention of fatal accidents, particularly in transport sectors. The framework is based around a pipeline structure with user-replaceable signal processing modules. This structure can encapsulate a wide variety of feature extraction and classification techniques and can be applied to detecting a variety of aspects of cognitive state. Users of the framework can implement signal processing plugins in C++ or Python. The framework also provides a graphical user interface and the ability to save and load data to and from arbitrary file formats. Two small studies are reported which demonstrate the capabilities of the framework in typical applications: monitoring eye closure and detecting simulated microsleeps. While specifically designed for microsleep detection/prediction, the software framework can be just as appropriately applied to (i) other measures of cognitive state and (ii) development of biomedical instruments for multi-modal real-time physiological monitoring and event detection in intensive care, anaesthesiology, cardiology, neurosurgery, etc. The software framework has been made freely available for researchers to use and modify under an open source licence.