Big Data Approach to Realised Volatility Forecasting Using HAR Model Augmented With Limit Order Book and News

We provide new empirical evidence on volatility forecasting in relation to asymmetries present in the dynamics of both return and volatility processes. Leverage and volatility feedback effects of the S&P 500 price and volatility dynamics are examined using recently developed methodologies to detect jumps and to disentangle their size from continuous return and continuous volatility. Granted that jumps in both return and volatility are important components for generating the two effects, we find jumps in return can improve forecasts of volatility, while jumps in volatility improve volatility forecasts to a lesser extent. Moreover, disentangling jump and continuous variations into signed semivariances further improve the out-of-sample performance of volatility forecasting models, with negative jump semivariance being highly more informative then positive jump semivariance. The model proposed is able to capture many empirical stylized facts while still remaining parsimonious in terms of number of parameters to be estimated.

This survey starts with a general overview of the strategies for stock price change predictions based on market data and in particular Limit Order Book (LOB) data. The main discussion is devoted to the systematic analysis, comparison, and critical evaluation of the state-of-the-art studies in the research area of stock price movement predictions based on LOB data. LOB and Order Flow data are two of the most valuable information sources available to traders on the stock markets. Academic researchers are actively exploring the application of different quantitative methods and algorithms for this type of data to predict stock price movements. With the advancements in machine learning and subsequently in deep learning, the complexity and computational intensity of these models was growing, as well as the claimed predictive power. Some researchers claim accuracy of stock price movement prediction well in excess of 80%. These models are now commonly employed by automated market-making programs to set bids and ask quotes. If these results were also applicable to arbitrage trading strategies, then those algorithms could make a fortune for their developers. Thus, the open question is whether these results could be used to generate buy and sell signals that could be exploited with active trading. Therefore, this survey paper is intended to answer this question by reviewing these results and scrutinising their reliability. The ultimate conclusion from this analysis is that although considerable progress was achieved in this direction, even the state-of-art models can not guarantee a consistent profit in active trading. Taking this into account several suggestions for future research in this area were formulated along the three dimensions: input data, model’s architecture, and experimental setup. In particular, from the input data perspective, it is critical that the dataset is properly processed, up-to-date, and its size is sufficient for the particular model training. From the model architecture perspective, even though deep learning models are demonstrating a stronger performance than classical models, they are also more prone to over-fitting. To avoid over-fitting it is suggested to optimize the feature space, as well as a number of layers and neurons, and apply dropout functionality. The over-fitting problem can be also addressed by optimising the experimental setup in several ways: Introducing the early stopping mechanism; Saving the best weights of the model achieved during the training; Testing the model on the out-of-sample data, which should be separated from the validation and training samples. Finally, it is suggested to always conduct the trading simulation under realistic market conditions considering transactions costs, bid–ask spreads, and market impact.

An experimental approach to the study of limit order books lies in the availability of data. Most of the results presented in this book - in any case, those we have produced ourselves - use the Thomson Reuters Tick History (TRTH) database. All exchange-traded assets worldwide are present in the TRTH database, where they are identified by their Reuters Identification Code (RIC). Similar to most historical databases directly provided by the exchanges, the TRTH data come into the form of two separate files, a trade file recording all transactions, and an event file recording every change in the limit order book. Some very specific information, such as traders’ identities, cannot be publicly disclosed for obvious confidentiality reasons, but in theory, one could reconstruct the sequence of order arrivals of all types using this trade and event files. After explaining the algorithm used for the processing of limit order book data, we describe the specific data sets that have been used at various places in this book. That way, our results can be reproduced, extended and, of course, challenged, based on the very same data sets we have used. Limit Order Book Data Processing Because one cannot distinguish market orders from cancellations just by observing changes in the limit order book (the “event” file), and since, the timestamps of the “trade” and “event” files are asynchronous , we use a matching procedure to reconstruct the order book events. In a nutshell, we proceed as follows for each stock and each trading day: 1. Parse the “event” file to compute order book state variations: • If the variation is positive (volume at one or more price levels has increased), then label the event as a limit order. • If the variation is negative (volume at one or more price levels has decreased), then label the event as a “likely market order”. • If no variation—this happens when there is just a renumbering in the field “Level” that does not affect the state of the book—do not count an event. 2. Parse the “trade” file and for each trade: • Compare the trade price and volume to likely market orders whose timestamps are in, where t Tr is the trade timestamp and Δ t is a predefined, market-dependent time window.

We introduce new machine learning methods for clustering traders who are actively trading in a modern electronic exchange which uses a matching engine to track aggregate and individual‐level limit order books. Each trader's individual limit order book is centered (with the current best bid and ask prices acting as a central reference), and the patterns in the individual limit order books are identified using a Wasserstein distance. The method is illustrated using simulated limit order book data and limit order book data from a stock exchange in Canada.

The Predictive Power of Limit Order Book for Future Volatility, Trade Price, and Speed of Trading

The limit order book of an exchange represents an information store of market participants’ future aims and for many traders the information held in this store is of interest. However, information loss occurs between orders being entered into the exchange and limit order book data being sent out. We present an online algorithm which carries out Bayesian inference to replace information lost at the level of the exchange server and apply our proof of concept algorithm to real historical data from some of the world’s most liquid futures contracts as traded on CME GLOBEX, EUREX and NYSE Liffe exchanges.

The Distribution of Uncertainty: Evidence from the VIX Options Market

Big data analytics and mining aims to discover implicit, previously unknown, and potentially useful information and knowledge from big data sets that contain huge volumes of valuable veracious data collected or generated at a high velocity from a wide variety of rich data sources. Among different big data analytic and mining tasks, this chapter focuses on frequent pattern mining. By relying on the MapReduce programming model, researchers only need to specify the “map” and “reduce” functions to discover (organizational) knowledge from (i) big data sets of precise data in a breadth-first manner or depth-first manner and/or from (ii) big data sets of uncertain data. Such a big data analytics process can be sped up by focusing the mining according to the user-specified constraints that express the user interests. The resulting (constrained or unconstrained) frequent patterns mined from big data sets provide users with new insights and a sound understanding of users' patterns. Such (organizational) knowledge is useful is many real-life information science and technology applications.

Big data analytics and mining aims to discover implicit, previously unknown, and potentially useful information and knowledge from big data sets that contain huge volumes of valuable veracious data collected or generated at a high velocity from a wide variety of rich data sources. Among different big data analytic and mining tasks, this chapter focuses on frequent pattern mining. By relying on the MapReduce programming model, researchers only need to specify the “map” and “reduce” functions to discover (organizational) knowledge from (i) big data sets of precise data in a breadth-first manner or depth-first manner and/or from (ii) big data sets of uncertain data. Such a big data analytics process can be sped up by focusing the mining according to the user-specified constraints that express the user interests. The resulting (constrained or unconstrained) frequent patterns mined from big data sets provide users with new insights and a sound understanding of users' patterns. Such (organizational) knowledge is useful is many real-life information science and technology applications.

Big data sets are composed of many records with various attributes. In order to extract information from big data sets, this paper researched a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$V$</tex> -type big data which satisfied the logical characteristics of “attribute disjunction”. Based on the inverse packet set theory, this paper put forward some new concepts of big data block, such as matrix, element of block. Using these concepts, this paper presented two structures of block reasoning, and then presented three theorems for block element, such as intelligent separation theorem, intelligent retrieval the-orem, and equivalence class theorem; Based on the separation and retrieval criteria of block element, this paper presented the separation and retrieval algorithm of block element. Finally, this paper researched the application of big data intelligent retrieval in credit default.

Market participants who need to trade a significant number of securities within a given period can face high transaction costs. In this paper, we document how improvements in intraday liquidity forecasts can help reduce total transaction costs. We compare various approaches for forecasting intraday transaction costs, including autoregressive and machine learning models, using comprehensive ultra‐high‐frequency limit order book data for a sample of NYSE stocks from 2002 to 2012. Our results indicate that improved liquidity forecasts can significantly decrease total transaction costs. Simple models capturing seasonality in market liquidity tend to outperform alternative models.

The paper studies the processing of large information data arrays (Big Data) in construction. The issues of the applicability of the big data concept (Big Data) at various stages of the life cycle of buildings and structures are considered. Methods for data conversion for their further processing are proposed. The methods used in the analysis of "big data" allow working with unstructured data sets (Data Mining). An approach is considered, in which the analysis of arbitrary data can be reduced to text analysis, similar to the analysis of ordinary text messages. At the moment, it is important and interesting to isolate non-obvious links present in the analysed data. The advantage of using big data is that it is not necessary to advance hypotheses for testing. Hypotheses appear during data analysis. Dependence analysis is a basic approach when working with big data. The concept of an automatic big data analysis system is proposed. For data mining, text analysis algorithms should be used, and discriminant functions should be used for the main problem to be solved (data classification).

Big Data Analytics methods take advantage of techniques from the fields of data mining, machine learning, or statistics with a focus on analysing large quantities of data (aka `big datasets') with modern technologies. Big data sets appear in remote sensing in the sense of large volumes, but also in the sense of an ever increasing amount of spectral bands (i.e., high-dimensional data). The remote sensing has traditionally used the above described techniques for a wide variety of application such as classification (e.g., land cover analysis using different spectral bands from satellite data), but more recently scalability challenges occur when using traditional (often serial) methods. This paper addresses observed scalability limits when using support vector machines (SVMs) for classification and discusses scalable and parallel developments used in concrete application areas of remote sensing. Different approaches that are based on massively parallel methods are discussed as well as recent developments in parallel methods.

Order Book Commonalities and Bond Return Predictability

This study aimed to understand the emerging effects big data is having across different business functions as well as how much the insights gathered, from big data sets, are starting to influence business strategy. The literature highlights a lack of understanding around big data, with very little research focusing on big data’s applications outside of short term marketing strategies. This research investigated the potential barriers to big data integration within businesses, and sought to understand how these could be removed. In order to achieve an in depth understanding, an interview based approach was used. A wide range of business sources were examined to further understand the gaps identified by previous research. Primary research took the form of in-depth interviews. Respondents were carefully selected based on industry knowledge and experience, working for companies such as Ebiquity PLC, Satalia and Barclays. The interviews were used to compare and contrast the experts’ opinions and establish whether the key themes identified in the literature held true in the primary research. This approach enabled real insight to be gathered in order to concisely examine and critique the current literature as well as enabled further themes to be investigated. This investigation found how critical big data is becoming to business success, including understanding an emerging gulf between those who have adopted data big sets and those who have not. It was also understood that, on the whole, knowledge and understanding is lagging far behind benchmarks set by some businesses. The paper concludes by explaining why big data is not influencing business strategy. A focused understanding approach is outlined as the preferred solution to bridge the gulf between the ‘haves’ and ‘have not’s’.