Improved TFIDF in big news retrieval: An empirical study

Ubiquitous access to internet has resulted in more and more people going online to get their daily dose of news. In a 2010 survey conducted by the Pew Project for Excellence in Journalism, 41% of the respondents said they get most of their news online, 10% more than those who said they got most of their news from a newspaper. A lot of socio-technical factors have contributed to this phenomenal rise in adoption of online news in recent years. One of the biggest reasons why people are increasingly reading news online is because it facilitates discussion with peers (Nguyen 2010), offering different viewpoints which aid in forming a rounded personal opinion about the news story. The Pew survey found that 37% of online news users (and 51% of 18-29 year olds) think that commenting on news stories is an important feature to have. A lot of people tend to shape their opinion by reading discussion comments, reflective articles, blogs and even tweets about the news. Hence, an increasing number of people rely on online sources of news – be it news websites or news aggregator services like Digg, Reddit, Google Reader, Flipboard, Pulse etc. The problem with these news websites and aggregators is that the only way people can gather public opinion is by actively searching through the endless stream of comments and feeds, filtering out spam (which is a growing problem) and then reading the relevant posts. A top trending story on Twitter will typically see multiple tweets per second, and keeping up with the rapid flow of incoming tweets is quite cumbersome and cognitively taxing. Hence it becomes increasingly difficult and time consuming for someone who wants to get the pulse of the people affected by a news story. Furthermore, in certain scenarios people might want to look at more fine grained opinions. Currently, there is no elegant way to extract geographic and demographic impact of a news story. What is the public sentiment in Indonesia about the Arab Spring? How did the public opinion about the Wikileaks disclosures change as the story unfolded during the course of a year? It is very difficult and tedious to observe such patterns using the currently available news providers. This work attempts to solve these problems by proposing a news aggregator platform which pulls news stories from various sources and also aggregates public responses, reflections, opinions and sentiments associated with those stories. This data is presented in ways that are easily understandable so readers can make better sense of the stories unfolding across the globe. Such a news aggregator platform that gathers and display public opinion and sentiments about a story, must deal with various challenges – 1. Opinions are very subjective. Different people feel about a story in different ways. With such an enormous amount of diverse opinions and subjectivity, how can we possibly aggregate the responses into something that makes sense as a whole? ________________________________ Acknowledgements: Prof. Yardi, S. for the guidance Sethi, P. (2013). Public opinion aggregation by annotation and tagging of online news stories. iConference 2013 Proceedings (pp. 891-894). doi:10.9776/13453 Copyright is held by the author. iConference 2013 February 12-15, 2013 Fort Worth, TX, USA 892 2. There isn’t really a unified web standard for expressing opinion (in textual form). Some people tweet in 140 characters, while others write elaborate blog posts. Some websites employ tags which a reader can use to define and classify their public opinion, while others rely on threaded comments and comment ranking systems. How can a platform be flexible enough to adapt to all these varied standards so that it can extract valuable data from various sources? Perhaps the platform can create a new standard of expression on the web which is flexible and comprehensive enough to be used to express diverse views about every news story in the world. 3. How to filter out spam while extracting public opinion? 4. Once the platform has access to the data it needs, how should it be displayed to the reader in a way that makes sense? What forms of visualizations, illustrations and graphical representations can be employed to give the reader a holistic view of how people feel about a story? 5. How can the platform determine and convey effects of geographical, demographic and temporal variations as the story unfolds? These are just a few out of possibly many issues which must be dealt with. Previous research on similar public opinion aggregation services has greatly focused on natural language processing, data mining and text categorization and clustering. Xiaojun (2010) proposed a framework for crawling the web for comments and applying various data mining algorithms on the data to extract relevant information. Diakopoulos and Shamma (2010) used tweets posted in conjunction with the live presidential debate between Barack Obama and John McCain to gauge public opinion. Brody and Diakopoulos (2011) studied the use of word lengthening to detect sentiment in microblogs. This research proposes a solution – The Opinionated Reader, which relies on sentiment tags and annotations associated with a news story. The essential idea is to create a commenting, discussion and sharing plug-in which can be used by news websites and aggregators as a commenting solution for their news pages. Users wanting to share or comment on a news story through the plug-in are asked to tag the news story with sentiment tags and annotate the story with their reaction (happy/positive or sad/negative). These tags and annotations are stored, aggregated and linked to each news story. A mobile application provides the front-end interface for users to access the news stories and the aggregated sentiment associated with each story. The basic architecture is explained as follows: The Opinionated Reader – Mobile/Tablet App The app fetches news articles from various web sources, based on the interests and preferences configured by the user. In every news article, a portion of the screen real estate is reserved for Opinions which shows graphical visualizations and illustrations of the public opinion surrounding the news story. These visualizations include: A Sentiment Graph indicating the popular tags associated with the story (E.g.: “Shocking”, “Inspiring”, “Amusing” etc.). See Figure 1 for example visualization. A Positivity Graph which plots the level of positivity associated with the story on a time scale from when the news broke. See Figure 2 for example visualization. Figure 1. Example Sentiment Graph Figure 2. Example Positivity Graph iConference 2013 February 12-15, 2013 Fort Worth, TX, USA 893 The user can choose to see these visualizations for a particular time period in the evolution history of the news story, or for a specific country. The app also facilitates people to tag and annotate news articles from within its interface. The Opinionated Reader – Commenting and Sharing Web Plug-in These days, a common way of adding discussion and commenting functionality to news websites is by using 3 rd party services (like DISQUS). The Opinionated Reader is a similar service which can be embedded into the news articles of various news websites to enable commenting and sharing. When someone wishes to comment on an article, the comment is directed through this plug-in, which allows the users to annotate the article with the sentiment tags and reactions along with their comments. The Opinionated Reader saves this information along with the commenter’s location and date of comment (See Figure 3). Figure 3. The Opinionated Reader Web Plug-in 'Add Comment' dialog mockup The Opinionated Reader – Back-end The Back-end maintains a database of news items extracted from RSS feeds of various news websites. Each news article is linked with the sentiment tags and reaction/positivity annotations extracted from the comments and annotations gathered by the commenting plug-in. This data is used by the mobile/tablet app to generate visualizations (Sentiment Graph and Positivity Graph). The back-end also performs data mining on the tags and annotations for geographies and tracks the opinions across time. The back end system responds to queries received from the mobile app with the news story and associated tags and annotations, which are then rendered by the mobile app for the user. iConference 2013 February 12-15, 2013 Fort Worth, TX, USA 894 Discussion and Conclusion This design idea is still in a nascent state and has long hours of research, brainstorming, designing and development to go before it can be realized into something tangible. Twitter has grown exponentially in importance as a news source and it would be vastly valuable to integrate Twitter with The Opinionated Reader. Possibilities include use of special hash tags and natural language processing of tweets to extract public sentiment. The current design supports only two reaction annotations – positive and negative. Not every news story fits this annotation paradigm. Further research about human reactions to news stories might unveil interesting insights which would help zero in on a more robust annotation rubric. Lastly, since this service is envisioned to be non-curated and non-moderated, the value served by the app depends on the users themselves. Greater adoption will lead to more annotations and tags, which translates into a more accurate public opinion as presented to the user.

With the development of information technology, obtaining information quickly and accurately has become an indispensable part of people’s lives. Text classification can filter and organize massive text data to find valuable information, and its practical application is of great significance. This paper proposes Chinese text classification method using FastText and term frequency-inverse document frequency optimization(I-FastText). This method introduces term frequency-inverse document frequency(TF-IDF) optimization into the input layer of FastText model, removes words with high frequency and low discrimination ability, and achieves high-quality word vectors to train the text classification model. The experimental results on the THUCNews dataset shown that the Chinese text classification accuracy of I-FastText is significantly better than TF-IDF and FastText methods.

Reviews of women's tops in the market are valuable information if processed properly. Merchants can conduct product review analysis to obtain information that can be used to evaluate products and services. Product review analysis activities are not enough just to see the number of stars, it is necessary to see the entire content of the review comments to be able to know the intent of the review. Sentiment analysis system is a system used to automatically analyze online product reviews to obtain information including sentiment information that is part of online reviews. The data is classified using Naive Bayes. The data collected were 1,000 product reviews of women's tops as samples. The purpose of this study is to determine the sentiment analysis of female top product reviews using the Naive Bayes algorithm. The stages of this research include data collection, labeling, pre-processing, sentiment classification, and evaluation. In the pre-processing stage there are 6 stages, namely Cleaning emoticons & symbols, Case folding, Word Normalizer, Tokenize, Stopword Removal and Stemming. TF-IDF (Term Frequency - Inverse Document Frequency) method is used for word weighting. The data will be classified into 3 (three) classes, namely negative, positive and neutral. The data will then be evaluated using accuracy parameter testing. The test results show an accuracy value of 89%, this result shows that the product reviews of women's tops are more positive.

Twitter has become the foremost standard of social media in today’s world. Over 335 million users are online monthly, and near about 80% are accessing it through their mobiles. Further, Twitter is now supporting 35+ which enhance its usage too much. It facilitates people having different languages. Near about 21% of the total users are from US and 79% of total users are outside of US. A tweet is restricted to a hundred and forty characters; hence it contains such information which is more concise and much valuable. Due to its usage, it is estimated that five hundred million tweets are sent per day by different categories of people including teacher, students, celebrities, officers, musician, etc. So, there is a huge amount of data that is increasing on a daily basis that need to be categorized. The important key feature is to find the keywords in the huge data that is helpful for identifying a twitter for classification. For this purpose, Term Frequency-Inverse Document Frequency (TF-IDF) and Loglikelihood methods are chosen for keywords extracted from the music field and perform a comparative analysis on both results. In the end, relevance is performed from 5 users so that finally we can take a decision to make assumption on the basis of experiments that which method is best. This analysis is much valuable because it gives a more accurate estimation which method’s results are more reliable.

The purpose of this paper is to look into the recent refugee crisis in Europe and to argue on how this crisis turned into a humanitarian problem at a global scene. Moreover, this migration crisis has been developed into a security issue in Europe and in the same time affecting the dynamics of the European integration. For Europe, the integration of refugees is a big challenge. The debates have been focused on immigrant population as a whole rather than asylum-seekers in particular. The debate was often difficult to control because of different meaning between those who favour more assimilation policies against those who argue for protection for the newcomers' cultural identity. This research paper is trying to find out how the Europe is dealing with this huge challenge of immigrants and with their integration.

Ecumenical Chronicle

In 2015, various news media reported dramatic scenes of people crossing the Aegean Sea, including a tragic image of a drowned Syrian boy Alan Kurdi through which the refugee crisis in Europe seemed to become a global issue. The situation in Greece worsened after the closure of the Greek-Macedonian border or western Balkan route in March 2016. During the same period of time in 2017, some international media seem to challenge this somewhat optimistic attitude toward the refugee crisis in Idomeni by disseminating juxtaposed photographs reflecting two different times, before and after the closure of the camp. One of the contexts, the immediate visual context, is applicable to this chapter as it is about how an image is related to other images. The fragmentary nature of juxtaposed images helps "produce a sense of the 'presence' of history rather than its meaning" and "evoke the desire for affective encounter with past that cannot be reduced to a desire for its meaning".

With the rapid development of the internet technology, a large amount of internet text data can be obtained. The text classification (TC) technology plays a very important role in processing massive text data, but the accuracy of classification is directly affected by the performance of term weighting in TC. Due to the original design of information retrieval (IR), term frequency-inverse document frequency (TF-IDF) is not effective enough for TC, especially for processing text data with unbalanced distributions in internet media reports. Therefore, the variance between the DF value of a particular term and the average of all DFsDF¯, namely, the document frequency variance (ADF), is proposed to enhance the ability in processing text data with unbalanced distribution. Then, the normal TF-IDF is modified by the proposed ADF for processing unbalanced text collection in four different ways, namely, TF-IADF, TF-IADF+, TF-IADFnorm, and TF-IADF+norm. As a result, an effective model can be established for the TC task of internet media reports. A series of simulations have been carried out to evaluate the performance of the proposed methods. Compared with TF-IDF on state-of-the-art classification algorithms, the effectiveness and feasibility of the proposed methods are confirmed by simulation results.

News articles categorization is a supervised learning approach in which news articles are assigned category labels based on likelihood demonstrated by a training set of labeled articles. A system for automatic categorization of news articles into a standard set of categories has been implemented. The proposed work will use Term Frequency–Inverse Document Frequency (TF-IDF) term weighting scheme for optimization of classification techniques to get more optimized results and use two supervised learning approaches, i.e., Support Vector Machine (SVM) and K-Nearest neighbor (kNN) and compare the performances of both classifiers. Each news document is preprocessed and transformed into a term-document matrix (Tsoumakas et al. in Data mining and knowledge discovery handbook. Springer, Berlin, pp 667–685 (2010) [1]). After preprocessing and transforming each news article into a vector of weights, TF-IDF term weighting scheme was used for weighting the word. TF-IDF weighted the words calculating the number of words that appear in a document. An unknown news item is also transformed into a vector of keyword weights, and then categorized into suitable categories such as Sports, Business, and Science and Technology. The system purposed in research work was trained on the collection of approximately 300 categorized news articles extracted from the various Indian newspaper websites and tested on a different set of 60 randomly extracted news items from the same sources (Trstenjak et al. in Proc Eng 69:1356–1364 (2014) [2], Buana et al. Int J Comput Appl 50:37–42 [3]). It has been observed that the performances of both algorithms improve when TF-IDF approach is used.

The performance of text classification can be affected by the choice of appropriate term weighting scheme as well as other parameters. The terminology supervised term weighting scheme has become popular in recent years, as it may provide discriminative representation in vector space for text documents belonging to different classes. A term weighting scheme generally consists of three factors, namely term frequency factor, collection frequency factor, and length normalization factor. The researchers mostly have been focused on developing new collection frequency factors in term weighting studies. However, the term frequency factor has an important role, especially in supervised term weighting. In this study, we extensively analyzed the effects of using different term frequency factors on seven supervised term weighting schemes. While six of these supervised term weighting schemes were applied in the previous studies in the literature, we derived one of them from an existing feature selection method and it was not used as a weighting method before. This analysis is performed using SVM and Roccio classifiers on two widely known benchmark datasets with different characteristics. Experimental results showed that modification of term frequency factor in supervised term weighting schemes increased the performance of almost all weighting schemes. Also, term weighting schemes using square root function-based term frequency factor (SQRT_TF) are more successful than the ones using term frequency (TF) and logarithmic function-based term frequency (LOG_TF) factors. TF term frequency factor seems as the least effective one among three different term frequency factors according to the experimental results and statistical analysis.

BackgroundThe Syria crisis has forced more than 4 million people to leave their homeland. As a result, in 2016, an overwhelming number of refugees reached Germany. In response to this, it was of utmost importance to set up refugee camps and to provide humanitarian aid, but a health surveillance system was also implemented in order to obtain rapid information about emerging diseases.ObjectiveThe present study describes the effects of using digital paper and pen (DPP) technology on the speed, sequence, and behavior of epidemiological documentation in a refugee camp.MethodsDPP technology was used to examine documentation speed, sequence, and behavior. The data log of the digital pens used to fill in the documentation was analyzed, and each pen stroke in a field was recorded using a timestamp. Documentation time was the difference between first and last stroke on the paper, which includes clinical examination and translation.ResultsFor three months, 495 data sets were recorded. After corrections had been made, 421 data sets were considered valid and subjected to further analysis. The median documentation time was 41:41 min (interquartile range 29:54 min; mean 45:02 min; SD 22:28 min). The documentation of vital signs ended up having the strongest effect on the overall time of documentation. Furthermore, filling in the free-text field clinical findings or therapy or measures required the most time (mean 16:49 min; SD 20:32 min). Analysis of the documentation sequence revealed that the final step of coding the diagnosis was a time-consuming step that took place once the form had been completed.ConclusionsWe concluded that medical documentation using DPP technology leads to both an increase in documentation speed and data quality through the compliance of the data recorders who regard the tool to be convenient in everyday routine. Further analysis of more data sets will allow optimization of the documentation form used. Thus, DPP technology is an effective tool for the medical documentation process in refugee camps.

Retorno:Salvadoran Repatriation and the Landscape of Memory Interview with Mark Menjivar Tatiana Reinoza (bio) INTRODUCTION The Blue Star Contemporary Art Museum in San Antonio recently commissioned a multi-media installation by the Borderland Collective, a loosely-knit group of Texas-based artists and educators who produce socially engaged projects.1 Their installation, Northern Triangle (2014), takes its name from the geographic region, one of the most violent in the world, as well as the trade agreement that binds the countries of El Salvador, Guatemala, and Honduras.2 Led by artists Mark Menjivar (b. 1980), Jason Reed, and art historian Erina Duganne, Northern Triangle responds to the recent wave of unaccompanied Central American minors apprehended at the U.S.-Mexico border. On the occasion of the project's initial debut, I sat down with Menjivar to discuss Retorno, a series of photographs made in collaboration with a community of former refugees in El Salvador that figures prominently in the installation, and the complicated relationship that continues to shape politics between Central America and the United States. In 2014, authorities apprehended about 68,000 minors fleeing the violence of the Northern Triangle. Once in custody, immigration officers placed the children in the newly constructed private prisons that dot the South Texas landscape. The detention centers became their temporary dwelling prior to facing deportation proceedings, a remarkable shift in policy given the region's former reputation as an entry-point to the Sanctuary Movement, a religious and political movement that provided safe-haven to Central Americans fleeing war-torn countries in the 1980s. As an example of the current migrant crisis that characterizes our time, and parallels the Syrian refugee crisis in Europe, these Central American minors transformed into what legal historian Mae Ngai calls "impossible subjects." Their presence poses a problem, not only for enforcing the boundaries of the nation-state, but also for visual modes of representation. Taking the refugee crisis as a point of departure, Borderland Collective embarks on a conceptual project that problematizes the violence of the Northern Triangle as a schema long in the making. Inspired by the New York-based art collective, Group Material, and their curatorial mediations, particularly, Timeline: A Chronicle of U.S. Intervention in Central and Latin America (1984), Borderland Collective places art objects in conversation with historical documents from the Library of Congress, the National Archives, U.S. Customs and Border Protection, news footage, human rights cases, and personal documents. Reed and Menjivar contribute photographs, while Duganne establishes a research-based approach to the display and accompanying wall text. Several invited artists, including Adriana Corral, Vincent Valdez, and Ricky Yanas, extend this dialogue to create open-ended works that question the constructed nature of mainstream images of the crisis. The result is a provocative and disquieting installation that doubles as a history museum and classroom. One critic commented that the installation "functions as a kind of puzzle."3 Viewers move through the space slowly (Fig. 1), connecting images with historical narratives, approaching three-dimensional objects such as an empty blue water barrel that signals the stakes as bodies, not much different from our own, traverse geopolitical borders. The work on display Click for larger view View full resolution Fig 1. Installation shot of Northern Triangle (2014–) at Blue Star Contemporary Art Museum, San Antonio. Courtesy of Borderland Collective. [End Page 181] Click for larger view View full resolution Fig 2. Installation shot of Northern Triangle (2014–) with Mark Menjivar's View from Santa Marta on the left. Courtesy of Borderland Collective. deconstructs "the language of colonization and power," as one Artforum critic put it, in order to unsettle our assumptions about a region whose violent history is often shaped by outside forces.4 It is within this diverse range of objects, discourses, and texts that viewers get acquainted with Menjivar's luscious photographs. (Fig. 2) Born in Richmond, Virginia, to a Puerto Rican mother and a Salvadoran-American father, Menjivar is a military brat, whose unusual upbringing took him overseas to Honduras, Panama, and El Salvador during the 1980s, an intense period of U.S. intervention in Central America. While many viewed the surge in unaccompanied Central American children at the border...

Attitudes towards migrants and refugees are created and reflected at the level of public policies, as well as in local communities which cultivate traditional approaches and a specific worldview. The refugee crisis in Europe in the mid-2010s showed how public opinion translated into voting behaviour and became a source of strength for nationalist anti-immigrant movements and parties across the continent. East-Central Europe was no exception, regardless of the absence of a long-term, massive inflow of refugees. Nevertheless, the migration crisis created a new political narrative which exploited deeply rooted resentments, complexes, and fears. This article aims to analyse the official policy responses to the refugee crisis in the four East-Central European countries: Poland, Hungary, Slovakia, and the Czech Republic, which together constitute the so-called Visegrad Four. It puts the emphasis on the discriminatory practice of misnaming the refugees, which became deeply anchored in the political discourse of these countries. Based on a qualitative content analysis supplemented by the findings of public opinion polls, the argument developed in the article is that reluctant and defensive attitudes towards the refugees have been determined by the revival of parochialism as a radical reaction to the challenges of global trends and supra-local processes. The theoretical framing of the refugee problem is built on politicization, in connection with the concept of parochialism, seen from political and social perspectives, and the meaning of the use of the misnomer as a policy instrument. The article concludes that the migration crisis petrified traditional cleavages at the supra-local level, reinforcing simultaneously the sense of parochial altruism and hostility towards “the other.”

The Refugee Crisis in Europe

The refugee crisis in Europe: will it increase disparity in women's health?