Effective Text Mining in Twitter using Clustering Techniques

The following sample essay on Effective Text Mining in Twitter using Clustering Techniques tells about exploration text mining.

Text Mining is considered to be an important part in today’s life style. From the point of decision makers, this collection of text mining gives us important source of information. In this paper we are discussing about text mining on twitter using cluster analysis. Twitter is a microblogging platform for million users who share their information, views, opinion on others and their attitudes.

usage of Social media is increasing day by day and there are lots of social media sites in internet apart from other media sites twitter is the most popular blog among them. The primary goal of this paper is to explore text mining, extract and analyze useful information from unstructured text using one approach such as Cluster analysis. This method allows people to dig data more effectively and efficiently. we can fetch the real time twitter tweets on a particular topic and stored it into R and then we can apply several text mining steps on the tweets to pre-process the tweets text and then we can analyze the preprocess data by visualizing them.

After Preprocessing clustering algorithms are applied on text data. The different clusters formed are compared through different parameters. The observed results show that the hierarchical clustering algorithm performs better than another algorithm.

Social Networks have become one of the crucial sources of social information and have attracted interest and curiosity in social research and commerce.

Get quality help now

RhizMan

Verified

Proficient in: Data

4.9 (247)

“ Rhizman is absolutely amazing at what he does . I highly recommend him if you need an assignment done ”

+84 relevant experts are online

Hire writer

In addition, techniques of acquiring and reading huge amounts of information generated through social media have in themselves end up the interest of researchers. As a social on-line microblogging server, Twitter lets in users to broadcast and engage with posts known as tweets. Twitter has unexpectedly received global-wide popularity Since its launch. Text data is a good example of unstructured data, that is one of the simplest forms of information that can be generated in maximum scenarios. The records can be kept non-public most of the listing or made public and unrestricted. Unstructured textual content is effortlessly processed and perceived by means of human beings however is substantially tougher for machines to recognize. This quantity of textual content is an invaluable source of facts and expertise. As a result, there’s a desperate need to layout techniques and algorithms to efficiently process this avalanche of text in an extensive sort of programs. Text mining techniques are associated to conventional statistics mining, and expertise discovery strategies Text mining techniques are related to standard information mining, and information discovery techniques, with some specifications.

This paper deals with the analysis of large collections of written resources to generate new information, and to transform unstructured text into structured data for use in further analysis. Text mining is getting a lot attention these last years, due to an exponential increase in digital text data from web pages and social media services such as Twitter. Particularly, Twitter which is widely used and fast growing in real world blog. More than 336 million monthly active twitter users create enormous text data through their tweets every day. Twitter data constitutes a rich source that can be used for capturing information about any topic imaginable. This data can be used in different use cases such as finding trends related to a specific keyword, measuring brand sentiment, and gathering feedback about new products and services. People upload their thoughts, opinions and real-world happenings on this twitter site. Tweet content has been used not only as a rapid and inexpensive way to glimpse public opinion in general, but also within many other leading industries. Twitter provides hash tags, which is used to categorize the topics for tweets. If a hash tag is used by many people, then twitter sorts the topic as current trending topic. Despite the growing attention to analyzing user-generated content from social media, most health researchers have little knowledge about how to apply content-mining methods.

Clustering analysis is one of the data mining techniques which is used to extract underlying patterns in data. One main application of cluster analysis is in text-mining, the analysis of large collections of text to ?nd similarities between documents. We used a collection of data extracted from the twitter public API. A very common issue with today’s real time data is the presence of linguistic noise which cause constrains in topic flow and time series. Preprocessing algorithms can be applied on text data for better results.

Today twitter has a vast heap of data and unfortunately, most of it is unstructured in nature. There is a large quantity of data in the form of free flow text residing in Twitter’s data servers. While there are many descriptive and predictive techniques in place that help process and analyze structured (i.e. numeric) data, fewer techniques exist that are targeted towards analyzing natural language data. To overcome these problems, we applied clustering techniques to the text mining that enables to programmatically organize the data. This would help narrow down the volume of unstructured data on a broader spectrum. This would help to understand the context with top keywords instead of trying to understand millions of rows. Our paper aims in analyzing the text data using clustering techniques.

Text mining is the way toward exploring and analyzing a lot of unstructured text data helped by software that can identify concepts, designs, topics, keywords and different attributes in the data. Text mining, also known as text data mining or knowledge discovery process from the textual databases.

Text Mining is like data mining, for storing information in organized manner and despite what might be expected text mining utilizes texts that are unstructured or semi-organized. However, one of the initial phases in the text mining process is to arrange and structure the information in some design so it tends to be exposed to both predictive and descriptive analysis. The data is collected through World Wide Web, twitter, government portals, face book, blog, news articles, digital libraries, electronic mail and so on. Roughly 80% of the hierarchical information is put away in unstructured form. The forthright work incorporates categorizing, clustering and labeling content; abridging data indexes; creating scientific classifications; and extracting data about things like word frequencies and connections between data substances. Analytical models are then rushed to create discoveries that can help drive business techniques and operational activities.

Clustering is the gathering of a specific arrangement of objects dependent on their attributes, collecting them as indicated by their connections. Regarding data mining, this strategy segments the data actualizing a particular join algorithm, most appropriate for the ideal data analysis. Cluster analysis is identified with different methods that are utilized to isolate information objects into groups. For example, clustering can be viewed as a form of arrangement in that it makes a marking of objects with cluster labels. Nevertheless, it gets these labels only from the data.

There are three explicit clustering techniques that represent wide categories of algorithms and outline an assortment of concepts: K-means, agglomerative hierarchical clustering, and DBSCAN. K-means: This is a prototype-based, separated clustering technique that endeavors to discover a user- determined number of clusters (K), which are characterized by their centroids.

Agglomerative Hierarchical Clustering: This clustering method alludes to a gathering of firmly related clustering techniques that produce a hierarchical clustering by beginning with each idea as a singleton cluster and over combining the two nearest clusters until a single, all-inclusive cluster leftover. A portion of these techniques have a normal clarification as far as graph-based clustering, while others have a clarification as far as a prototype-based approach. This is a thickness-based clustering algorithm that produces a partitioned clustering, in which the quantity of clusters is naturally resolute by the algorithm.

Cluster analysis is a field of data analysis that extracts fundamental examples in data. One utilization of cluster analysis is in content mining, the analysis of extensive accumulations of content to discover similitudes between reports. Support vector machine, na?ve bayes classifier, density-based clustering and k-means clustering algorithms are the most and recently suggested text mining algorithms. Twitter, social media blog which is being used to connect people with same interests. This procedure of associating individuals who are finished outsiders should be possible with the utilization of hashtags. Hashtags, which are indicated with the “#” prefix, are added to Tweets so individuals from the network can partake in the discussion. Twitter has a huge number of clients on the planet. In this paper the examination of Twitter data is performed through the text contained in hash tags.

Sentiment analysis can be seen as a use of text order, which goes back to the work on probabilistic text arrangement. The fundamental assignment of text arrangement is the means by which to name texts with a predefined set of classes. Text arrangement has been connected in different regions, for example, record ordering, record sifting, word sense disambiguation, and so on as overviewed in Sebastiani8. One of the focal issues in text arrangement is the means by which to speak to the substance of a text in request to encourage a powerful grouping. From examines in data recovery frameworks, one of the most prominent and effective technique is to speak to a text by the accumulation of terms show up in it. The closeness between reports is characterized by utilizing the term recurrence opposite archive recurrence. Subsequent to Preprocessing clustering algorithms are connected on text information.

We then form different cluster which can be compared using different parameters. Once different clusters are formed, we can then apply different data mining algorithms to the clusters formed. We mainly use hashtags to form the cluster. In case of any events users use hashtags to react about the event. Text mining has its foundations in practically every one of the zones. Visualization helps in better comprehension of the separated substance from the raw data. It further gives a reasonable image of the data that must be conveyed. Text mining incorporated with Visualization gives a superior and quick understanding of interpreted outcomes. Text visualization has two forms, Topic, And Feature based. In Topic based strategy, topics and occasions are imagined through representation methods. A portion of the systems incorporate Tag clouds which portray the keywords or named elements? They use highlights like shading, size and format based on convenience and significance. Data scene gives a topographical perspective on expansive arrangement of records for examination. Text Flow strategy consolidates topic mining and intuitive representation strategies to outwardly break down the advancement of topics at the appropriate time. In Feature based technique, Word clouds are regularly produced to give an instinctive visual outline of reports by showing the keywords in a conservative design. Aspect Atlas strategy coordinates hub connect graph with thickness guide to outwardly break down the multifaceted relations of the report. Generally, we gather Tweets for text mining. We then apply preprocessing to the data to form the clustering and which leads to evaluation of clustering Algorithms and then Visualization. The below figures give as an overview of process that takes place when applying the clustering techniques for twitter data.

Applying k-means algorithm which is quite commonly used algorithm to form the clusters of the data. This algorithm is used to divide the data points of the data to clusters. Using this the distance between each data points and its cluster is reduced in other words it is minimized. At first k-means picks k random points from the information space, not really points in the information, and appoints them as centroids. Then taking the centroids we then reassign them to minimize the distance among them and also the data points in each cluster formed. Then we will reassign the formed data points to the closest centroid. We will continue this process till we reach the convergence. The words that are comparable in tweets are grouped together in single cluster and divergent words are grouped in various clusters.

Considering the sample data tweets during world cup, let’s consider tweets with cloud words world cup. We can start by applying k-means algorithm, the distance between the centroids and divided n data points can be measured with different metrics like, cosine and Euclidean distances. With the help of the cosine, the distance is defined as the distance between two data points and with the help of Euclidean, the distance is defined as the magnitude of difference between the data points that are formed in the cluster. With genuine information, it is once in a while di?cult to realize what number of clusters are required before performing the algorithm.

Now let’s see how we can apply clustering technique to tweets of world cup. To start first we need to remove all same tweets in twitter same tweets are termed ad retweets. This removal can be considered as of the preprocessing technique to the twitter data. Now let’s remove noise, Tweets are composed and posted absent much update. In other words that tweets will contain noise. A portion of that noise in the vocabulary of tweets can be expelled with a stop list and by stemming. When we take a gt’ander at an accumulation of tweets, we need the tweets that are the most firmly identified with one speci?c point. Tweets on the edges of clusters are as yet identified with the theme only not as intently. Hence, we can evacuate them as noise without harming the significance of the bunch. Now we have to find the most common topics and select the number of the topics to form the clusters. For example, we can create the Laplacian matrix (L) such that L=D?C where D is a diagonal matrix with entries corresponding to the sum of the rows of the consensus matrix, C. Then we have to cluster all the tweets with words world cup. At that point we made a word cloud for each clustering request to picture the general subjects all through the tweets. But this method has an issue, as it is all about find the cluster number of the different tweets. We have other techniques that can be used to find specifically about each cluster.

In this paper, the principles, related to research field theories and the application of effective text mining in Twitter using clustering technique were discussed. We learnt what is Text mining, process of text mining, approaches, issues, area, advantages and disadvantages. Twitter is one of the most popular social media sites on the web and creates enormous data every day. Text data mining provides better knowledge on information discovery and will assist further in decision making. Sentiment analysis provides a good method of showing the emotions and sentiments found in each tweet and of summarizing the results. Since we only looked deeply into text data, further research could prove that other algorithms are better for different types of data.