{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Text Analysis with Python\n",
"\n",
"Textual analysis is an essential tool in many fields today. With the rise of the internet and freely available data, it is easier than ever to download some text and analyse it. \n",
"\n",
"In this notebook, we will learn how to do textual analysis using some data from Twitter.\n",
"\n",
"## Table of contents:\n",
"- [1. Removing unnecessary data](#remove)\n",
"- [2. Filtering the text](#filter)\n",
"- [3. Wordclouds](#wordcloud)\n",
"- [4. Sentiment analysis](#sentiment)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us first load the dataset we are going to be using.\n",
"\n",
"The data for this notebook is extracted from the University of Edinburgh twitter account. It contains all tweets from 20/08/2014 to 07/02/2019, and there are some quite exciting finds in it!\n",
"\n",
"The data is initially loaded in a pandas DataFrame, and it is recommended that you keep it inside that for the whole notebook.\n",
"\n",
"The columns of the data are:\n",
"- `text` - the tweet itself\n",
"- `screen_name` - the name of the account that tweeted the tweet\n",
"- `created_at` - timestamp of the tweet. Contains date and time of publishing\n",
"- `retweet_count` - how many people have shared the tweet\n",
"- `favourite_count` - how many people have liked the tweet\n",
"\n",
"Run the cell below to load the data and see the first 5 columns"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# first load the dataset\n",
"df = pd.read_csv(\"data/uoe_tweets_07022019.csv\", index_col=0)\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now with that out of the way, let's begin doing some text analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Removing unnecessary data <a name=\"remove\"></a>\n",
"\n",
"From the dataset, it is quite evident that we don't need the `screen_name` column as all of the tweets are only from the @EdinburghUni profile. We can also consider simplifying the `created_at` column as we don't need the exact time of publication.\n",
"\n",
"Let's now extract only the date of the tweet. We have access to the `DateTime` object but don't worry about the details here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# extract only date\n",
"df[\"date\"] = df.created_at.str.slice(0,10)\n",
"df[\"date\"] = pd.to_datetime(df.date)\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 1\n",
"Remove the `screen_name` and `created_at` columns from the dataframe above"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. Filtering the text <a name=\"filter\"></a>\n",
"A critical stage of any text processing.\n",
"\n",
"First, let us convert all characters to lowercase. This avoids having multiple copies of the same words. For example in the previous word counting exercise, the words `Edinburgh` and `edinburgh` would be considered two different words, whereas they are the same!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# convert to lowercase\n",
"df.text = df.text.str.lower()\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's remove punctuation. It is possible to use the `.strip()` method but that doesn't catch all punctuation characters like `%` or `#`. Instead, we would like to use the `.replace` method combined with regular expressions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 2\n",
"Complete the code below in such a way that you catch as many punctuation characters as you can!\n",
"\n",
"For this, you will have to use regular expressions. If you are not familiar with them, you can have a look at this [webpage](https://stackoverflow.com/questions/39782418/remove-punctuations-in-pandas)\n",
"\n",
"*Note: the template code below stores the cleaned up text in a separate column. Once you are happy with the filtering just put the filtered text in the `text` column*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_test = df[\"text\"].str.replace(\"Put your regex here\", \"\")\n",
"df_test.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 3\n",
"\n",
"It would be interesting to see how frequently the university has made posts. Can you group all posts according to their date of publication and then create a plot through time of how many posts have been made each day.\n",
"\n",
"You can use the `DataFrame.groupby()` method to find occurrences. Don't forget to label the axis of your plot.\n",
"\n",
"You can transfrom a `DataFrame.groupby()` object back into a DataFrame by using its `.count()` method"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 4\n",
"Here we will identify the 10 most favourited posts. This can easily be done using the [`nlargest`](https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.nlargest.html) method and then simply indexing into the original dataframe.\n",
"\n",
"Find the 10 most popular posts, print them and see what they are about!\n",
"\n",
"You might want to use the `.iloc` method to index into the original data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 3. Wordclouds <a name=\"wordcloud\"></a>\n",
"Who doesn't like them right?\n",
"\n",
"A wordcloud is a picture which consists of bundled words from a piece of text. The higher the occurrence of the words, then the larger the text of the cloud is!\n",
"\n",
"The easiest way to create wordclouds in Python is with the `wordcloud` package. Before using it, however, we must first download it. Run the cell below to do just that"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Creating a wordcloud\n",
"# first need to install\n",
"!pip install wordcloud"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we are ready to go, let's import the package and creata a small wordcloud from arbitrary words"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import wordcloud as ws\n",
"\n",
"words = \"avocado banana apple orange tomato banana apple banana grape pear\"\n",
"\n",
"wordcloud = ws.WordCloud(width=1000, height=500, background_color=\"white\").generate(words)\n",
"fig, ax = plt.subplots(figsize=(10, 5))\n",
"ax.imshow(wordcloud, interpolation=\"bilinear\")\n",
"plt.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 5\n",
"\n",
"Now that you know how to create a wordcloud, can you create one with all of the tweets we have in our dataset?\n",
"\n",
"*Hint: you will have to use an `str` method to concatenate all of the tweets into one big block of text.* "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 4. Sentiment analysis <a name=\"sentiment\"></a>\n",
"\n",
"Here we will detect the sentiment of the tweets: that is, we will find the polarity and subjectivity of the tweet.\n",
"\n",
"For that, we will be using the `textblob` package (which we have to install first). Let's first see how we can use it on a small example of just the first 5 tweets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# run this cell to install the package\n",
"!pip install textblob"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One of the incredibly useful tools in text analysis is **lambda functions**. These are shorthand functions which you can use when you want to define a simple function but don't want to go through the whole process with the `def` keyword.\n",
"\n",
"They have the format\n",
"```python\n",
"lambda arguments: expression\n",
"```\n",
"but are best understood through an example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# define normal functino\n",
"def square(x):\n",
" return x ** 2\n",
"\n",
"# define lambda function\n",
"f = lambda x: x**2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Both have the same output\n",
"print(square(5))\n",
"print(f(5))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you know how `lambda` functions work, let's put them to use!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from textblob import TextBlob\n",
"\n",
"# select only 5 tweets\n",
"tweet = df.text[0:5]\n",
"tweet"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# do sentiment analysis\n",
"tweet.apply(lambda x: TextBlob(x).sentiment[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What does this mean?\n",
"\n",
"The values you would expect are within a range of \\[-1; 1\\]. A number closer to 1 means that the text has a positive statement and values closer to -1 means that the text has a negative statement.\n",
"\n",
"Bear in mind that this is using a machine learning model which is naturally prone to errors."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 6\n",
"\n",
"Apply sentiment analysis to the dataset of tweets. Find the most positive tweet and the most negative tweet. Does the sentiment analysis of `TextBlob` perform well?\n",
"\n",
"*Hint: you might want to store the sentiment inside `df` for convenience*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}