python-data-2-words.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 2 - Analysis of a text file\n",
    "\n",
    "The purpose of this section is to do basic text analysis with a practical purpose.\n",
    "\n",
    "In this notebook you will:\n",
    "* Find the number of occurrences of the word 'and'\n",
    "* Find the number of occurrences of any word\n",
    "* Construct a dictionary of all words that have appeared in the text\n",
    "* Attempt to clean up the text, leaving only words\n",
    "\n",
    "For this we will be using a text called [Humanistic Nursing by Josephine G. Paterson and Loretta T. Zderad](http://www.gutenberg.org/ebooks/25020). You should already have the file uploaded to your workspace. If that is not the case, you can download the `.txt` file from the link provided.\n",
    "\n",
    "Estimated duration: 1h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with files\n",
    "First, we will need to learn how to operate with files. Python offers many versetile ways and here we will touch on the basics needed for the exercise.\n",
    "\n",
    "We cam open a file simply by using the `open()` function and supplying it with the filename. By default the function opens files in read-only mode `\"r\"` and reads the text by lines.\n",
    "\n",
    "PS. No need to read the full text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"humanistic_nursing.txt\"\n",
    "file = open(filename)\n",
    "\n",
    "# print the file line by line\n",
    "for line in file:\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us skip reading the text for the time being.\n",
    "\n",
    "In the example about we opened the file in read-only mode. What if we want to write to a file? Then we must open it in write-only mode(`\"w\"`):\n",
    "```python\n",
    "file = open(filename, \"w\")\n",
    "```\n",
    "\n",
    "but be careful with this, as this will overwrite any file with that filename. In our case if we use\n",
    "```python\n",
    "filename = \"humanistic_nursing.txt\"\n",
    "file = open(filename, \"w\")\n",
    "```\n",
    "we would see that the file previous file will have been **deleted**. An alternative of that is to use the `\"x\"` write-only mode which again creates a new file for writing but fails if that file already exists.\n",
    "\n",
    "Here is a table of all available Python file modes:\n",
    "\n",
    "| Mode | Description |\n",
    "|----|:--|\n",
    "| r  | Read-only mode |\n",
    "| w  | Write-only mode;<br>creates a new file (erasing the data for any file with the same name) |\n",
    "| x  | Write-only mode;<br>creates a new file, but fails if the file path already exists |\n",
    "| a  | Apend to existing file; creates the file if it does not already exist |\n",
    "| r+  | Read and write |\n",
    "| b | Add to mode for binary files (i.e., `rb` or `wb`) |\n",
    "| t | Text mode for files (automatically decoding bytes to Univode). <br> This is the default if not specified. <br> Add `t` to other modes to use this (i.e., `rt` or `xt` ) |\n",
    "\n",
    "\n",
    "Once a file is opened we can read it in a couple of ways:\n",
    "- `read()` - returns a certain number of charecters from the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"humanistic_nursing.txt\"\n",
    "file = open(filename)\n",
    "\n",
    "print(file.read(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`read()` advances the file handle's position by the number of bytes read.\n",
    "- `tell()` - gives you the current position in a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file.tell())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if we read from the file again, we will advance the handle\n",
    "print(file.read(5))\n",
    "print(file.tell())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `seek()` - changes the position of the file's handle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file.seek(3)\n",
    "file.tell()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, after you are done with a file is important to close it. If you don't then the file would not be accessible elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a list of all methods available to files\n",
    "\n",
    "| Method | Description |\n",
    "|----|:--|\n",
    "| read(s[size])  | Return data from file as a string, with original size argument <br>indicating the number of bytes to read. |\n",
    "| readlines([size]) | Return list of lines in the file, with optional `size` argument. |\n",
    "| write(str) | write passed string to file. |\n",
    "| close() | Close the handle. |\n",
    "| flush() | Flush the internal I/O buffer to disk. |\n",
    "| seek(pos) | Move to indicated file position(integer) |\n",
    "| tell() | Return current file position as integer|\n",
    "| closed | True if the file is closed |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting \"and\"\n",
    "Let's see how we can count the occurances of `and` in the file. Run the cell below.\n",
    "\n",
    "_Note: Remember that you can deconstruct a line into individual words using the `split()` method - ie. `line.split()`._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def countAnd(filename):\n",
    "    counter = 0\n",
    "    file = open(filename)\n",
    "    \n",
    "    for line in file:\n",
    "        for word in line.split():\n",
    "            if word == \"and\":\n",
    "                counter += 1\n",
    "                \n",
    "    return counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify your function\n",
    "countAnd(\"humanistic_nursing.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully you will see 1922 for `\"and\"`. What happens if you search for `\"And\"` instead?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1 : Count any word\n",
    "Based on the function above, now write your own function which counts the occurances of any word. For example:\n",
    "```python\n",
    "countAny(filename, \"patient\")\n",
    "```\n",
    "will return the occurances of the word \"medicin\" in the file filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def countAny(filename, des_word):\n",
    "    # [ WRITE YOUR CODE HERE ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify your function\n",
    "countAny(filename, \"patient\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Patient should occur 125 times in the text. You can also try using the function for other words!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Count multiple words\n",
    "Before this exercise you should be familiar with Python dictionaries. If you're not, please see [here](https://docs.python.org/3/tutorial/datastructures.html#dictionaries).\n",
    "\n",
    "Intuitively, we can first fill in the dictionary keys with the words of the text. Afterwards we can count the occurances of each word and and fill in the appropriate dictionary value.\n",
    "\n",
    "*Hint: Can we use `countAny()` for this?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def countAll(filename, words):\n",
    "    # [ WRITE YOUR CODE HERE ]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify your function\n",
    "countAll(filename, [\"patient\", \"and\", \"the\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should expect `{'patient': 125, 'and': 1922, 'the': 2604}`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: Cleaning up \n",
    "Unless you have already accounted for it, your counter was thrown off by some words. Consider this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(countAny(filename, \"work\"))\n",
    "print(countAny(filename, \"work.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the word is the same, you have different values for the 2 keys.\n",
    "\n",
    "In order to fix this we have to clean up the words before they are counted.\n",
    "\n",
    "There are multiple ways to do this. You can get some ideas from the [String methods page](https://docs.python.org/3/library/stdtypes.html#string-methods)\n",
    "\n",
    "Some things to consider:\n",
    "- Do you account for upper and lowercase versions of the same word?\n",
    "- Can you clean up the text before splitting it into words?\n",
    "- Consider writing a seperate function for cleaning the text. This will allow you to test how effective your cleaning is.\n",
    "- Are you accounting for the copyright text at the beginning and ending of the file?\n",
    "- Are there any cases where cleaning up the text could give ambigious results?\n",
    "\n",
    "Now write a function that opens up the text, cleans all of the words and returns a big long list of words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleanText(filename):\n",
    "    # [ WRITE YOUR CODE HERE ]\n",
    "                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify your function\n",
    "cleanText(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: The longest word\n",
    "Create a function to find and return the longest word. You are intended to reuse functions you previously made!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# [ WRITE YOUR CODE HERE ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting with matplotlib\n",
    "\n",
    "Let's now see how you can plot in Python, using a library called `matplotlib`. It is fairly intuative and we can start by plotting a histogram of all word occurances in the text. For that we will use the functions you made previously.\n",
    "\n",
    "To make computation faster we can find the occurances of just the first 100 words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# clean up all the text and extract a dictionary of\n",
    "# word occurances\n",
    "all_words = cleanText(filename)\n",
    "word_dict = countAll(filename, all_words[:10])\n",
    "\n",
    "# get a list of the words we counted and their frequencies\n",
    "word_list = list(word_dict.keys())\n",
    "word_freq_list = list(word_dict.values())\n",
    "                 \n",
    "# create the plot\n",
    "plt.bar(word_list, word_freq_list)\n",
    "plt.xticks(rotation=45) # rotate the x axis labels\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}