python-data-2-words.ipynb

 
 Notebook 2 - Analysis of a text file
The purpose of this section is to do basic text analysis with a practical purpose.
In this notebook you will:

Find the number of occurrences of the word 'and'
Find the number of occurrences of any word
Construct a dictionary of all words that have appeared in the text
Attempt to clean up the text, leaving only words
For this we will be using a text called Humanistic Nursing by Josephine G. Paterson and Loretta T. Zderad. You should already have the file uploaded to your workspace. If that is not the case, you can download the .txt file from the link provided.
Estimated duration: 1h
 Dealing with files
First, we will need to learn how to operate with files. Python offers many versatile ways and here we will touch on the basics needed for the exercise.
We can open a file simply by using the open() function and supplying it with the path of the desired file. 
Python will start looking in the same directory (folder) as the current notebook by default. The current directory has the file path ., and a file in a subdirectory of the current one would have the file path ./subdirectory/file_name.type. If the current directory lives in another directory, its parent directory is accessed by the filepath ...
By default the function opens files in read-only mode "r" and reads the text by lines.
PS. No need to read the full text.
 filename = "./data/humanistic_nursing.txt"
file = open(filename)

# print the file line by line
for line in file:
    print(line)
 
 Let us skip reading the text for the time being.
In the example about we opened the file in read-only mode. What if we want to write to a file? Then we must open it in write-only mode("w"):
file = open(filename, "w")

but be careful with this, as this will overwrite any file with that filename. In our case if we use
filename = "./data/humanistic_nursing.txt"
file = open(filename, "w")

we would see that the file previous file will have been deleted. An alternative of that is to use the "x" write-only mode which again creates a new file for writing but fails if that file already exists.
Here is a table of all available Python file modes:


Mode
Description


r
Read-only mode


w
Write-only mode;
creates a new file (erasing the data for any file with the same name)


x
Write-only mode;
creates a new file, but fails if the file path already exists


a
Apend to existing file; creates the file if it does not already exist


r+
Read and write


b
Add to mode for binary files (i.e., rb or wb)


t
Text mode for files (automatically decoding bytes to Unicode). 
 This is the default if not specified. 
 Add t to other modes to use this (i.e., rt or xt )


Once a file is opened we can read it in a couple of ways:

read() - returns a certain number of characters from the file.
 filename = "./data/humanistic_nursing.txt"
file = open(filename)

print(file.read(10))
 
 read() advances the file handle's position by the number of bytes read.

tell() - gives you the current position in a file
 print(file.tell())
 
 # if we read from the file again, we will advance the handle
print(file.read(5))
print(file.tell())
 
 
seek() - changes the position of the file's handle
 file.seek(3)
file.tell()
 
 Finally, after you are done with a file is important to close it. If you don't then the file would not be accessible elsewhere.
 file.read()
 
 Here is a list of all methods available to files


Method
Description


read([size])
Return data from file as a string, with optional size argument 
indicating the number of bytes to read.


readlines([size])
Return list of lines in the file, with optional size argument.


write(str)
write passed string to file.


close()
Close the handle.


flush()
Flush the internal I/O buffer to disk.


seek(pos)
Move to indicated file position(integer)


tell()
Return current file position as integer


closed
True if the file is closed


 Counting "and"
Let's see how we can count the occurances of and in the file. Run the cell below.
Note: Remember that you can deconstruct a line into individual words using the split() method - ie. line.split().
 def countAnd(file_path):
    counter = 0
    file = open(file_path)
    
    for line in file:
        for word in line.split():
            if word == "and":
                counter += 1
                
    return counter
 
 # Verify your function
countAnd("./data/humanistic_nursing.txt")
 
 Hopefully you will see 1922 for "and". What happens if you search for "And" instead?
 Exercise 1 : Count any word
Based on the function above, now write your own function which counts the occurances of any word. For example:
countAny(filename, "medicine")

will return the occurences of the word "medicine" in the file filename.
 def countAny(file_path, des_word):
    # [ WRITE YOUR CODE HERE ]
    
 
 # Verify your function
countAny(file_path, "patient")
 
 Patient should occur 125 times in the text. You can also try using the function for other words!
 Exercise 2: Count multiple words
Before this exercise you should be familiar with Python dictionaries. If you're not, please see here.
Write a function which takes a file path and a list of words, and returns a dictionary mapping each word to its frequency in the given file.
Intuitively, we can first fill in the dictionary keys with the words in our list. Afterwards we can count the occurrences of each word and and fill in the appropriate dictionary value.
Hint: Can we use countAny() for this?
 def countAll(file_path, words):
    # [ WRITE YOUR CODE HERE ]
    
 
 # Verify your function
countAll(filename, ["patient", "and", "the"])
 
 You should expect {'patient': 125, 'and': 1922, 'the': 2604}
 Exercise 3: Cleaning up
Unless you have already accounted for it, your counter was thrown off by some words. Consider this:
 print(countAny(filename, "work"))
print(countAny(filename, "work."))
 
 Although the word is the same, you have different values for the 2 keys.
In order to fix this we have to clean up the words before they are counted.
There are multiple ways to do this. You can get some ideas from the String methods page
Some things to consider:

Do you account for upper and lowercase versions of the same word?
Can you clean up the text before splitting it into words?
Consider writing a seperate function for cleaning the text. This will allow you to test how effective your cleaning is.
Are you accounting for the copyright text at the beginning and ending of the file?
Are there any cases where cleaning up the text could give ambigious results?
Now write a function that opens up the text, cleans all of the words and returns a big long list of words:
 def cleanText(file_path):
    # [ WRITE YOUR CODE HERE ]
             
 
 # Verify your function
cleanText(filename)
 
 Exercise 4: The longest word
Create a function to find and return the longest word. You are intended to reuse functions you previously made!
 def longestWord(file_path):
    # [ WRITE YOUR CODE HERE ]
    
 
 Plotting with matplotlib
Let's now see how you can plot in Python, using a library called matplotlib. It is fairly intuitive and we can start by plotting a histogram of all word occurances in the text. For that we will use the functions you made previously.
To make computation faster we can find the occurances of just the first 100 words.
 import matplotlib.pyplot as plt

# clean up all the text and extract a dictionary of
# word occurances
all_words = cleanText(filename)
word_dict = countAll(filename, all_words[:10])

# get a list of the words we counted and their frequencies
word_list = list(word_dict.keys())
word_freq_list = list(word_dict.values())
                 
# create the plot
plt.bar(word_list, word_freq_list)
plt.xticks(rotation=45) # rotate the x axis labels
plt.show()