{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Notebook 5 - pandas\n",
"[pandas](http://pandas.pydata.org) provides high-level data structures and functions designed to make working with structured or tabular data fast, easy and expressive. The primary objects in pandas that we will be using are the `DataFrame`, a tabular, column-oriented data structure with both row and column labels, and the `Series`, a one-dimensional labeled array object.\n",
"\n",
"pandas blends the high-performance, array-computing ideas of NumPy with the flexible data manipulation capabilities of spreadsheets and relational databases. It provides sophisticated indexing functionality to make it easy to reshape, slice and perform aggregations.\n",
"\n",
"While pandas adopts many coding idioms from NumPy, the most significant difference is that pandas is designed for working with tabular or heterogeneous data. NumPy, by contrast, is best suited for working with homogeneous numerical array data.\n",
"<br>\n",
"\n",
"## Table of Contents:\n",
"- [Data Structures](#structures)\n",
" - [Series](#series)\n",
" - [DataFrame](#dataframe)\n",
"- [Essential Functionality](#ess_func)\n",
" - [Reindexing](#reindexing)\n",
" - [Dropping Entries](#removing)\n",
" - [Indexing, Slicing and Filtering](#indexing)\n",
" - [Arithmetic Operations](#arithmetic)\n",
"- [Summarizing and Computing Descriptive Statistics](#sums)\n",
"- [Loading and storing data](#loading)\n",
" - [Text Format](#text) \n",
" - [Web Scraping](#web)\n",
"- [Data Cleaning and preperation](#cleaning)\n",
" - [Handling missing data](#missing)\n",
" - [Data transformation](#transformation)\n",
"\n",
"The common pandas import statment is shown below:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Common pandas import statement\n",
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Structures <a name=\"structures\"></a>\n",
"## Series <a name=\"series\"></a>\n",
"A Series is a one-dimensional array-like object containing a sequence of values and an associated array of data labels called its index.\n",
"\n",
"The easiest way to make a Series is from an array of data:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"data = pd.Series([4, 7, -5, 3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now try printing out data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The string representation of a Series displayed interactively shows the index on the left and the values on the right. Because we didn't specify an index, the default on is simply integers 0 through N-1.\n",
"\n",
"You can output only the values of a Series using \n",
"```python\n",
"data.values\n",
"```\n",
"or you can get only the indices using\n",
"```python\n",
"data.index\n",
"```\n",
"Try it out below!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can specify custom indeces when intialising the Series"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"data2 = pd.Series([4, 7, -5, 3], index=[\"a\", \"b\", \"c\", \"d\"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can use these labels to access the data similar to a normal array"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data2[\"a\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another way to think about Series is as a fixed-length ordered dictionary. Furthermore, you can actually define a Series in a similar manner to a dictionary"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"cities = {\"Glasgow\" : 599650, \"Edinburgh\" : 464990, \"Abardeen\" : 196670, \"Dundee\" : 147710}\n",
"data3 = pd.Series(cities)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Glasgow 599650\n",
"Edinburgh 464990\n",
"Abardeen 196670\n",
"Dundee 147710\n",
"dtype: int64"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can do arithmetic operations between Series similar to NumPy arrays. Even if you have 2 datasets with different data, arithmetic operations will be aligned according to their indices.\n",
"\n",
"Let's look at an example"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"cities_uk = {\"Birmingham\" : 1092330, \"Leeds\": 751485, \"Glasgow\" : 599650,\n",
" \"Manchester\" : 503127, \"Edinburgh\" : 464990}\n",
"data4 = pd.Series(cities_uk)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Abardeen NaN\n",
"Birmingham NaN\n",
"Dundee NaN\n",
"Edinburgh 929980.0\n",
"Glasgow 1199300.0\n",
"Leeds NaN\n",
"Manchester NaN\n",
"dtype: float64"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data3 + data4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how some of the results are NaN? Well, that is because there were no instances of those cities within both of the datasets. You can usually extract NaNs from a Series with\n",
"```python\n",
"data4.isnull()\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DataFrame <a name=\"dataframe\"></a>\n",
"A DataFrame represents a rectangular table of data and contains an ordered collection of columns, each of which can be a different value type. The DataFrame has both row and column index and can be thought of as a dict of Series all sharing the same index.\n",
"\n",
"The most common way to create a DataFrame is with dicts"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"data = {\"cities\" : [\"Glasgow\", \"Edinburgh\", \"Abardeen\", \"Dundee\"],\n",
" \"population\" : [599650, 464990, 196670, 147710],\n",
" \"year\" : [2011, 2013, 2013, 2013]}\n",
"frame = pd.DataFrame(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Try printing it out"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>cities</th>\n",
" <th>population</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Glasgow</td>\n",
" <td>599650</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Edinburgh</td>\n",
" <td>464990</td>\n",
" <td>2013</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Abardeen</td>\n",
" <td>196670</td>\n",
" <td>2013</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Dundee</td>\n",
" <td>147710</td>\n",
" <td>2013</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" cities population year\n",
"0 Glasgow 599650 2011\n",
"1 Edinburgh 464990 2013\n",
"2 Abardeen 196670 2013\n",
"3 Dundee 147710 2013"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"frame"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Jupyter Notebooks prints it out in a nice table but the basic version of this is also just as readable!\n",
"\n",
"Additionally you can also specify the order of columns during initialisation"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"frame2 = pd.DataFrame(data, columns=[\"year\", \"cities\", \"population\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can retrieve a particular column from a DataFrame with\n",
"```python\n",
"frame[\"cities\"]\n",
"```\n",
"The result is going to be a Series\n",
"\n",
"Additionally, you can retrieve a row from the dataset using\n",
"```python\n",
"frame[1]\n",
"```\n",
"Try it out below"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to add and modify the columns of a DataFrame"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"frame2[\"size\"] = 100"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>year</th>\n",
" <th>cities</th>\n",
" <th>population</th>\n",
" <th>size</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2011</td>\n",
" <td>Glasgow</td>\n",
" <td>599650</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2013</td>\n",
" <td>Edinburgh</td>\n",
" <td>464990</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2013</td>\n",
" <td>Abardeen</td>\n",
" <td>196670</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2013</td>\n",
" <td>Dundee</td>\n",
" <td>147710</td>\n",
" <td>100</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" year cities population size\n",
"0 2011 Glasgow 599650 100\n",
"1 2013 Edinburgh 464990 100\n",
"2 2013 Abardeen 196670 100\n",
"3 2013 Dundee 147710 100"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"frame2"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"frame2[\"size\"] = [175, 264, 65.1, 60] # in km^2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similar to dicts, columns can be deleted using\n",
"```python\n",
"del frame2[\"size\"]\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another common way of creating DataFrames is from a nested dict of dicts:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>cities</th>\n",
" <th>population</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Glasgow</td>\n",
" <td>599650</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Edinburgh</td>\n",
" <td>464990</td>\n",
" <td>2013</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Abardeen</td>\n",
" <td>196670</td>\n",
" <td>2013</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Dundee</td>\n",
" <td>147710</td>\n",
" <td>2013</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" cities population year\n",
"0 Glasgow 599650 2011\n",
"1 Edinburgh 464990 2013\n",
"2 Abardeen 196670 2013\n",
"3 Dundee 147710 2013"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data2 = {\"Glasgow\": {2011: 599650},\n",
" \"Edinburgh\": {2013:464990},\n",
" \"Abardeen\": {2013: 196670}}\n",
"\n",
"frame3 = pd.DataFrame(data)\n",
"frame3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a table of different ways of initialising a DataFrame for your reference\n",
"\n",
"| Type | Notes |\n",
"| --- | --- |\n",
"| 2D ndarray | A matrix of data; passing optional row and column labels |\n",
"| dict of arrays, lists, or tuples | Each sequence becomes a column in the DataFrame; all sequences must be the same length |\n",
"| dict of Series | Each value becomes a column; indexes from each Series are unioned together to<br>form the result's row index if not explicit index is passed |\n",
"| dict of dicts | Each inner dict becomes a column; keys are unioned to form the row<br>index as in the \"dict of Series\" case |\n",
"| List of dicts or Series | Each item becomes a row in the DataFrame; union of dict keys or<br>Series indices becomes the DataFrame's column labels |\n",
"| List of lists or tuples | Treated as the \"2D ndarray\" case |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Essential Functionality <a name=\"ess_func\"></a>\n",
"In this section, we will go through the fundamental mechanics of interacting with the data contained in a Series or DaraFrame."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reindexing <a name=\"reindexing\"></a>\n",
"With pandas it is easy to restructure the order of your columns and rows using the `reindex` function. Let's have a look at an example:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a 1\n",
"b 2\n",
"c 3\n",
"d 4\n",
"e 5\n",
"dtype: int64"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# first define a new Series\n",
"s = pd.Series([1, 2, 3, 4, 5], index=['a', 'b', 'c', 'd', 'e'])\n",
"s"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"d 4\n",
"b 2\n",
"a 1\n",
"c 3\n",
"e 5\n",
"dtype: int64"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Now you can reshuffle the indices\n",
"s = s.reindex(['d', 'b', 'a', 'c', 'e'])\n",
"s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Easy as that! This can also be extended for DataFrames, where you can reorder both the columns and indices at the same time!"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Edinburgh</th>\n",
" <th>Glasgow</th>\n",
" <th>Aberdeen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Edinburgh Glasgow Aberdeen\n",
"a 0 1 2\n",
"b 3 4 5\n",
"c 6 7 8"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# first define a new Dataframe\n",
"data = np.reshape(np.arange(9), (3,3))\n",
"df = pd.DataFrame(data, index=[\"a\", \"b\", \"c\"],\n",
" columns=[\"Edinburgh\", \"Glasgow\", \"Aberdeen\"])\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"# Now we can restructure it with reindex\n",
"df = df.reindex(index=[\"a\", \"d\", \"c\", \"b\"],\n",
" columns=[\"Aberdeen\", \"Glasgow\", \"Edinburgh\", \"Dundee\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice something interesting? We can actually add new indices and columns using the `reindex` method. This results in the new slots in our table to be filled in with `NaN` values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Removing columns/indices <a name=\"removing\"></a>\n",
"Similar to above, it is easy to remove entries. This is done with the `drop()` method and can be applied to both columns and indices:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Edinburgh</th>\n",
" <th>Glasgow</th>\n",
" <th>Aberdeen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Edinburgh Glasgow Aberdeen\n",
"a 0 1 2\n",
"c 6 7 8"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# define new DataFrame\n",
"data = np.reshape(np.arange(9), (3,3))\n",
"df = pd.DataFrame(data, index=[\"a\", \"b\", \"c\"],\n",
" columns=[\"Edinburgh\", \"Glasgow\", \"Aberdeen\"])\n",
"\n",
"df.drop(\"b\")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Glasgow</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>7</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Glasgow\n",
"a 1\n",
"b 4\n",
"c 7"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You can also drop from a column\n",
"df.drop([\"Aberdeen\", \"Edinburgh\"], axis=\"columns\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indexing, slicing and filtering <a name=\"indexing\"></a>\n",
"\n",
"### Indexing\n",
"\n",
"Series indexing works analogously to NumPy array indexing (i.e. `data[...]`). You can also use the Series' index values instead of only integers:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a 0\n",
"b 1\n",
"c 2\n",
"d 3\n",
"dtype: int64"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s = pd.Series(np.arange(4), index=['a', 'b', 'c', 'd'])\n",
"s"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s[1]"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s[3]"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s[\"c\"]"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b 1\n",
"d 3\n",
"dtype: int64"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s[[1,3]]"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a 0\n",
"b 1\n",
"dtype: int64"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s[s<2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A subtle difference when indexing in pandas is that unlike in normal Python, slicing here is inclusive at the end-point."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b 1\n",
"c 2\n",
"dtype: int64"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s[\"b\":\"c\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of the above also apply to DataFrames:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Edinburgh</th>\n",
" <th>Glasgow</th>\n",
" <th>Aberdeen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Edinburgh Glasgow Aberdeen\n",
"a 0 1 2\n",
"b 3 4 5\n",
"c 6 7 8"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = np.reshape(np.arange(9), (3,3))\n",
"df = pd.DataFrame(data, index=[\"a\", \"b\", \"c\"],\n",
" columns=[\"Edinburgh\", \"Glasgow\", \"Aberdeen\"])\n",
"\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Edinburgh</th>\n",
" <th>Glasgow</th>\n",
" <th>Aberdeen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Edinburgh Glasgow Aberdeen\n",
"a 0 1 2\n",
"b 3 4 5"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[:2]"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a 1\n",
"b 4\n",
"c 7\n",
"Name: Glasgow, dtype: int64"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[\"Glasgow\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### loc and iloc\n",
"For DataFrame label-indexing on the rows, you can use `loc` for labels and `iloc` for integer-indexing."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Edinburgh 3\n",
"Glasgow 4\n",
"Aberdeen 5\n",
"Name: b, dtype: int64"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.loc[\"b\"]"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Glasgow 4\n",
"Aberdeen 5\n",
"Name: b, dtype: int64"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.loc[\"b\", [\"Glasgow\", \"Aberdeen\"]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's try `iloc`"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Edinburgh 3\n",
"Glasgow 4\n",
"Aberdeen 5\n",
"Name: b, dtype: int64"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.iloc[1]"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Glasgow 4\n",
"Aberdeen 5\n",
"Name: b, dtype: int64"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.iloc[1, [1,2]]"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Edinburgh</th>\n",
" <th>Glasgow</th>\n",
" <th>Aberdeen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Edinburgh Glasgow Aberdeen\n",
"a 0 1 2\n",
"b 3 4 5"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.iloc[:2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Summary of indexing:\n",
"\n",
"| Type | Notes |\n",
"| -- | -- |\n",
"| df\\[val\\] | Select single column or sequency of columns from a DataFrame |\n",
"| df.loc\\[val\\] | Select single row or subset of rows from a DataFrame by label |\n",
"| df.loc\\[:, val\\] | Select single column or subset of columns by label |\n",
"| df.loc\\[val1, val2\\] | Select both rows and columns by label |\n",
"| df.iloc\\[idx\\] | Select single row or subset of rows from DataFrame by integer position |\n",
"| df.iloc\\[:, idx\\] | Select single column or subset of columns by integer position |\n",
"| df.iloc\\[idx1, idx2\\] | Select both rows and columns by integer position |\n",
"| reindex method | Select either rows or columns by labels |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 1\n",
"A dataset of random numbers is created below. Index the 47th column and the 22nd row. You should get the number **4621**.\n",
"\n",
"*Note: Remember that Python uses 0-based indexing*"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4621"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.DataFrame(np.reshape(np.arange(10000), (100,100)))\n",
"\n",
"df.iloc[46,21]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 2\n",
"Using the same DataFrame from the previous exercise, obtain all rows starting from row 85 to 97."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>...</th>\n",
" <th>90</th>\n",
" <th>91</th>\n",
" <th>92</th>\n",
" <th>93</th>\n",
" <th>94</th>\n",
" <th>95</th>\n",
" <th>96</th>\n",
" <th>97</th>\n",
" <th>98</th>\n",
" <th>99</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>84</th>\n",
" <td>8400</td>\n",
" <td>8401</td>\n",
" <td>8402</td>\n",
" <td>8403</td>\n",
" <td>8404</td>\n",
" <td>8405</td>\n",
" <td>8406</td>\n",
" <td>8407</td>\n",
" <td>8408</td>\n",
" <td>8409</td>\n",
" <td>...</td>\n",
" <td>8490</td>\n",
" <td>8491</td>\n",
" <td>8492</td>\n",
" <td>8493</td>\n",
" <td>8494</td>\n",
" <td>8495</td>\n",
" <td>8496</td>\n",
" <td>8497</td>\n",
" <td>8498</td>\n",
" <td>8499</td>\n",
" </tr>\n",
" <tr>\n",
" <th>85</th>\n",
" <td>8500</td>\n",
" <td>8501</td>\n",
" <td>8502</td>\n",
" <td>8503</td>\n",
" <td>8504</td>\n",
" <td>8505</td>\n",
" <td>8506</td>\n",
" <td>8507</td>\n",
" <td>8508</td>\n",
" <td>8509</td>\n",
" <td>...</td>\n",
" <td>8590</td>\n",
" <td>8591</td>\n",
" <td>8592</td>\n",
" <td>8593</td>\n",
" <td>8594</td>\n",
" <td>8595</td>\n",
" <td>8596</td>\n",
" <td>8597</td>\n",
" <td>8598</td>\n",
" <td>8599</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86</th>\n",
" <td>8600</td>\n",
" <td>8601</td>\n",
" <td>8602</td>\n",
" <td>8603</td>\n",
" <td>8604</td>\n",
" <td>8605</td>\n",
" <td>8606</td>\n",
" <td>8607</td>\n",
" <td>8608</td>\n",
" <td>8609</td>\n",
" <td>...</td>\n",
" <td>8690</td>\n",
" <td>8691</td>\n",
" <td>8692</td>\n",
" <td>8693</td>\n",
" <td>8694</td>\n",
" <td>8695</td>\n",
" <td>8696</td>\n",
" <td>8697</td>\n",
" <td>8698</td>\n",
" <td>8699</td>\n",
" </tr>\n",
" <tr>\n",
" <th>87</th>\n",
" <td>8700</td>\n",
" <td>8701</td>\n",
" <td>8702</td>\n",
" <td>8703</td>\n",
" <td>8704</td>\n",
" <td>8705</td>\n",
" <td>8706</td>\n",
" <td>8707</td>\n",
" <td>8708</td>\n",
" <td>8709</td>\n",
" <td>...</td>\n",
" <td>8790</td>\n",
" <td>8791</td>\n",
" <td>8792</td>\n",
" <td>8793</td>\n",
" <td>8794</td>\n",
" <td>8795</td>\n",
" <td>8796</td>\n",
" <td>8797</td>\n",
" <td>8798</td>\n",
" <td>8799</td>\n",
" </tr>\n",
" <tr>\n",
" <th>88</th>\n",
" <td>8800</td>\n",
" <td>8801</td>\n",
" <td>8802</td>\n",
" <td>8803</td>\n",
" <td>8804</td>\n",
" <td>8805</td>\n",
" <td>8806</td>\n",
" <td>8807</td>\n",
" <td>8808</td>\n",
" <td>8809</td>\n",
" <td>...</td>\n",
" <td>8890</td>\n",
" <td>8891</td>\n",
" <td>8892</td>\n",
" <td>8893</td>\n",
" <td>8894</td>\n",
" <td>8895</td>\n",
" <td>8896</td>\n",
" <td>8897</td>\n",
" <td>8898</td>\n",
" <td>8899</td>\n",
" </tr>\n",
" <tr>\n",
" <th>89</th>\n",
" <td>8900</td>\n",
" <td>8901</td>\n",
" <td>8902</td>\n",
" <td>8903</td>\n",
" <td>8904</td>\n",
" <td>8905</td>\n",
" <td>8906</td>\n",
" <td>8907</td>\n",
" <td>8908</td>\n",
" <td>8909</td>\n",
" <td>...</td>\n",
" <td>8990</td>\n",
" <td>8991</td>\n",
" <td>8992</td>\n",
" <td>8993</td>\n",
" <td>8994</td>\n",
" <td>8995</td>\n",
" <td>8996</td>\n",
" <td>8997</td>\n",
" <td>8998</td>\n",
" <td>8999</td>\n",
" </tr>\n",
" <tr>\n",
" <th>90</th>\n",
" <td>9000</td>\n",
" <td>9001</td>\n",
" <td>9002</td>\n",
" <td>9003</td>\n",
" <td>9004</td>\n",
" <td>9005</td>\n",
" <td>9006</td>\n",
" <td>9007</td>\n",
" <td>9008</td>\n",
" <td>9009</td>\n",
" <td>...</td>\n",
" <td>9090</td>\n",
" <td>9091</td>\n",
" <td>9092</td>\n",
" <td>9093</td>\n",
" <td>9094</td>\n",
" <td>9095</td>\n",
" <td>9096</td>\n",
" <td>9097</td>\n",
" <td>9098</td>\n",
" <td>9099</td>\n",
" </tr>\n",
" <tr>\n",
" <th>91</th>\n",
" <td>9100</td>\n",
" <td>9101</td>\n",
" <td>9102</td>\n",
" <td>9103</td>\n",
" <td>9104</td>\n",
" <td>9105</td>\n",
" <td>9106</td>\n",
" <td>9107</td>\n",
" <td>9108</td>\n",
" <td>9109</td>\n",
" <td>...</td>\n",
" <td>9190</td>\n",
" <td>9191</td>\n",
" <td>9192</td>\n",
" <td>9193</td>\n",
" <td>9194</td>\n",
" <td>9195</td>\n",
" <td>9196</td>\n",
" <td>9197</td>\n",
" <td>9198</td>\n",
" <td>9199</td>\n",
" </tr>\n",
" <tr>\n",
" <th>92</th>\n",
" <td>9200</td>\n",
" <td>9201</td>\n",
" <td>9202</td>\n",
" <td>9203</td>\n",
" <td>9204</td>\n",
" <td>9205</td>\n",
" <td>9206</td>\n",
" <td>9207</td>\n",
" <td>9208</td>\n",
" <td>9209</td>\n",
" <td>...</td>\n",
" <td>9290</td>\n",
" <td>9291</td>\n",
" <td>9292</td>\n",
" <td>9293</td>\n",
" <td>9294</td>\n",
" <td>9295</td>\n",
" <td>9296</td>\n",
" <td>9297</td>\n",
" <td>9298</td>\n",
" <td>9299</td>\n",
" </tr>\n",
" <tr>\n",
" <th>93</th>\n",
" <td>9300</td>\n",
" <td>9301</td>\n",
" <td>9302</td>\n",
" <td>9303</td>\n",
" <td>9304</td>\n",
" <td>9305</td>\n",
" <td>9306</td>\n",
" <td>9307</td>\n",
" <td>9308</td>\n",
" <td>9309</td>\n",
" <td>...</td>\n",
" <td>9390</td>\n",
" <td>9391</td>\n",
" <td>9392</td>\n",
" <td>9393</td>\n",
" <td>9394</td>\n",
" <td>9395</td>\n",
" <td>9396</td>\n",
" <td>9397</td>\n",
" <td>9398</td>\n",
" <td>9399</td>\n",
" </tr>\n",
" <tr>\n",
" <th>94</th>\n",
" <td>9400</td>\n",
" <td>9401</td>\n",
" <td>9402</td>\n",
" <td>9403</td>\n",
" <td>9404</td>\n",
" <td>9405</td>\n",
" <td>9406</td>\n",
" <td>9407</td>\n",
" <td>9408</td>\n",
" <td>9409</td>\n",
" <td>...</td>\n",
" <td>9490</td>\n",
" <td>9491</td>\n",
" <td>9492</td>\n",
" <td>9493</td>\n",
" <td>9494</td>\n",
" <td>9495</td>\n",
" <td>9496</td>\n",
" <td>9497</td>\n",
" <td>9498</td>\n",
" <td>9499</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>9500</td>\n",
" <td>9501</td>\n",
" <td>9502</td>\n",
" <td>9503</td>\n",
" <td>9504</td>\n",
" <td>9505</td>\n",
" <td>9506</td>\n",
" <td>9507</td>\n",
" <td>9508</td>\n",
" <td>9509</td>\n",
" <td>...</td>\n",
" <td>9590</td>\n",
" <td>9591</td>\n",
" <td>9592</td>\n",
" <td>9593</td>\n",
" <td>9594</td>\n",
" <td>9595</td>\n",
" <td>9596</td>\n",
" <td>9597</td>\n",
" <td>9598</td>\n",
" <td>9599</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>12 rows × 100 columns</p>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3 4 5 6 7 8 9 ... 90 \\\n",
"84 8400 8401 8402 8403 8404 8405 8406 8407 8408 8409 ... 8490 \n",
"85 8500 8501 8502 8503 8504 8505 8506 8507 8508 8509 ... 8590 \n",
"86 8600 8601 8602 8603 8604 8605 8606 8607 8608 8609 ... 8690 \n",
"87 8700 8701 8702 8703 8704 8705 8706 8707 8708 8709 ... 8790 \n",
"88 8800 8801 8802 8803 8804 8805 8806 8807 8808 8809 ... 8890 \n",
"89 8900 8901 8902 8903 8904 8905 8906 8907 8908 8909 ... 8990 \n",
"90 9000 9001 9002 9003 9004 9005 9006 9007 9008 9009 ... 9090 \n",
"91 9100 9101 9102 9103 9104 9105 9106 9107 9108 9109 ... 9190 \n",
"92 9200 9201 9202 9203 9204 9205 9206 9207 9208 9209 ... 9290 \n",
"93 9300 9301 9302 9303 9304 9305 9306 9307 9308 9309 ... 9390 \n",
"94 9400 9401 9402 9403 9404 9405 9406 9407 9408 9409 ... 9490 \n",
"95 9500 9501 9502 9503 9504 9505 9506 9507 9508 9509 ... 9590 \n",
"\n",
" 91 92 93 94 95 96 97 98 99 \n",
"84 8491 8492 8493 8494 8495 8496 8497 8498 8499 \n",
"85 8591 8592 8593 8594 8595 8596 8597 8598 8599 \n",
"86 8691 8692 8693 8694 8695 8696 8697 8698 8699 \n",
"87 8791 8792 8793 8794 8795 8796 8797 8798 8799 \n",
"88 8891 8892 8893 8894 8895 8896 8897 8898 8899 \n",
"89 8991 8992 8993 8994 8995 8996 8997 8998 8999 \n",
"90 9091 9092 9093 9094 9095 9096 9097 9098 9099 \n",
"91 9191 9192 9193 9194 9195 9196 9197 9198 9199 \n",
"92 9291 9292 9293 9294 9295 9296 9297 9298 9299 \n",
"93 9391 9392 9393 9394 9395 9396 9397 9398 9399 \n",
"94 9491 9492 9493 9494 9495 9496 9497 9498 9499 \n",
"95 9591 9592 9593 9594 9595 9596 9597 9598 9599 \n",
"\n",
"[12 rows x 100 columns]"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.iloc[84:96]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Arithmetic <a name=\"arithmetic\"></a>\n",
"When you are performing arithmetic operations between two objects, if any index pairs are not the same, the respective index in the result will be the union of the index pair. Let's have a look"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a NaN\n",
"b 1.0\n",
"c 3.0\n",
"d 5.0\n",
"e NaN\n",
"k NaN\n",
"dtype: float64"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s1 = pd.Series(np.arange(5), index=[\"a\", \"b\", \"c\", \"d\", \"e\"])\n",
"s2 = pd.Series(np.arange(4), index=[\"b\", \"c\", \"d\", \"k\"])\n",
"s1 + s2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The internal data alignment introduces missing values in the label locations that don't overlap. It is similar for DataFrames:"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c d\n",
"0 0 1 2 3\n",
"1 4 5 6 7\n",
"2 8 9 10 11"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1 = pd.DataFrame(np.arange(12).reshape((3,4)),\n",
" columns=list(\"abcd\"))\n",
"df1"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>12</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>15</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" c d e f\n",
"0 0 1 2 3\n",
"1 4 5 6 7\n",
"2 8 9 10 11\n",
"3 12 13 14 15"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df2 = pd.DataFrame(np.arange(16).reshape((4,4)),\n",
" columns=list(\"cdef\"))\n",
"df2"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2.0</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>10.0</td>\n",
" <td>12.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>18.0</td>\n",
" <td>20.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c d e f\n",
"0 NaN NaN 2.0 4.0 NaN NaN\n",
"1 NaN NaN 10.0 12.0 NaN NaN\n",
"2 NaN NaN 18.0 20.0 NaN NaN\n",
"3 NaN NaN NaN NaN NaN NaN"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# adding the two\n",
"df1+df2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how where we don't have matching values from `df1` and `df2` the output of the addition operation is `NaN` since there are no two numbers to add.\n",
"\n",
"Well, we can \"fix\" that by filling in the `NaN` values. This effectively tells pandas where there are no two values to add, assume that the missing value is just zero."
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>2.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.0</td>\n",
" <td>5.0</td>\n",
" <td>10.0</td>\n",
" <td>12.0</td>\n",
" <td>6.0</td>\n",
" <td>7.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8.0</td>\n",
" <td>9.0</td>\n",
" <td>18.0</td>\n",
" <td>20.0</td>\n",
" <td>10.0</td>\n",
" <td>11.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>12.0</td>\n",
" <td>13.0</td>\n",
" <td>14.0</td>\n",
" <td>15.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c d e f\n",
"0 0.0 1.0 2.0 4.0 2.0 3.0\n",
"1 4.0 5.0 10.0 12.0 6.0 7.0\n",
"2 8.0 9.0 18.0 20.0 10.0 11.0\n",
"3 NaN NaN 12.0 13.0 14.0 15.0"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1.add(df2, fill_value=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another important point here is although the normal arithmetic operations work here, there also exist dedicated methods like `DataFrame.add()` which achieve the same functionality + a bit extra.\n",
"\n",
"Here's a list of all arithmetic operations within pandas:\n",
"\n",
"| Operator | Method | Description |\n",
"| -- | -- | -- |\n",
"| + | add, radd | Addition |\n",
"| - | sub, rsub | Subtraction |\n",
"| / | div, rdiv | Division |\n",
"| // | floordiv, rfloordiv | Floor division |\n",
"| * | mul, rmul | Multiplication |\n",
"| ** | pow, rpow | Exponentiation |\n",
"\n",
"Notice how some of the methods have `r` in front of them? That stands for reversed and effectively reverses the operands. For example\n",
"\n",
"```python\n",
"df1.div(df2)\n",
"```\n",
"would be the same as\n",
"```python\n",
"df1/df2\n",
"```\n",
"\n",
"but....\n",
"```python\n",
"df1.rdiv(df2)\n",
"```\n",
"would be the same as\n",
"```python\n",
"df2/df1\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 3\n",
"Create a (3,3) DataFrame and square all elements in it."
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>9</td>\n",
" <td>16</td>\n",
" <td>25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>36</td>\n",
" <td>49</td>\n",
" <td>64</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 0 1 4\n",
"1 9 16 25\n",
"2 36 49 64"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.DataFrame(np.arange(9).reshape(3,3)) ** 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Broadcasting\n",
"Similar to numpy, in pandas you can also broadcast data structures. Let's consider a simple example:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>12</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>15</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 0 1 2 3\n",
"1 4 5 6 7\n",
"2 8 9 10 11\n",
"3 12 13 14 15"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1 = pd.DataFrame(np.arange(16).reshape((4,4)))\n",
"df1"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 0\n",
"1 1\n",
"2 2\n",
"3 3\n",
"dtype: int64"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df2 = pd.Series(np.arange(4))\n",
"df2"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>8</td>\n",
" <td>8</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>12</td>\n",
" <td>12</td>\n",
" <td>12</td>\n",
" <td>12</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 0 0 0 0\n",
"1 4 4 4 4\n",
"2 8 8 8 8\n",
"3 12 12 12 12"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1 - df2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how the Series of [0, 1, 2, 3] got removed from each row? That is called broadcasting.\n",
"\n",
"It can also be used for columns, but for that, you have to use the method arithmetic operations."
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" <td>12</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 0 1 2 3\n",
"1 3 4 5 6\n",
"2 6 7 8 9\n",
"3 9 10 11 12"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1.sub(df2, axis=\"index\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sorting\n",
"Sorting is an important built-in operation of pandas. Let's have a look at how you can do it:"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>d</th>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>12</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>15</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"b 0 1 2 3\n",
"a 4 5 6 7\n",
"d 8 9 10 11\n",
"c 12 13 14 15"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1 = pd.DataFrame(np.arange(16).reshape((4,4)), index=[\"b\", \"a\", \"d\", \"c\"])\n",
"df1"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>12</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>15</td>\n",
" </tr>\n",
" <tr>\n",
" <th>d</th>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"a 4 5 6 7\n",
"b 0 1 2 3\n",
"c 12 13 14 15\n",
"d 8 9 10 11"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df2 = df1.sort_index()\n",
"df2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Easy as that. Furthermore, you can also sort along the column axis with\n",
"```python\n",
"df1.sort_index(axis=1)\n",
"```\n",
"\n",
"You can also sort by the actual values inside, but you have to give the column by which you want to sort."
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a\n",
"0 4\n",
"1 3\n",
"2 6\n",
"3 1\n",
"4 3\n",
"5 5"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1 = pd.DataFrame([4, 3, 6, 1, 3, 5], columns=[\"a\"])\n",
"df1"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a\n",
"3 1\n",
"1 3\n",
"4 3\n",
"0 4\n",
"5 5\n",
"2 6"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1.sort_values(by=\"a\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summarizing and computing descriptive stats <a name=\"sums\"></a>\n",
"`pandas` is equipped with common mathematical and statistical methods. Most of which fall into the category of reductions or summary statistics. These are methods that extract a single value from a list of values. For example, you can extract the mean of a `Series` object like this:"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>12</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>15</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>16</td>\n",
" <td>17</td>\n",
" <td>18</td>\n",
" <td>19</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c d\n",
"0 0 1 2 3\n",
"1 4 5 6 7\n",
"2 8 9 10 11\n",
"3 12 13 14 15\n",
"4 16 17 18 19"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.DataFrame(np.arange(20).reshape(5,4),\n",
" columns=[\"a\", \"b\", \"c\", \"d\"])\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a 40\n",
"b 45\n",
"c 50\n",
"d 55\n",
"dtype: int64"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how that created the sum of each column?\n",
"\n",
"Well you can actually make that the other way around by adding an extra option to `sum()`"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 6\n",
"1 22\n",
"2 38\n",
"3 54\n",
"4 70\n",
"dtype: int64"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.sum(axis=\"columns\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A similar method also exists for obtaining the mean of data:"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"a 8.0\n",
"b 9.0\n",
"c 10.0\n",
"d 11.0\n",
"dtype: float64"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, the mother of the methods we discussed here is `describe()` "
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>5.000000</td>\n",
" <td>5.000000</td>\n",
" <td>5.000000</td>\n",
" <td>5.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>8.000000</td>\n",
" <td>9.000000</td>\n",
" <td>10.000000</td>\n",
" <td>11.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>6.324555</td>\n",
" <td>6.324555</td>\n",
" <td>6.324555</td>\n",
" <td>6.324555</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>0.000000</td>\n",
" <td>1.000000</td>\n",
" <td>2.000000</td>\n",
" <td>3.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>4.000000</td>\n",
" <td>5.000000</td>\n",
" <td>6.000000</td>\n",
" <td>7.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>8.000000</td>\n",
" <td>9.000000</td>\n",
" <td>10.000000</td>\n",
" <td>11.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>12.000000</td>\n",
" <td>13.000000</td>\n",
" <td>14.000000</td>\n",
" <td>15.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>16.000000</td>\n",
" <td>17.000000</td>\n",
" <td>18.000000</td>\n",
" <td>19.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c d\n",
"count 5.000000 5.000000 5.000000 5.000000\n",
"mean 8.000000 9.000000 10.000000 11.000000\n",
"std 6.324555 6.324555 6.324555 6.324555\n",
"min 0.000000 1.000000 2.000000 3.000000\n",
"25% 4.000000 5.000000 6.000000 7.000000\n",
"50% 8.000000 9.000000 10.000000 11.000000\n",
"75% 12.000000 13.000000 14.000000 15.000000\n",
"max 16.000000 17.000000 18.000000 19.000000"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are some of the summary methods:\n",
"\n",
"| Method | Description |\n",
"| -- | -- |\n",
"| count | Return number of non-NA values |\n",
"| describe | Set of summary statistics |\n",
"| min, max | Minimum, maximum values |\n",
"| argmin, argmax | Index locations at which the minimum or maximum value is obtained | \n",
"| sum | Sum of values |\n",
"| mean | Mean of values |\n",
"| median | Arithmetic median of values |\n",
"| std | Sample standard deviation of values\n",
"| value_counts() | Counts the number of occurrences of each unique element in a column |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 4\n",
"\n",
"A random DataFrame is created below. Find it's mean and standard deviation, then normalise it column-wise according to the formula:\n",
"\n",
"$$ Y = \\frac{X - \\mu}{\\sigma} $$\n",
"\n",
"Where X is your dataset, $\\mu$ is the mean and $\\sigma$ is the standard deviation.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>...</th>\n",
" <th>90</th>\n",
" <th>91</th>\n",
" <th>92</th>\n",
" <th>93</th>\n",
" <th>94</th>\n",
" <th>95</th>\n",
" <th>96</th>\n",
" <th>97</th>\n",
" <th>98</th>\n",
" <th>99</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.513870</td>\n",
" <td>1.396583</td>\n",
" <td>-0.246373</td>\n",
" <td>0.412241</td>\n",
" <td>1.579367</td>\n",
" <td>0.363903</td>\n",
" <td>-1.323891</td>\n",
" <td>-1.113526</td>\n",
" <td>-0.877806</td>\n",
" <td>0.881624</td>\n",
" <td>...</td>\n",
" <td>1.061778</td>\n",
" <td>1.647589</td>\n",
" <td>0.608297</td>\n",
" <td>0.931085</td>\n",
" <td>-0.305219</td>\n",
" <td>-0.679682</td>\n",
" <td>1.323216</td>\n",
" <td>-1.000440</td>\n",
" <td>1.158411</td>\n",
" <td>0.271465</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.631868</td>\n",
" <td>-1.075335</td>\n",
" <td>0.170484</td>\n",
" <td>1.453664</td>\n",
" <td>1.357602</td>\n",
" <td>-1.597306</td>\n",
" <td>1.342091</td>\n",
" <td>-1.775401</td>\n",
" <td>1.129466</td>\n",
" <td>-1.180717</td>\n",
" <td>...</td>\n",
" <td>-1.806966</td>\n",
" <td>-0.856221</td>\n",
" <td>0.223695</td>\n",
" <td>-0.972374</td>\n",
" <td>0.176362</td>\n",
" <td>0.792399</td>\n",
" <td>-0.528541</td>\n",
" <td>0.796424</td>\n",
" <td>0.728051</td>\n",
" <td>-1.104085</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>-0.514866</td>\n",
" <td>0.374350</td>\n",
" <td>-1.009489</td>\n",
" <td>0.086802</td>\n",
" <td>-1.696162</td>\n",
" <td>0.194635</td>\n",
" <td>1.231866</td>\n",
" <td>1.121161</td>\n",
" <td>0.736265</td>\n",
" <td>-0.563802</td>\n",
" <td>...</td>\n",
" <td>-1.273403</td>\n",
" <td>-1.178026</td>\n",
" <td>1.542241</td>\n",
" <td>1.006310</td>\n",
" <td>0.538693</td>\n",
" <td>0.789394</td>\n",
" <td>0.540420</td>\n",
" <td>0.742471</td>\n",
" <td>0.203087</td>\n",
" <td>-1.486384</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.886998</td>\n",
" <td>0.052952</td>\n",
" <td>1.716351</td>\n",
" <td>-0.169000</td>\n",
" <td>-1.037321</td>\n",
" <td>-1.554467</td>\n",
" <td>1.375881</td>\n",
" <td>1.256623</td>\n",
" <td>-0.206046</td>\n",
" <td>-0.299600</td>\n",
" <td>...</td>\n",
" <td>0.967154</td>\n",
" <td>1.519549</td>\n",
" <td>1.556544</td>\n",
" <td>-0.068761</td>\n",
" <td>1.182859</td>\n",
" <td>-0.647793</td>\n",
" <td>-0.700245</td>\n",
" <td>0.705043</td>\n",
" <td>-0.825873</td>\n",
" <td>1.851311</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1.391114</td>\n",
" <td>0.418370</td>\n",
" <td>-1.037966</td>\n",
" <td>0.312837</td>\n",
" <td>-0.964962</td>\n",
" <td>-1.330757</td>\n",
" <td>0.641659</td>\n",
" <td>1.299796</td>\n",
" <td>0.038224</td>\n",
" <td>0.356430</td>\n",
" <td>...</td>\n",
" <td>-0.912055</td>\n",
" <td>0.942265</td>\n",
" <td>-0.054322</td>\n",
" <td>0.174229</td>\n",
" <td>-0.782228</td>\n",
" <td>-0.613781</td>\n",
" <td>-1.241414</td>\n",
" <td>0.528770</td>\n",
" <td>-1.546610</td>\n",
" <td>-0.655011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>-0.978640</td>\n",
" <td>0.882280</td>\n",
" <td>1.098095</td>\n",
" <td>1.637879</td>\n",
" <td>1.623599</td>\n",
" <td>1.164122</td>\n",
" <td>0.459580</td>\n",
" <td>-1.200620</td>\n",
" <td>-0.101569</td>\n",
" <td>-1.025493</td>\n",
" <td>...</td>\n",
" <td>-0.178119</td>\n",
" <td>0.153881</td>\n",
" <td>-1.489814</td>\n",
" <td>0.010346</td>\n",
" <td>1.043027</td>\n",
" <td>1.248067</td>\n",
" <td>1.028140</td>\n",
" <td>-0.870393</td>\n",
" <td>-0.706648</td>\n",
" <td>1.744279</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>0.735374</td>\n",
" <td>1.603069</td>\n",
" <td>-0.607167</td>\n",
" <td>1.751502</td>\n",
" <td>1.457694</td>\n",
" <td>-1.035909</td>\n",
" <td>0.547590</td>\n",
" <td>0.782969</td>\n",
" <td>1.545038</td>\n",
" <td>1.538226</td>\n",
" <td>...</td>\n",
" <td>0.969993</td>\n",
" <td>-0.325719</td>\n",
" <td>0.994623</td>\n",
" <td>-1.531772</td>\n",
" <td>1.115576</td>\n",
" <td>0.459712</td>\n",
" <td>0.949289</td>\n",
" <td>-0.787418</td>\n",
" <td>-0.295029</td>\n",
" <td>0.903614</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>-1.348100</td>\n",
" <td>-0.399150</td>\n",
" <td>-1.218192</td>\n",
" <td>1.062848</td>\n",
" <td>0.749450</td>\n",
" <td>-1.029018</td>\n",
" <td>0.392363</td>\n",
" <td>0.421657</td>\n",
" <td>0.277607</td>\n",
" <td>1.341795</td>\n",
" <td>...</td>\n",
" <td>0.564476</td>\n",
" <td>-1.216358</td>\n",
" <td>1.071405</td>\n",
" <td>-0.202719</td>\n",
" <td>0.165891</td>\n",
" <td>-0.592823</td>\n",
" <td>0.805909</td>\n",
" <td>0.591497</td>\n",
" <td>-1.093653</td>\n",
" <td>-1.055484</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>1.285660</td>\n",
" <td>0.526640</td>\n",
" <td>-0.327848</td>\n",
" <td>-0.794804</td>\n",
" <td>0.136058</td>\n",
" <td>0.355371</td>\n",
" <td>1.387508</td>\n",
" <td>-0.973951</td>\n",
" <td>-1.477451</td>\n",
" <td>1.033896</td>\n",
" <td>...</td>\n",
" <td>0.944201</td>\n",
" <td>-0.329253</td>\n",
" <td>0.670113</td>\n",
" <td>0.898907</td>\n",
" <td>-1.234780</td>\n",
" <td>-1.424890</td>\n",
" <td>1.544366</td>\n",
" <td>-0.845917</td>\n",
" <td>0.405283</td>\n",
" <td>0.693724</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>1.564215</td>\n",
" <td>-0.530578</td>\n",
" <td>-1.378221</td>\n",
" <td>1.484833</td>\n",
" <td>-0.121855</td>\n",
" <td>-1.147937</td>\n",
" <td>-0.646704</td>\n",
" <td>1.363795</td>\n",
" <td>-1.422009</td>\n",
" <td>-0.908234</td>\n",
" <td>...</td>\n",
" <td>-1.839381</td>\n",
" <td>0.365275</td>\n",
" <td>1.490728</td>\n",
" <td>-0.444500</td>\n",
" <td>0.555258</td>\n",
" <td>1.260674</td>\n",
" <td>-0.156228</td>\n",
" <td>-1.821950</td>\n",
" <td>1.436147</td>\n",
" <td>-1.629020</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>-1.512300</td>\n",
" <td>1.257621</td>\n",
" <td>-0.950904</td>\n",
" <td>0.097102</td>\n",
" <td>0.066319</td>\n",
" <td>1.201285</td>\n",
" <td>-1.479766</td>\n",
" <td>1.081254</td>\n",
" <td>-0.700119</td>\n",
" <td>0.551702</td>\n",
" <td>...</td>\n",
" <td>-0.511107</td>\n",
" <td>-0.801593</td>\n",
" <td>-0.291530</td>\n",
" <td>-0.149648</td>\n",
" <td>-0.885823</td>\n",
" <td>-0.679197</td>\n",
" <td>-1.809390</td>\n",
" <td>1.392497</td>\n",
" <td>1.610417</td>\n",
" <td>0.636615</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>0.676772</td>\n",
" <td>-0.382604</td>\n",
" <td>0.152292</td>\n",
" <td>1.184203</td>\n",
" <td>0.885681</td>\n",
" <td>-1.561482</td>\n",
" <td>0.544845</td>\n",
" <td>0.825152</td>\n",
" <td>0.851938</td>\n",
" <td>0.386944</td>\n",
" <td>...</td>\n",
" <td>0.706691</td>\n",
" <td>-1.175663</td>\n",
" <td>-0.441140</td>\n",
" <td>-1.470221</td>\n",
" <td>-0.088202</td>\n",
" <td>-0.315637</td>\n",
" <td>0.481602</td>\n",
" <td>0.378751</td>\n",
" <td>-1.696612</td>\n",
" <td>-0.433806</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>0.240678</td>\n",
" <td>-1.393210</td>\n",
" <td>0.187365</td>\n",
" <td>-0.798240</td>\n",
" <td>0.226787</td>\n",
" <td>-0.633688</td>\n",
" <td>-0.989422</td>\n",
" <td>-1.293567</td>\n",
" <td>1.050305</td>\n",
" <td>0.351630</td>\n",
" <td>...</td>\n",
" <td>1.583936</td>\n",
" <td>0.168961</td>\n",
" <td>-0.907365</td>\n",
" <td>1.536351</td>\n",
" <td>0.692415</td>\n",
" <td>-0.558356</td>\n",
" <td>1.183726</td>\n",
" <td>0.025276</td>\n",
" <td>0.922382</td>\n",
" <td>-0.886471</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>0.340047</td>\n",
" <td>0.938168</td>\n",
" <td>-1.118608</td>\n",
" <td>-0.523279</td>\n",
" <td>-0.761692</td>\n",
" <td>0.586611</td>\n",
" <td>0.016273</td>\n",
" <td>-0.059960</td>\n",
" <td>-0.971746</td>\n",
" <td>0.203671</td>\n",
" <td>...</td>\n",
" <td>-1.582610</td>\n",
" <td>-1.445128</td>\n",
" <td>0.929069</td>\n",
" <td>0.529699</td>\n",
" <td>-0.288482</td>\n",
" <td>0.100620</td>\n",
" <td>-0.339643</td>\n",
" <td>-1.342788</td>\n",
" <td>0.285311</td>\n",
" <td>-1.530031</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>-0.216206</td>\n",
" <td>-0.081457</td>\n",
" <td>-0.066678</td>\n",
" <td>0.712012</td>\n",
" <td>-0.933661</td>\n",
" <td>0.107191</td>\n",
" <td>1.538183</td>\n",
" <td>1.312979</td>\n",
" <td>1.597155</td>\n",
" <td>0.204361</td>\n",
" <td>...</td>\n",
" <td>0.851806</td>\n",
" <td>1.574912</td>\n",
" <td>-1.057158</td>\n",
" <td>0.897952</td>\n",
" <td>0.334998</td>\n",
" <td>-1.499413</td>\n",
" <td>-0.557911</td>\n",
" <td>0.893911</td>\n",
" <td>-1.525342</td>\n",
" <td>-0.772764</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>-0.393874</td>\n",
" <td>-0.217956</td>\n",
" <td>0.505092</td>\n",
" <td>1.288478</td>\n",
" <td>0.743926</td>\n",
" <td>1.529977</td>\n",
" <td>-1.468972</td>\n",
" <td>0.813537</td>\n",
" <td>-1.114886</td>\n",
" <td>-0.573397</td>\n",
" <td>...</td>\n",
" <td>-0.397053</td>\n",
" <td>0.327747</td>\n",
" <td>0.515145</td>\n",
" <td>0.648666</td>\n",
" <td>-0.041444</td>\n",
" <td>0.037236</td>\n",
" <td>0.563206</td>\n",
" <td>1.500904</td>\n",
" <td>0.985118</td>\n",
" <td>1.295535</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>1.448163</td>\n",
" <td>-1.468940</td>\n",
" <td>0.296056</td>\n",
" <td>-0.724831</td>\n",
" <td>0.647826</td>\n",
" <td>-0.180198</td>\n",
" <td>-1.700891</td>\n",
" <td>0.269400</td>\n",
" <td>0.671001</td>\n",
" <td>0.743764</td>\n",
" <td>...</td>\n",
" <td>-1.011690</td>\n",
" <td>0.961830</td>\n",
" <td>-0.257341</td>\n",
" <td>-1.322110</td>\n",
" <td>-0.140934</td>\n",
" <td>1.127069</td>\n",
" <td>0.961686</td>\n",
" <td>1.278722</td>\n",
" <td>1.053308</td>\n",
" <td>0.611221</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>0.909765</td>\n",
" <td>1.254015</td>\n",
" <td>1.629871</td>\n",
" <td>-0.727945</td>\n",
" <td>0.364806</td>\n",
" <td>-1.244102</td>\n",
" <td>-0.660909</td>\n",
" <td>-1.646826</td>\n",
" <td>1.657767</td>\n",
" <td>1.276033</td>\n",
" <td>...</td>\n",
" <td>-0.281521</td>\n",
" <td>1.723603</td>\n",
" <td>-1.584701</td>\n",
" <td>-0.319174</td>\n",
" <td>-1.711071</td>\n",
" <td>-1.183856</td>\n",
" <td>1.509604</td>\n",
" <td>0.526637</td>\n",
" <td>-0.805211</td>\n",
" <td>-1.732134</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>-0.734810</td>\n",
" <td>-0.748138</td>\n",
" <td>1.446927</td>\n",
" <td>0.207259</td>\n",
" <td>0.760452</td>\n",
" <td>1.116326</td>\n",
" <td>-0.842161</td>\n",
" <td>-0.789068</td>\n",
" <td>0.532212</td>\n",
" <td>0.759320</td>\n",
" <td>...</td>\n",
" <td>1.465185</td>\n",
" <td>0.401349</td>\n",
" <td>-0.966394</td>\n",
" <td>-0.362602</td>\n",
" <td>1.127971</td>\n",
" <td>0.752127</td>\n",
" <td>-0.106078</td>\n",
" <td>1.265856</td>\n",
" <td>0.211697</td>\n",
" <td>0.269893</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>0.190232</td>\n",
" <td>-0.380382</td>\n",
" <td>-0.956613</td>\n",
" <td>0.161797</td>\n",
" <td>0.358565</td>\n",
" <td>1.054044</td>\n",
" <td>-0.502211</td>\n",
" <td>1.479995</td>\n",
" <td>0.543587</td>\n",
" <td>-0.610027</td>\n",
" <td>...</td>\n",
" <td>0.905991</td>\n",
" <td>0.590617</td>\n",
" <td>0.979145</td>\n",
" <td>-1.668567</td>\n",
" <td>1.387817</td>\n",
" <td>-0.542651</td>\n",
" <td>-1.742281</td>\n",
" <td>-1.127449</td>\n",
" <td>-1.631693</td>\n",
" <td>0.071839</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>0.171497</td>\n",
" <td>-0.568547</td>\n",
" <td>-0.181602</td>\n",
" <td>1.280796</td>\n",
" <td>-0.607141</td>\n",
" <td>-1.314850</td>\n",
" <td>0.877709</td>\n",
" <td>-1.303426</td>\n",
" <td>-0.263439</td>\n",
" <td>-0.578028</td>\n",
" <td>...</td>\n",
" <td>0.463058</td>\n",
" <td>-0.343679</td>\n",
" <td>-1.369579</td>\n",
" <td>1.682706</td>\n",
" <td>-0.492890</td>\n",
" <td>1.073619</td>\n",
" <td>1.642233</td>\n",
" <td>-1.790007</td>\n",
" <td>-0.242860</td>\n",
" <td>-0.465673</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>-1.773808</td>\n",
" <td>-0.806683</td>\n",
" <td>1.710190</td>\n",
" <td>1.529340</td>\n",
" <td>0.149329</td>\n",
" <td>0.015717</td>\n",
" <td>-1.664649</td>\n",
" <td>-0.333638</td>\n",
" <td>0.959662</td>\n",
" <td>-1.626720</td>\n",
" <td>...</td>\n",
" <td>-0.076300</td>\n",
" <td>0.181243</td>\n",
" <td>-0.343203</td>\n",
" <td>1.127871</td>\n",
" <td>-1.857502</td>\n",
" <td>-0.760829</td>\n",
" <td>-1.716068</td>\n",
" <td>-0.238514</td>\n",
" <td>0.830074</td>\n",
" <td>1.099593</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>-0.232519</td>\n",
" <td>-0.199589</td>\n",
" <td>0.034674</td>\n",
" <td>-0.284041</td>\n",
" <td>0.924887</td>\n",
" <td>1.394155</td>\n",
" <td>0.390990</td>\n",
" <td>0.467035</td>\n",
" <td>-0.711218</td>\n",
" <td>-0.232838</td>\n",
" <td>...</td>\n",
" <td>0.516714</td>\n",
" <td>0.670454</td>\n",
" <td>1.216928</td>\n",
" <td>1.133620</td>\n",
" <td>-1.013026</td>\n",
" <td>-1.129269</td>\n",
" <td>1.325263</td>\n",
" <td>1.292575</td>\n",
" <td>0.501611</td>\n",
" <td>1.526500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>-1.171088</td>\n",
" <td>0.141293</td>\n",
" <td>1.626429</td>\n",
" <td>0.168721</td>\n",
" <td>-1.727745</td>\n",
" <td>1.386053</td>\n",
" <td>0.263236</td>\n",
" <td>1.137646</td>\n",
" <td>0.360819</td>\n",
" <td>1.443515</td>\n",
" <td>...</td>\n",
" <td>0.862208</td>\n",
" <td>-1.184068</td>\n",
" <td>1.456381</td>\n",
" <td>0.892589</td>\n",
" <td>-0.674029</td>\n",
" <td>-1.409532</td>\n",
" <td>0.050445</td>\n",
" <td>-0.772777</td>\n",
" <td>1.630130</td>\n",
" <td>1.186272</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>-1.583545</td>\n",
" <td>0.526075</td>\n",
" <td>-0.406596</td>\n",
" <td>-1.025639</td>\n",
" <td>-1.104152</td>\n",
" <td>-1.492721</td>\n",
" <td>0.094901</td>\n",
" <td>-1.687825</td>\n",
" <td>0.522421</td>\n",
" <td>-0.947742</td>\n",
" <td>...</td>\n",
" <td>-0.274502</td>\n",
" <td>-1.410834</td>\n",
" <td>-0.511294</td>\n",
" <td>-1.479222</td>\n",
" <td>1.539949</td>\n",
" <td>-0.185147</td>\n",
" <td>-1.623003</td>\n",
" <td>1.281739</td>\n",
" <td>0.106110</td>\n",
" <td>-0.587262</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>-0.247518</td>\n",
" <td>-0.116995</td>\n",
" <td>0.005844</td>\n",
" <td>1.353575</td>\n",
" <td>1.295124</td>\n",
" <td>-0.134358</td>\n",
" <td>1.053640</td>\n",
" <td>-1.656545</td>\n",
" <td>-0.071846</td>\n",
" <td>0.962144</td>\n",
" <td>...</td>\n",
" <td>-1.318647</td>\n",
" <td>1.559977</td>\n",
" <td>-0.074145</td>\n",
" <td>-0.378504</td>\n",
" <td>0.869430</td>\n",
" <td>-0.069882</td>\n",
" <td>-1.669445</td>\n",
" <td>-0.498185</td>\n",
" <td>-1.576615</td>\n",
" <td>-0.777916</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>1.587095</td>\n",
" <td>0.230919</td>\n",
" <td>0.362958</td>\n",
" <td>-1.250820</td>\n",
" <td>1.657170</td>\n",
" <td>-0.718975</td>\n",
" <td>0.330160</td>\n",
" <td>-0.126438</td>\n",
" <td>-1.220022</td>\n",
" <td>-1.125991</td>\n",
" <td>...</td>\n",
" <td>-0.649107</td>\n",
" <td>1.306017</td>\n",
" <td>0.482203</td>\n",
" <td>0.145802</td>\n",
" <td>1.448299</td>\n",
" <td>-0.520351</td>\n",
" <td>-1.056655</td>\n",
" <td>-0.805319</td>\n",
" <td>-1.254778</td>\n",
" <td>0.418933</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>0.935906</td>\n",
" <td>-0.145264</td>\n",
" <td>-1.250488</td>\n",
" <td>-1.470936</td>\n",
" <td>1.506688</td>\n",
" <td>-1.313323</td>\n",
" <td>1.518933</td>\n",
" <td>0.610692</td>\n",
" <td>-0.491017</td>\n",
" <td>-1.627820</td>\n",
" <td>...</td>\n",
" <td>0.902349</td>\n",
" <td>1.508875</td>\n",
" <td>-1.127943</td>\n",
" <td>-0.570377</td>\n",
" <td>0.352445</td>\n",
" <td>-0.110769</td>\n",
" <td>-1.567606</td>\n",
" <td>-1.748038</td>\n",
" <td>-1.510443</td>\n",
" <td>0.548930</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>0.719676</td>\n",
" <td>-0.814732</td>\n",
" <td>-1.725769</td>\n",
" <td>-1.235741</td>\n",
" <td>-0.586766</td>\n",
" <td>0.796515</td>\n",
" <td>-1.080151</td>\n",
" <td>0.608303</td>\n",
" <td>0.782711</td>\n",
" <td>-0.265900</td>\n",
" <td>...</td>\n",
" <td>1.390754</td>\n",
" <td>-1.492590</td>\n",
" <td>-0.500209</td>\n",
" <td>-0.435704</td>\n",
" <td>0.911975</td>\n",
" <td>1.868319</td>\n",
" <td>-0.857274</td>\n",
" <td>1.058341</td>\n",
" <td>-0.547400</td>\n",
" <td>0.169497</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>-0.209103</td>\n",
" <td>1.559806</td>\n",
" <td>-0.736238</td>\n",
" <td>0.930045</td>\n",
" <td>-1.183643</td>\n",
" <td>-1.525615</td>\n",
" <td>0.653475</td>\n",
" <td>-1.798585</td>\n",
" <td>-1.294609</td>\n",
" <td>0.813010</td>\n",
" <td>...</td>\n",
" <td>-1.518635</td>\n",
" <td>0.192210</td>\n",
" <td>-0.855648</td>\n",
" <td>-1.609667</td>\n",
" <td>-0.225023</td>\n",
" <td>-0.247551</td>\n",
" <td>-0.314739</td>\n",
" <td>0.907275</td>\n",
" <td>-1.162187</td>\n",
" <td>0.812885</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>70</th>\n",
" <td>-0.374306</td>\n",
" <td>-0.509877</td>\n",
" <td>1.139351</td>\n",
" <td>0.463223</td>\n",
" <td>-0.122952</td>\n",
" <td>0.257494</td>\n",
" <td>0.984939</td>\n",
" <td>0.808924</td>\n",
" <td>-0.961834</td>\n",
" <td>-0.648294</td>\n",
" <td>...</td>\n",
" <td>0.403268</td>\n",
" <td>-1.599651</td>\n",
" <td>-1.036633</td>\n",
" <td>1.665229</td>\n",
" <td>-0.067446</td>\n",
" <td>1.100297</td>\n",
" <td>-0.316657</td>\n",
" <td>-0.748823</td>\n",
" <td>-1.401503</td>\n",
" <td>0.561532</td>\n",
" </tr>\n",
" <tr>\n",
" <th>71</th>\n",
" <td>-0.850873</td>\n",
" <td>-0.903579</td>\n",
" <td>1.431794</td>\n",
" <td>-0.028323</td>\n",
" <td>1.481066</td>\n",
" <td>-0.906899</td>\n",
" <td>1.372466</td>\n",
" <td>-0.408602</td>\n",
" <td>1.639764</td>\n",
" <td>1.169778</td>\n",
" <td>...</td>\n",
" <td>1.226428</td>\n",
" <td>1.316993</td>\n",
" <td>0.943541</td>\n",
" <td>0.488501</td>\n",
" <td>-0.272406</td>\n",
" <td>-1.459338</td>\n",
" <td>1.011081</td>\n",
" <td>-1.737450</td>\n",
" <td>1.757685</td>\n",
" <td>0.941178</td>\n",
" </tr>\n",
" <tr>\n",
" <th>72</th>\n",
" <td>0.455261</td>\n",
" <td>-1.356238</td>\n",
" <td>-1.665984</td>\n",
" <td>-1.043824</td>\n",
" <td>0.780949</td>\n",
" <td>-1.618006</td>\n",
" <td>0.502182</td>\n",
" <td>-0.488663</td>\n",
" <td>-0.570120</td>\n",
" <td>0.715019</td>\n",
" <td>...</td>\n",
" <td>0.338761</td>\n",
" <td>-1.490060</td>\n",
" <td>0.993442</td>\n",
" <td>0.918406</td>\n",
" <td>0.071260</td>\n",
" <td>-1.580971</td>\n",
" <td>0.975308</td>\n",
" <td>0.125080</td>\n",
" <td>-1.371668</td>\n",
" <td>-0.822416</td>\n",
" </tr>\n",
" <tr>\n",
" <th>73</th>\n",
" <td>1.398525</td>\n",
" <td>-0.630447</td>\n",
" <td>0.869634</td>\n",
" <td>1.186189</td>\n",
" <td>-0.693715</td>\n",
" <td>-0.056440</td>\n",
" <td>0.763450</td>\n",
" <td>0.636438</td>\n",
" <td>-0.362792</td>\n",
" <td>-0.817460</td>\n",
" <td>...</td>\n",
" <td>-1.068658</td>\n",
" <td>1.172448</td>\n",
" <td>-1.668875</td>\n",
" <td>-0.478816</td>\n",
" <td>0.243758</td>\n",
" <td>-0.956590</td>\n",
" <td>-0.921063</td>\n",
" <td>0.527734</td>\n",
" <td>-0.551796</td>\n",
" <td>0.041946</td>\n",
" </tr>\n",
" <tr>\n",
" <th>74</th>\n",
" <td>1.195139</td>\n",
" <td>-0.649233</td>\n",
" <td>-1.749415</td>\n",
" <td>0.481350</td>\n",
" <td>-0.879385</td>\n",
" <td>0.861043</td>\n",
" <td>0.543140</td>\n",
" <td>-1.344145</td>\n",
" <td>1.061763</td>\n",
" <td>0.812840</td>\n",
" <td>...</td>\n",
" <td>-0.217410</td>\n",
" <td>-0.466573</td>\n",
" <td>0.166679</td>\n",
" <td>0.379972</td>\n",
" <td>-1.893531</td>\n",
" <td>-1.453595</td>\n",
" <td>0.445461</td>\n",
" <td>-1.830092</td>\n",
" <td>-1.130029</td>\n",
" <td>-1.471436</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75</th>\n",
" <td>1.219581</td>\n",
" <td>-1.355438</td>\n",
" <td>-0.143437</td>\n",
" <td>1.374460</td>\n",
" <td>0.535197</td>\n",
" <td>0.517635</td>\n",
" <td>1.051577</td>\n",
" <td>-0.687447</td>\n",
" <td>1.252113</td>\n",
" <td>0.701283</td>\n",
" <td>...</td>\n",
" <td>-1.661371</td>\n",
" <td>-1.249170</td>\n",
" <td>-1.097037</td>\n",
" <td>-1.135348</td>\n",
" <td>-1.109167</td>\n",
" <td>0.297080</td>\n",
" <td>1.468701</td>\n",
" <td>0.526295</td>\n",
" <td>0.359638</td>\n",
" <td>-0.103874</td>\n",
" </tr>\n",
" <tr>\n",
" <th>76</th>\n",
" <td>-0.772830</td>\n",
" <td>-0.029715</td>\n",
" <td>0.074375</td>\n",
" <td>-0.490430</td>\n",
" <td>1.480858</td>\n",
" <td>0.587917</td>\n",
" <td>-1.002865</td>\n",
" <td>0.425643</td>\n",
" <td>-0.961998</td>\n",
" <td>-0.647699</td>\n",
" <td>...</td>\n",
" <td>1.679556</td>\n",
" <td>-1.541915</td>\n",
" <td>0.857821</td>\n",
" <td>-0.828734</td>\n",
" <td>0.806509</td>\n",
" <td>-1.216620</td>\n",
" <td>0.150009</td>\n",
" <td>-0.438745</td>\n",
" <td>1.178379</td>\n",
" <td>-1.657935</td>\n",
" </tr>\n",
" <tr>\n",
" <th>77</th>\n",
" <td>-0.970610</td>\n",
" <td>1.621268</td>\n",
" <td>0.976901</td>\n",
" <td>-1.000428</td>\n",
" <td>0.486647</td>\n",
" <td>-1.027562</td>\n",
" <td>-0.227339</td>\n",
" <td>-0.018820</td>\n",
" <td>-0.136400</td>\n",
" <td>-0.170005</td>\n",
" <td>...</td>\n",
" <td>0.491278</td>\n",
" <td>0.778125</td>\n",
" <td>0.015861</td>\n",
" <td>0.396135</td>\n",
" <td>-0.920930</td>\n",
" <td>-0.695792</td>\n",
" <td>-0.312448</td>\n",
" <td>-0.247626</td>\n",
" <td>1.490141</td>\n",
" <td>-0.907475</td>\n",
" </tr>\n",
" <tr>\n",
" <th>78</th>\n",
" <td>-1.565164</td>\n",
" <td>1.675003</td>\n",
" <td>0.266903</td>\n",
" <td>-1.066366</td>\n",
" <td>-0.482731</td>\n",
" <td>1.611849</td>\n",
" <td>-1.662038</td>\n",
" <td>-0.139250</td>\n",
" <td>-1.121025</td>\n",
" <td>-1.244859</td>\n",
" <td>...</td>\n",
" <td>-1.004240</td>\n",
" <td>-0.931624</td>\n",
" <td>0.501343</td>\n",
" <td>-0.268321</td>\n",
" <td>0.382614</td>\n",
" <td>-1.530718</td>\n",
" <td>0.115311</td>\n",
" <td>1.254945</td>\n",
" <td>0.306358</td>\n",
" <td>-0.465282</td>\n",
" </tr>\n",
" <tr>\n",
" <th>79</th>\n",
" <td>-1.588639</td>\n",
" <td>-0.972567</td>\n",
" <td>-0.409528</td>\n",
" <td>-0.931689</td>\n",
" <td>-1.281164</td>\n",
" <td>0.806443</td>\n",
" <td>-1.223291</td>\n",
" <td>0.080806</td>\n",
" <td>-1.573407</td>\n",
" <td>1.335915</td>\n",
" <td>...</td>\n",
" <td>0.917508</td>\n",
" <td>-0.523602</td>\n",
" <td>0.950502</td>\n",
" <td>1.499240</td>\n",
" <td>-1.719770</td>\n",
" <td>1.244792</td>\n",
" <td>0.629544</td>\n",
" <td>-1.428163</td>\n",
" <td>0.994619</td>\n",
" <td>-0.018197</td>\n",
" </tr>\n",
" <tr>\n",
" <th>80</th>\n",
" <td>0.040967</td>\n",
" <td>1.623519</td>\n",
" <td>-0.717045</td>\n",
" <td>-1.237592</td>\n",
" <td>-0.636891</td>\n",
" <td>0.066840</td>\n",
" <td>-0.408493</td>\n",
" <td>-0.823917</td>\n",
" <td>-1.419024</td>\n",
" <td>1.440702</td>\n",
" <td>...</td>\n",
" <td>1.614721</td>\n",
" <td>0.755044</td>\n",
" <td>-1.705512</td>\n",
" <td>1.534536</td>\n",
" <td>1.396306</td>\n",
" <td>-1.434376</td>\n",
" <td>-1.635364</td>\n",
" <td>-1.417336</td>\n",
" <td>0.186586</td>\n",
" <td>-1.529835</td>\n",
" </tr>\n",
" <tr>\n",
" <th>81</th>\n",
" <td>1.471870</td>\n",
" <td>0.065310</td>\n",
" <td>0.981485</td>\n",
" <td>1.151370</td>\n",
" <td>-1.252160</td>\n",
" <td>-0.416664</td>\n",
" <td>-0.109791</td>\n",
" <td>-0.216107</td>\n",
" <td>0.083090</td>\n",
" <td>-0.307073</td>\n",
" <td>...</td>\n",
" <td>-1.734761</td>\n",
" <td>-1.161675</td>\n",
" <td>-0.098638</td>\n",
" <td>1.035213</td>\n",
" <td>1.393117</td>\n",
" <td>0.589706</td>\n",
" <td>0.940245</td>\n",
" <td>-0.886534</td>\n",
" <td>-0.368786</td>\n",
" <td>0.087320</td>\n",
" </tr>\n",
" <tr>\n",
" <th>82</th>\n",
" <td>0.443744</td>\n",
" <td>1.315541</td>\n",
" <td>-0.305109</td>\n",
" <td>0.001978</td>\n",
" <td>-0.079711</td>\n",
" <td>-0.171958</td>\n",
" <td>1.419704</td>\n",
" <td>1.473038</td>\n",
" <td>1.729698</td>\n",
" <td>0.986627</td>\n",
" <td>...</td>\n",
" <td>0.287173</td>\n",
" <td>1.571421</td>\n",
" <td>-0.454864</td>\n",
" <td>0.867487</td>\n",
" <td>-0.977401</td>\n",
" <td>0.555789</td>\n",
" <td>0.076087</td>\n",
" <td>0.173998</td>\n",
" <td>0.091385</td>\n",
" <td>0.482194</td>\n",
" </tr>\n",
" <tr>\n",
" <th>83</th>\n",
" <td>0.029843</td>\n",
" <td>0.767682</td>\n",
" <td>-0.993877</td>\n",
" <td>1.717344</td>\n",
" <td>-0.995321</td>\n",
" <td>-0.941138</td>\n",
" <td>-0.989065</td>\n",
" <td>1.184264</td>\n",
" <td>1.613218</td>\n",
" <td>1.674028</td>\n",
" <td>...</td>\n",
" <td>1.575743</td>\n",
" <td>-0.263912</td>\n",
" <td>0.753665</td>\n",
" <td>1.514962</td>\n",
" <td>0.051583</td>\n",
" <td>0.512572</td>\n",
" <td>-1.121129</td>\n",
" <td>-1.858649</td>\n",
" <td>1.716891</td>\n",
" <td>-1.288875</td>\n",
" </tr>\n",
" <tr>\n",
" <th>84</th>\n",
" <td>0.943205</td>\n",
" <td>-0.062812</td>\n",
" <td>0.137365</td>\n",
" <td>1.673334</td>\n",
" <td>1.457726</td>\n",
" <td>1.421604</td>\n",
" <td>-0.296355</td>\n",
" <td>1.440719</td>\n",
" <td>-0.053801</td>\n",
" <td>-1.800137</td>\n",
" <td>...</td>\n",
" <td>-1.788354</td>\n",
" <td>1.578393</td>\n",
" <td>0.091042</td>\n",
" <td>-0.914965</td>\n",
" <td>0.418812</td>\n",
" <td>-0.225515</td>\n",
" <td>1.083822</td>\n",
" <td>-1.846586</td>\n",
" <td>-0.881830</td>\n",
" <td>1.241672</td>\n",
" </tr>\n",
" <tr>\n",
" <th>85</th>\n",
" <td>1.143780</td>\n",
" <td>-1.227919</td>\n",
" <td>-0.038486</td>\n",
" <td>-0.991480</td>\n",
" <td>0.334727</td>\n",
" <td>-0.003853</td>\n",
" <td>0.906715</td>\n",
" <td>-1.280948</td>\n",
" <td>-0.299935</td>\n",
" <td>-0.922888</td>\n",
" <td>...</td>\n",
" <td>1.605559</td>\n",
" <td>-0.014674</td>\n",
" <td>-1.172934</td>\n",
" <td>-1.618927</td>\n",
" <td>0.367595</td>\n",
" <td>0.110084</td>\n",
" <td>0.603344</td>\n",
" <td>-0.004074</td>\n",
" <td>-0.474540</td>\n",
" <td>-0.316894</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86</th>\n",
" <td>-0.894993</td>\n",
" <td>-0.105199</td>\n",
" <td>-1.185358</td>\n",
" <td>-0.835168</td>\n",
" <td>0.180074</td>\n",
" <td>-0.075097</td>\n",
" <td>0.988637</td>\n",
" <td>0.710944</td>\n",
" <td>0.236767</td>\n",
" <td>-0.706053</td>\n",
" <td>...</td>\n",
" <td>-0.825000</td>\n",
" <td>0.699982</td>\n",
" <td>1.274639</td>\n",
" <td>0.371174</td>\n",
" <td>0.052800</td>\n",
" <td>-1.530699</td>\n",
" <td>1.676907</td>\n",
" <td>-0.063711</td>\n",
" <td>0.262439</td>\n",
" <td>-0.854201</td>\n",
" </tr>\n",
" <tr>\n",
" <th>87</th>\n",
" <td>-0.719007</td>\n",
" <td>-0.071196</td>\n",
" <td>1.467955</td>\n",
" <td>0.876265</td>\n",
" <td>0.449004</td>\n",
" <td>0.403144</td>\n",
" <td>-0.545273</td>\n",
" <td>-1.497419</td>\n",
" <td>0.844362</td>\n",
" <td>0.976808</td>\n",
" <td>...</td>\n",
" <td>0.433169</td>\n",
" <td>-0.853505</td>\n",
" <td>-1.251983</td>\n",
" <td>-0.917957</td>\n",
" <td>1.104403</td>\n",
" <td>-1.157357</td>\n",
" <td>0.643374</td>\n",
" <td>0.211782</td>\n",
" <td>-0.543510</td>\n",
" <td>-0.295384</td>\n",
" </tr>\n",
" <tr>\n",
" <th>88</th>\n",
" <td>-0.901002</td>\n",
" <td>-1.211165</td>\n",
" <td>-0.040284</td>\n",
" <td>-1.324867</td>\n",
" <td>0.388089</td>\n",
" <td>-1.272009</td>\n",
" <td>-1.084499</td>\n",
" <td>1.501458</td>\n",
" <td>0.328346</td>\n",
" <td>-0.762704</td>\n",
" <td>...</td>\n",
" <td>-0.708126</td>\n",
" <td>-0.261010</td>\n",
" <td>1.474982</td>\n",
" <td>-1.051313</td>\n",
" <td>1.270748</td>\n",
" <td>1.439772</td>\n",
" <td>1.314050</td>\n",
" <td>1.017364</td>\n",
" <td>-0.429077</td>\n",
" <td>-1.597541</td>\n",
" </tr>\n",
" <tr>\n",
" <th>89</th>\n",
" <td>-1.581999</td>\n",
" <td>0.329491</td>\n",
" <td>-1.593734</td>\n",
" <td>-1.164570</td>\n",
" <td>-0.062949</td>\n",
" <td>0.090999</td>\n",
" <td>1.343263</td>\n",
" <td>0.140127</td>\n",
" <td>1.047308</td>\n",
" <td>-1.889868</td>\n",
" <td>...</td>\n",
" <td>-0.049631</td>\n",
" <td>0.298881</td>\n",
" <td>1.320489</td>\n",
" <td>-0.547441</td>\n",
" <td>-0.743548</td>\n",
" <td>-0.616080</td>\n",
" <td>0.404941</td>\n",
" <td>-1.630911</td>\n",
" <td>0.959541</td>\n",
" <td>0.794803</td>\n",
" </tr>\n",
" <tr>\n",
" <th>90</th>\n",
" <td>1.384090</td>\n",
" <td>-0.844580</td>\n",
" <td>-0.743064</td>\n",
" <td>-0.630105</td>\n",
" <td>-1.202180</td>\n",
" <td>-0.535705</td>\n",
" <td>-1.129373</td>\n",
" <td>0.385504</td>\n",
" <td>1.286978</td>\n",
" <td>-1.661751</td>\n",
" <td>...</td>\n",
" <td>-0.187767</td>\n",
" <td>0.542780</td>\n",
" <td>0.900464</td>\n",
" <td>-1.498064</td>\n",
" <td>1.308647</td>\n",
" <td>-0.556545</td>\n",
" <td>0.042047</td>\n",
" <td>-0.086129</td>\n",
" <td>0.397801</td>\n",
" <td>-0.990178</td>\n",
" </tr>\n",
" <tr>\n",
" <th>91</th>\n",
" <td>-0.861300</td>\n",
" <td>1.182918</td>\n",
" <td>-0.226505</td>\n",
" <td>-0.187358</td>\n",
" <td>-0.301630</td>\n",
" <td>0.584219</td>\n",
" <td>1.485748</td>\n",
" <td>1.264118</td>\n",
" <td>-1.216566</td>\n",
" <td>0.193704</td>\n",
" <td>...</td>\n",
" <td>0.048404</td>\n",
" <td>-1.545104</td>\n",
" <td>1.450316</td>\n",
" <td>-0.095114</td>\n",
" <td>1.409027</td>\n",
" <td>0.779509</td>\n",
" <td>0.218895</td>\n",
" <td>0.658453</td>\n",
" <td>0.122641</td>\n",
" <td>-1.373730</td>\n",
" </tr>\n",
" <tr>\n",
" <th>92</th>\n",
" <td>-0.824783</td>\n",
" <td>-1.077926</td>\n",
" <td>0.891118</td>\n",
" <td>0.517056</td>\n",
" <td>-0.194426</td>\n",
" <td>1.435527</td>\n",
" <td>-0.239051</td>\n",
" <td>1.204818</td>\n",
" <td>-1.407671</td>\n",
" <td>-0.787198</td>\n",
" <td>...</td>\n",
" <td>0.444560</td>\n",
" <td>-0.254857</td>\n",
" <td>-0.679714</td>\n",
" <td>-1.054029</td>\n",
" <td>0.926229</td>\n",
" <td>1.437916</td>\n",
" <td>0.708352</td>\n",
" <td>-1.236232</td>\n",
" <td>-1.261535</td>\n",
" <td>-0.488064</td>\n",
" </tr>\n",
" <tr>\n",
" <th>93</th>\n",
" <td>0.785421</td>\n",
" <td>0.333046</td>\n",
" <td>0.050298</td>\n",
" <td>-1.126874</td>\n",
" <td>1.115591</td>\n",
" <td>1.287045</td>\n",
" <td>-0.095741</td>\n",
" <td>-0.176628</td>\n",
" <td>-1.383699</td>\n",
" <td>-1.748018</td>\n",
" <td>...</td>\n",
" <td>0.436805</td>\n",
" <td>-0.178552</td>\n",
" <td>-1.408312</td>\n",
" <td>-1.546488</td>\n",
" <td>1.305025</td>\n",
" <td>1.097852</td>\n",
" <td>0.815681</td>\n",
" <td>-1.185418</td>\n",
" <td>0.929243</td>\n",
" <td>0.570606</td>\n",
" </tr>\n",
" <tr>\n",
" <th>94</th>\n",
" <td>-1.189086</td>\n",
" <td>1.398801</td>\n",
" <td>-0.835365</td>\n",
" <td>-1.377584</td>\n",
" <td>0.708942</td>\n",
" <td>-0.990116</td>\n",
" <td>0.848418</td>\n",
" <td>0.566546</td>\n",
" <td>1.715034</td>\n",
" <td>-0.687138</td>\n",
" <td>...</td>\n",
" <td>1.412452</td>\n",
" <td>-0.520480</td>\n",
" <td>1.118536</td>\n",
" <td>0.857002</td>\n",
" <td>0.768750</td>\n",
" <td>0.795685</td>\n",
" <td>0.859676</td>\n",
" <td>-0.570313</td>\n",
" <td>0.916351</td>\n",
" <td>0.801122</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>-0.837958</td>\n",
" <td>-1.365456</td>\n",
" <td>0.184541</td>\n",
" <td>0.325820</td>\n",
" <td>0.127239</td>\n",
" <td>1.486810</td>\n",
" <td>0.300611</td>\n",
" <td>0.263508</td>\n",
" <td>-0.638503</td>\n",
" <td>1.576994</td>\n",
" <td>...</td>\n",
" <td>-0.991597</td>\n",
" <td>0.176691</td>\n",
" <td>0.532893</td>\n",
" <td>1.618070</td>\n",
" <td>0.765631</td>\n",
" <td>0.161272</td>\n",
" <td>0.437025</td>\n",
" <td>0.783952</td>\n",
" <td>-0.232400</td>\n",
" <td>-1.849131</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>-0.228477</td>\n",
" <td>-1.329432</td>\n",
" <td>1.195103</td>\n",
" <td>-0.498452</td>\n",
" <td>1.203354</td>\n",
" <td>1.253191</td>\n",
" <td>-1.484169</td>\n",
" <td>-0.672890</td>\n",
" <td>-1.012603</td>\n",
" <td>0.135583</td>\n",
" <td>...</td>\n",
" <td>-0.538102</td>\n",
" <td>-1.650884</td>\n",
" <td>-0.970296</td>\n",
" <td>-0.883116</td>\n",
" <td>-0.862598</td>\n",
" <td>1.180413</td>\n",
" <td>-0.473918</td>\n",
" <td>1.522473</td>\n",
" <td>1.424520</td>\n",
" <td>0.284405</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>-1.124243</td>\n",
" <td>-0.882249</td>\n",
" <td>-1.272757</td>\n",
" <td>0.278280</td>\n",
" <td>-0.231825</td>\n",
" <td>0.013484</td>\n",
" <td>0.751978</td>\n",
" <td>-0.734291</td>\n",
" <td>0.589624</td>\n",
" <td>0.708419</td>\n",
" <td>...</td>\n",
" <td>-0.997549</td>\n",
" <td>-1.623525</td>\n",
" <td>-0.069153</td>\n",
" <td>-1.075970</td>\n",
" <td>0.138066</td>\n",
" <td>1.117055</td>\n",
" <td>-0.235805</td>\n",
" <td>-1.164599</td>\n",
" <td>0.428549</td>\n",
" <td>1.483252</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>0.197683</td>\n",
" <td>0.271390</td>\n",
" <td>-0.438227</td>\n",
" <td>-1.475379</td>\n",
" <td>0.154414</td>\n",
" <td>-1.211315</td>\n",
" <td>0.887269</td>\n",
" <td>-0.991987</td>\n",
" <td>1.370059</td>\n",
" <td>1.412520</td>\n",
" <td>...</td>\n",
" <td>0.643335</td>\n",
" <td>-0.854826</td>\n",
" <td>-0.558995</td>\n",
" <td>0.906758</td>\n",
" <td>-0.873148</td>\n",
" <td>1.879480</td>\n",
" <td>-0.615526</td>\n",
" <td>0.301785</td>\n",
" <td>0.414691</td>\n",
" <td>0.160866</td>\n",
" </tr>\n",
" <tr>\n",
" <th>99</th>\n",
" <td>0.974161</td>\n",
" <td>-1.046334</td>\n",
" <td>-1.358202</td>\n",
" <td>-0.644732</td>\n",
" <td>-0.289628</td>\n",
" <td>-1.222582</td>\n",
" <td>0.960976</td>\n",
" <td>1.144446</td>\n",
" <td>0.867853</td>\n",
" <td>0.491895</td>\n",
" <td>...</td>\n",
" <td>1.205216</td>\n",
" <td>0.752640</td>\n",
" <td>-1.621746</td>\n",
" <td>0.842248</td>\n",
" <td>-0.701166</td>\n",
" <td>0.881687</td>\n",
" <td>-0.847757</td>\n",
" <td>0.553707</td>\n",
" <td>1.235270</td>\n",
" <td>0.379624</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100 rows × 100 columns</p>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3 4 5 6 \\\n",
"0 0.513870 1.396583 -0.246373 0.412241 1.579367 0.363903 -1.323891 \n",
"1 1.631868 -1.075335 0.170484 1.453664 1.357602 -1.597306 1.342091 \n",
"2 -0.514866 0.374350 -1.009489 0.086802 -1.696162 0.194635 1.231866 \n",
"3 0.886998 0.052952 1.716351 -0.169000 -1.037321 -1.554467 1.375881 \n",
"4 1.391114 0.418370 -1.037966 0.312837 -0.964962 -1.330757 0.641659 \n",
"5 -0.978640 0.882280 1.098095 1.637879 1.623599 1.164122 0.459580 \n",
"6 0.735374 1.603069 -0.607167 1.751502 1.457694 -1.035909 0.547590 \n",
"7 -1.348100 -0.399150 -1.218192 1.062848 0.749450 -1.029018 0.392363 \n",
"8 1.285660 0.526640 -0.327848 -0.794804 0.136058 0.355371 1.387508 \n",
"9 1.564215 -0.530578 -1.378221 1.484833 -0.121855 -1.147937 -0.646704 \n",
"10 -1.512300 1.257621 -0.950904 0.097102 0.066319 1.201285 -1.479766 \n",
"11 0.676772 -0.382604 0.152292 1.184203 0.885681 -1.561482 0.544845 \n",
"12 0.240678 -1.393210 0.187365 -0.798240 0.226787 -0.633688 -0.989422 \n",
"13 0.340047 0.938168 -1.118608 -0.523279 -0.761692 0.586611 0.016273 \n",
"14 -0.216206 -0.081457 -0.066678 0.712012 -0.933661 0.107191 1.538183 \n",
"15 -0.393874 -0.217956 0.505092 1.288478 0.743926 1.529977 -1.468972 \n",
"16 1.448163 -1.468940 0.296056 -0.724831 0.647826 -0.180198 -1.700891 \n",
"17 0.909765 1.254015 1.629871 -0.727945 0.364806 -1.244102 -0.660909 \n",
"18 -0.734810 -0.748138 1.446927 0.207259 0.760452 1.116326 -0.842161 \n",
"19 0.190232 -0.380382 -0.956613 0.161797 0.358565 1.054044 -0.502211 \n",
"20 0.171497 -0.568547 -0.181602 1.280796 -0.607141 -1.314850 0.877709 \n",
"21 -1.773808 -0.806683 1.710190 1.529340 0.149329 0.015717 -1.664649 \n",
"22 -0.232519 -0.199589 0.034674 -0.284041 0.924887 1.394155 0.390990 \n",
"23 -1.171088 0.141293 1.626429 0.168721 -1.727745 1.386053 0.263236 \n",
"24 -1.583545 0.526075 -0.406596 -1.025639 -1.104152 -1.492721 0.094901 \n",
"25 -0.247518 -0.116995 0.005844 1.353575 1.295124 -0.134358 1.053640 \n",
"26 1.587095 0.230919 0.362958 -1.250820 1.657170 -0.718975 0.330160 \n",
"27 0.935906 -0.145264 -1.250488 -1.470936 1.506688 -1.313323 1.518933 \n",
"28 0.719676 -0.814732 -1.725769 -1.235741 -0.586766 0.796515 -1.080151 \n",
"29 -0.209103 1.559806 -0.736238 0.930045 -1.183643 -1.525615 0.653475 \n",
".. ... ... ... ... ... ... ... \n",
"70 -0.374306 -0.509877 1.139351 0.463223 -0.122952 0.257494 0.984939 \n",
"71 -0.850873 -0.903579 1.431794 -0.028323 1.481066 -0.906899 1.372466 \n",
"72 0.455261 -1.356238 -1.665984 -1.043824 0.780949 -1.618006 0.502182 \n",
"73 1.398525 -0.630447 0.869634 1.186189 -0.693715 -0.056440 0.763450 \n",
"74 1.195139 -0.649233 -1.749415 0.481350 -0.879385 0.861043 0.543140 \n",
"75 1.219581 -1.355438 -0.143437 1.374460 0.535197 0.517635 1.051577 \n",
"76 -0.772830 -0.029715 0.074375 -0.490430 1.480858 0.587917 -1.002865 \n",
"77 -0.970610 1.621268 0.976901 -1.000428 0.486647 -1.027562 -0.227339 \n",
"78 -1.565164 1.675003 0.266903 -1.066366 -0.482731 1.611849 -1.662038 \n",
"79 -1.588639 -0.972567 -0.409528 -0.931689 -1.281164 0.806443 -1.223291 \n",
"80 0.040967 1.623519 -0.717045 -1.237592 -0.636891 0.066840 -0.408493 \n",
"81 1.471870 0.065310 0.981485 1.151370 -1.252160 -0.416664 -0.109791 \n",
"82 0.443744 1.315541 -0.305109 0.001978 -0.079711 -0.171958 1.419704 \n",
"83 0.029843 0.767682 -0.993877 1.717344 -0.995321 -0.941138 -0.989065 \n",
"84 0.943205 -0.062812 0.137365 1.673334 1.457726 1.421604 -0.296355 \n",
"85 1.143780 -1.227919 -0.038486 -0.991480 0.334727 -0.003853 0.906715 \n",
"86 -0.894993 -0.105199 -1.185358 -0.835168 0.180074 -0.075097 0.988637 \n",
"87 -0.719007 -0.071196 1.467955 0.876265 0.449004 0.403144 -0.545273 \n",
"88 -0.901002 -1.211165 -0.040284 -1.324867 0.388089 -1.272009 -1.084499 \n",
"89 -1.581999 0.329491 -1.593734 -1.164570 -0.062949 0.090999 1.343263 \n",
"90 1.384090 -0.844580 -0.743064 -0.630105 -1.202180 -0.535705 -1.129373 \n",
"91 -0.861300 1.182918 -0.226505 -0.187358 -0.301630 0.584219 1.485748 \n",
"92 -0.824783 -1.077926 0.891118 0.517056 -0.194426 1.435527 -0.239051 \n",
"93 0.785421 0.333046 0.050298 -1.126874 1.115591 1.287045 -0.095741 \n",
"94 -1.189086 1.398801 -0.835365 -1.377584 0.708942 -0.990116 0.848418 \n",
"95 -0.837958 -1.365456 0.184541 0.325820 0.127239 1.486810 0.300611 \n",
"96 -0.228477 -1.329432 1.195103 -0.498452 1.203354 1.253191 -1.484169 \n",
"97 -1.124243 -0.882249 -1.272757 0.278280 -0.231825 0.013484 0.751978 \n",
"98 0.197683 0.271390 -0.438227 -1.475379 0.154414 -1.211315 0.887269 \n",
"99 0.974161 -1.046334 -1.358202 -0.644732 -0.289628 -1.222582 0.960976 \n",
"\n",
" 7 8 9 ... 90 91 92 \\\n",
"0 -1.113526 -0.877806 0.881624 ... 1.061778 1.647589 0.608297 \n",
"1 -1.775401 1.129466 -1.180717 ... -1.806966 -0.856221 0.223695 \n",
"2 1.121161 0.736265 -0.563802 ... -1.273403 -1.178026 1.542241 \n",
"3 1.256623 -0.206046 -0.299600 ... 0.967154 1.519549 1.556544 \n",
"4 1.299796 0.038224 0.356430 ... -0.912055 0.942265 -0.054322 \n",
"5 -1.200620 -0.101569 -1.025493 ... -0.178119 0.153881 -1.489814 \n",
"6 0.782969 1.545038 1.538226 ... 0.969993 -0.325719 0.994623 \n",
"7 0.421657 0.277607 1.341795 ... 0.564476 -1.216358 1.071405 \n",
"8 -0.973951 -1.477451 1.033896 ... 0.944201 -0.329253 0.670113 \n",
"9 1.363795 -1.422009 -0.908234 ... -1.839381 0.365275 1.490728 \n",
"10 1.081254 -0.700119 0.551702 ... -0.511107 -0.801593 -0.291530 \n",
"11 0.825152 0.851938 0.386944 ... 0.706691 -1.175663 -0.441140 \n",
"12 -1.293567 1.050305 0.351630 ... 1.583936 0.168961 -0.907365 \n",
"13 -0.059960 -0.971746 0.203671 ... -1.582610 -1.445128 0.929069 \n",
"14 1.312979 1.597155 0.204361 ... 0.851806 1.574912 -1.057158 \n",
"15 0.813537 -1.114886 -0.573397 ... -0.397053 0.327747 0.515145 \n",
"16 0.269400 0.671001 0.743764 ... -1.011690 0.961830 -0.257341 \n",
"17 -1.646826 1.657767 1.276033 ... -0.281521 1.723603 -1.584701 \n",
"18 -0.789068 0.532212 0.759320 ... 1.465185 0.401349 -0.966394 \n",
"19 1.479995 0.543587 -0.610027 ... 0.905991 0.590617 0.979145 \n",
"20 -1.303426 -0.263439 -0.578028 ... 0.463058 -0.343679 -1.369579 \n",
"21 -0.333638 0.959662 -1.626720 ... -0.076300 0.181243 -0.343203 \n",
"22 0.467035 -0.711218 -0.232838 ... 0.516714 0.670454 1.216928 \n",
"23 1.137646 0.360819 1.443515 ... 0.862208 -1.184068 1.456381 \n",
"24 -1.687825 0.522421 -0.947742 ... -0.274502 -1.410834 -0.511294 \n",
"25 -1.656545 -0.071846 0.962144 ... -1.318647 1.559977 -0.074145 \n",
"26 -0.126438 -1.220022 -1.125991 ... -0.649107 1.306017 0.482203 \n",
"27 0.610692 -0.491017 -1.627820 ... 0.902349 1.508875 -1.127943 \n",
"28 0.608303 0.782711 -0.265900 ... 1.390754 -1.492590 -0.500209 \n",
"29 -1.798585 -1.294609 0.813010 ... -1.518635 0.192210 -0.855648 \n",
".. ... ... ... ... ... ... ... \n",
"70 0.808924 -0.961834 -0.648294 ... 0.403268 -1.599651 -1.036633 \n",
"71 -0.408602 1.639764 1.169778 ... 1.226428 1.316993 0.943541 \n",
"72 -0.488663 -0.570120 0.715019 ... 0.338761 -1.490060 0.993442 \n",
"73 0.636438 -0.362792 -0.817460 ... -1.068658 1.172448 -1.668875 \n",
"74 -1.344145 1.061763 0.812840 ... -0.217410 -0.466573 0.166679 \n",
"75 -0.687447 1.252113 0.701283 ... -1.661371 -1.249170 -1.097037 \n",
"76 0.425643 -0.961998 -0.647699 ... 1.679556 -1.541915 0.857821 \n",
"77 -0.018820 -0.136400 -0.170005 ... 0.491278 0.778125 0.015861 \n",
"78 -0.139250 -1.121025 -1.244859 ... -1.004240 -0.931624 0.501343 \n",
"79 0.080806 -1.573407 1.335915 ... 0.917508 -0.523602 0.950502 \n",
"80 -0.823917 -1.419024 1.440702 ... 1.614721 0.755044 -1.705512 \n",
"81 -0.216107 0.083090 -0.307073 ... -1.734761 -1.161675 -0.098638 \n",
"82 1.473038 1.729698 0.986627 ... 0.287173 1.571421 -0.454864 \n",
"83 1.184264 1.613218 1.674028 ... 1.575743 -0.263912 0.753665 \n",
"84 1.440719 -0.053801 -1.800137 ... -1.788354 1.578393 0.091042 \n",
"85 -1.280948 -0.299935 -0.922888 ... 1.605559 -0.014674 -1.172934 \n",
"86 0.710944 0.236767 -0.706053 ... -0.825000 0.699982 1.274639 \n",
"87 -1.497419 0.844362 0.976808 ... 0.433169 -0.853505 -1.251983 \n",
"88 1.501458 0.328346 -0.762704 ... -0.708126 -0.261010 1.474982 \n",
"89 0.140127 1.047308 -1.889868 ... -0.049631 0.298881 1.320489 \n",
"90 0.385504 1.286978 -1.661751 ... -0.187767 0.542780 0.900464 \n",
"91 1.264118 -1.216566 0.193704 ... 0.048404 -1.545104 1.450316 \n",
"92 1.204818 -1.407671 -0.787198 ... 0.444560 -0.254857 -0.679714 \n",
"93 -0.176628 -1.383699 -1.748018 ... 0.436805 -0.178552 -1.408312 \n",
"94 0.566546 1.715034 -0.687138 ... 1.412452 -0.520480 1.118536 \n",
"95 0.263508 -0.638503 1.576994 ... -0.991597 0.176691 0.532893 \n",
"96 -0.672890 -1.012603 0.135583 ... -0.538102 -1.650884 -0.970296 \n",
"97 -0.734291 0.589624 0.708419 ... -0.997549 -1.623525 -0.069153 \n",
"98 -0.991987 1.370059 1.412520 ... 0.643335 -0.854826 -0.558995 \n",
"99 1.144446 0.867853 0.491895 ... 1.205216 0.752640 -1.621746 \n",
"\n",
" 93 94 95 96 97 98 99 \n",
"0 0.931085 -0.305219 -0.679682 1.323216 -1.000440 1.158411 0.271465 \n",
"1 -0.972374 0.176362 0.792399 -0.528541 0.796424 0.728051 -1.104085 \n",
"2 1.006310 0.538693 0.789394 0.540420 0.742471 0.203087 -1.486384 \n",
"3 -0.068761 1.182859 -0.647793 -0.700245 0.705043 -0.825873 1.851311 \n",
"4 0.174229 -0.782228 -0.613781 -1.241414 0.528770 -1.546610 -0.655011 \n",
"5 0.010346 1.043027 1.248067 1.028140 -0.870393 -0.706648 1.744279 \n",
"6 -1.531772 1.115576 0.459712 0.949289 -0.787418 -0.295029 0.903614 \n",
"7 -0.202719 0.165891 -0.592823 0.805909 0.591497 -1.093653 -1.055484 \n",
"8 0.898907 -1.234780 -1.424890 1.544366 -0.845917 0.405283 0.693724 \n",
"9 -0.444500 0.555258 1.260674 -0.156228 -1.821950 1.436147 -1.629020 \n",
"10 -0.149648 -0.885823 -0.679197 -1.809390 1.392497 1.610417 0.636615 \n",
"11 -1.470221 -0.088202 -0.315637 0.481602 0.378751 -1.696612 -0.433806 \n",
"12 1.536351 0.692415 -0.558356 1.183726 0.025276 0.922382 -0.886471 \n",
"13 0.529699 -0.288482 0.100620 -0.339643 -1.342788 0.285311 -1.530031 \n",
"14 0.897952 0.334998 -1.499413 -0.557911 0.893911 -1.525342 -0.772764 \n",
"15 0.648666 -0.041444 0.037236 0.563206 1.500904 0.985118 1.295535 \n",
"16 -1.322110 -0.140934 1.127069 0.961686 1.278722 1.053308 0.611221 \n",
"17 -0.319174 -1.711071 -1.183856 1.509604 0.526637 -0.805211 -1.732134 \n",
"18 -0.362602 1.127971 0.752127 -0.106078 1.265856 0.211697 0.269893 \n",
"19 -1.668567 1.387817 -0.542651 -1.742281 -1.127449 -1.631693 0.071839 \n",
"20 1.682706 -0.492890 1.073619 1.642233 -1.790007 -0.242860 -0.465673 \n",
"21 1.127871 -1.857502 -0.760829 -1.716068 -0.238514 0.830074 1.099593 \n",
"22 1.133620 -1.013026 -1.129269 1.325263 1.292575 0.501611 1.526500 \n",
"23 0.892589 -0.674029 -1.409532 0.050445 -0.772777 1.630130 1.186272 \n",
"24 -1.479222 1.539949 -0.185147 -1.623003 1.281739 0.106110 -0.587262 \n",
"25 -0.378504 0.869430 -0.069882 -1.669445 -0.498185 -1.576615 -0.777916 \n",
"26 0.145802 1.448299 -0.520351 -1.056655 -0.805319 -1.254778 0.418933 \n",
"27 -0.570377 0.352445 -0.110769 -1.567606 -1.748038 -1.510443 0.548930 \n",
"28 -0.435704 0.911975 1.868319 -0.857274 1.058341 -0.547400 0.169497 \n",
"29 -1.609667 -0.225023 -0.247551 -0.314739 0.907275 -1.162187 0.812885 \n",
".. ... ... ... ... ... ... ... \n",
"70 1.665229 -0.067446 1.100297 -0.316657 -0.748823 -1.401503 0.561532 \n",
"71 0.488501 -0.272406 -1.459338 1.011081 -1.737450 1.757685 0.941178 \n",
"72 0.918406 0.071260 -1.580971 0.975308 0.125080 -1.371668 -0.822416 \n",
"73 -0.478816 0.243758 -0.956590 -0.921063 0.527734 -0.551796 0.041946 \n",
"74 0.379972 -1.893531 -1.453595 0.445461 -1.830092 -1.130029 -1.471436 \n",
"75 -1.135348 -1.109167 0.297080 1.468701 0.526295 0.359638 -0.103874 \n",
"76 -0.828734 0.806509 -1.216620 0.150009 -0.438745 1.178379 -1.657935 \n",
"77 0.396135 -0.920930 -0.695792 -0.312448 -0.247626 1.490141 -0.907475 \n",
"78 -0.268321 0.382614 -1.530718 0.115311 1.254945 0.306358 -0.465282 \n",
"79 1.499240 -1.719770 1.244792 0.629544 -1.428163 0.994619 -0.018197 \n",
"80 1.534536 1.396306 -1.434376 -1.635364 -1.417336 0.186586 -1.529835 \n",
"81 1.035213 1.393117 0.589706 0.940245 -0.886534 -0.368786 0.087320 \n",
"82 0.867487 -0.977401 0.555789 0.076087 0.173998 0.091385 0.482194 \n",
"83 1.514962 0.051583 0.512572 -1.121129 -1.858649 1.716891 -1.288875 \n",
"84 -0.914965 0.418812 -0.225515 1.083822 -1.846586 -0.881830 1.241672 \n",
"85 -1.618927 0.367595 0.110084 0.603344 -0.004074 -0.474540 -0.316894 \n",
"86 0.371174 0.052800 -1.530699 1.676907 -0.063711 0.262439 -0.854201 \n",
"87 -0.917957 1.104403 -1.157357 0.643374 0.211782 -0.543510 -0.295384 \n",
"88 -1.051313 1.270748 1.439772 1.314050 1.017364 -0.429077 -1.597541 \n",
"89 -0.547441 -0.743548 -0.616080 0.404941 -1.630911 0.959541 0.794803 \n",
"90 -1.498064 1.308647 -0.556545 0.042047 -0.086129 0.397801 -0.990178 \n",
"91 -0.095114 1.409027 0.779509 0.218895 0.658453 0.122641 -1.373730 \n",
"92 -1.054029 0.926229 1.437916 0.708352 -1.236232 -1.261535 -0.488064 \n",
"93 -1.546488 1.305025 1.097852 0.815681 -1.185418 0.929243 0.570606 \n",
"94 0.857002 0.768750 0.795685 0.859676 -0.570313 0.916351 0.801122 \n",
"95 1.618070 0.765631 0.161272 0.437025 0.783952 -0.232400 -1.849131 \n",
"96 -0.883116 -0.862598 1.180413 -0.473918 1.522473 1.424520 0.284405 \n",
"97 -1.075970 0.138066 1.117055 -0.235805 -1.164599 0.428549 1.483252 \n",
"98 0.906758 -0.873148 1.879480 -0.615526 0.301785 0.414691 0.160866 \n",
"99 0.842248 -0.701166 0.881687 -0.847757 0.553707 1.235270 0.379624 \n",
"\n",
"[100 rows x 100 columns]"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.DataFrame(np.random.uniform(0, 10, (100, 100)))\n",
"\n",
"df2 = (df - df.mean())/df.std()\n",
"df2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Loading and Storing <a name=\"loading\"></a>\n",
"Accessing data is a necessary first step for data science. In this section, the focus will be on data input and output in various formats using `pandas`\n",
"\n",
"Data usually fall into these categories:\n",
"- text files\n",
"- binary files (more efficient space-wise)\n",
"- web data\n",
"\n",
"## Text formats <a name=\"text\"></a>\n",
"The most common format in this category is by far `.csv`. This is an easy to read file format which is usually visualised like a spreadsheet. The data itself is usually separated with a `,` which is called the **delimiter**.\n",
"\n",
"Here is an example of a `.csv` file:\n",
"\n",
"```\n",
"Sell,List,Living,Rooms,Beds,Baths,Age,Acres,Taxes\n",
"142, 160, 28, 10, 5, 3, 60, 0.28, 3167\n",
"175, 180, 18, 8, 4, 1, 12, 0.43, 4033\n",
"129, 132, 13, 6, 3, 1, 41, 0.33, 1471\n",
"138, 140, 17, 7, 3, 1, 22, 0.46, 3204\n",
"232, 240, 25, 8, 4, 3, 5, 2.05, 3613\n",
"135, 140, 18, 7, 4, 3, 9, 0.57, 3028\n",
"150, 160, 20, 8, 4, 3, 18, 4.00, 3131\n",
"207, 225, 22, 8, 4, 2, 16, 2.22, 5158\n",
"271, 285, 30, 10, 5, 2, 30, 0.53, 5702\n",
" 89, 90, 10, 5, 3, 1, 43, 0.30, 2054\n",
" ```\n",
"\n",
"It detailed home sale statistics. The first line is called the header, and you can imagine that it is the name of the columns of a spreadsheet.\n",
"\n",
"Let's now see how we can load this data and analyse it. The file is located in the folder `data` and is called `homes.csv`. We can read it like this:"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"homes = pd.read_csv(\"data/homes.csv\")"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Sell</th>\n",
" <th>List</th>\n",
" <th>Living</th>\n",
" <th>Rooms</th>\n",
" <th>Beds</th>\n",
" <th>Baths</th>\n",
" <th>Age</th>\n",
" <th>Acres</th>\n",
" <th>Taxes</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>142</td>\n",
" <td>160</td>\n",
" <td>28</td>\n",
" <td>10</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>60</td>\n",
" <td>0.28</td>\n",
" <td>3167</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>175</td>\n",
" <td>180</td>\n",
" <td>18</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>12</td>\n",
" <td>0.43</td>\n",
" <td>4033</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>143</td>\n",
" <td>145</td>\n",
" <td>21</td>\n",
" <td>7</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>10</td>\n",
" <td>1.20</td>\n",
" <td>3529</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>129</td>\n",
" <td>132</td>\n",
" <td>13</td>\n",
" <td>6</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>41</td>\n",
" <td>0.33</td>\n",
" <td>1471</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>138</td>\n",
" <td>140</td>\n",
" <td>17</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>22</td>\n",
" <td>0.46</td>\n",
" <td>3204</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>232</td>\n",
" <td>240</td>\n",
" <td>25</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>5</td>\n",
" <td>2.05</td>\n",
" <td>3613</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>135</td>\n",
" <td>140</td>\n",
" <td>18</td>\n",
" <td>7</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>9</td>\n",
" <td>0.57</td>\n",
" <td>3028</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>150</td>\n",
" <td>160</td>\n",
" <td>20</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>18</td>\n",
" <td>4.00</td>\n",
" <td>3131</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>293</td>\n",
" <td>305</td>\n",
" <td>26</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>6</td>\n",
" <td>0.46</td>\n",
" <td>7088</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>207</td>\n",
" <td>225</td>\n",
" <td>22</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>16</td>\n",
" <td>2.22</td>\n",
" <td>5158</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>271</td>\n",
" <td>285</td>\n",
" <td>30</td>\n",
" <td>10</td>\n",
" <td>5</td>\n",
" <td>2</td>\n",
" <td>30</td>\n",
" <td>0.53</td>\n",
" <td>5702</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>89</td>\n",
" <td>90</td>\n",
" <td>10</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>43</td>\n",
" <td>0.30</td>\n",
" <td>2054</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>153</td>\n",
" <td>157</td>\n",
" <td>22</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" <td>18</td>\n",
" <td>0.38</td>\n",
" <td>4127</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>87</td>\n",
" <td>90</td>\n",
" <td>16</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>50</td>\n",
" <td>0.65</td>\n",
" <td>1445</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>234</td>\n",
" <td>238</td>\n",
" <td>25</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>1.61</td>\n",
" <td>2087</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>106</td>\n",
" <td>116</td>\n",
" <td>20</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>13</td>\n",
" <td>0.22</td>\n",
" <td>2818</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>175</td>\n",
" <td>180</td>\n",
" <td>22</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>15</td>\n",
" <td>2.06</td>\n",
" <td>3917</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>165</td>\n",
" <td>170</td>\n",
" <td>17</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>33</td>\n",
" <td>0.46</td>\n",
" <td>2220</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>166</td>\n",
" <td>170</td>\n",
" <td>23</td>\n",
" <td>9</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>37</td>\n",
" <td>0.27</td>\n",
" <td>3498</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>136</td>\n",
" <td>140</td>\n",
" <td>19</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>22</td>\n",
" <td>0.63</td>\n",
" <td>3607</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>148</td>\n",
" <td>160</td>\n",
" <td>17</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>2</td>\n",
" <td>13</td>\n",
" <td>0.36</td>\n",
" <td>3648</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>151</td>\n",
" <td>153</td>\n",
" <td>19</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>24</td>\n",
" <td>0.34</td>\n",
" <td>3561</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>180</td>\n",
" <td>190</td>\n",
" <td>24</td>\n",
" <td>9</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>10</td>\n",
" <td>1.55</td>\n",
" <td>4681</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>293</td>\n",
" <td>305</td>\n",
" <td>26</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>6</td>\n",
" <td>0.46</td>\n",
" <td>7088</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>167</td>\n",
" <td>170</td>\n",
" <td>20</td>\n",
" <td>9</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>46</td>\n",
" <td>0.46</td>\n",
" <td>3482</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>190</td>\n",
" <td>193</td>\n",
" <td>22</td>\n",
" <td>9</td>\n",
" <td>5</td>\n",
" <td>2</td>\n",
" <td>37</td>\n",
" <td>0.48</td>\n",
" <td>3920</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>184</td>\n",
" <td>190</td>\n",
" <td>21</td>\n",
" <td>9</td>\n",
" <td>5</td>\n",
" <td>2</td>\n",
" <td>27</td>\n",
" <td>1.30</td>\n",
" <td>4162</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>157</td>\n",
" <td>165</td>\n",
" <td>20</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>7</td>\n",
" <td>0.30</td>\n",
" <td>3785</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>110</td>\n",
" <td>115</td>\n",
" <td>16</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>26</td>\n",
" <td>0.29</td>\n",
" <td>3103</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>135</td>\n",
" <td>145</td>\n",
" <td>18</td>\n",
" <td>7</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>35</td>\n",
" <td>0.43</td>\n",
" <td>3363</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>567</td>\n",
" <td>625</td>\n",
" <td>64</td>\n",
" <td>11</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>0.85</td>\n",
" <td>12192</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>180</td>\n",
" <td>185</td>\n",
" <td>20</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>11</td>\n",
" <td>1.00</td>\n",
" <td>3831</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>183</td>\n",
" <td>188</td>\n",
" <td>17</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>2</td>\n",
" <td>16</td>\n",
" <td>3.00</td>\n",
" <td>3564</td>\n",
" </tr>\n",
" <tr>\n",
" <th>33</th>\n",
" <td>185</td>\n",
" <td>193</td>\n",
" <td>20</td>\n",
" <td>9</td>\n",
" <td>3</td>\n",
" <td>2</td>\n",
" <td>56</td>\n",
" <td>6.49</td>\n",
" <td>3765</td>\n",
" </tr>\n",
" <tr>\n",
" <th>34</th>\n",
" <td>152</td>\n",
" <td>155</td>\n",
" <td>17</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>33</td>\n",
" <td>0.70</td>\n",
" <td>3361</td>\n",
" </tr>\n",
" <tr>\n",
" <th>35</th>\n",
" <td>148</td>\n",
" <td>153</td>\n",
" <td>13</td>\n",
" <td>6</td>\n",
" <td>3</td>\n",
" <td>2</td>\n",
" <td>22</td>\n",
" <td>0.39</td>\n",
" <td>3950</td>\n",
" </tr>\n",
" <tr>\n",
" <th>36</th>\n",
" <td>152</td>\n",
" <td>159</td>\n",
" <td>15</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>25</td>\n",
" <td>0.59</td>\n",
" <td>3055</td>\n",
" </tr>\n",
" <tr>\n",
" <th>37</th>\n",
" <td>146</td>\n",
" <td>150</td>\n",
" <td>16</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>31</td>\n",
" <td>0.36</td>\n",
" <td>2950</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38</th>\n",
" <td>170</td>\n",
" <td>190</td>\n",
" <td>24</td>\n",
" <td>10</td>\n",
" <td>3</td>\n",
" <td>2</td>\n",
" <td>33</td>\n",
" <td>0.57</td>\n",
" <td>3346</td>\n",
" </tr>\n",
" <tr>\n",
" <th>39</th>\n",
" <td>106</td>\n",
" <td>116</td>\n",
" <td>20</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>13</td>\n",
" <td>0.22</td>\n",
" <td>2818</td>\n",
" </tr>\n",
" <tr>\n",
" <th>40</th>\n",
" <td>127</td>\n",
" <td>130</td>\n",
" <td>20</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>65</td>\n",
" <td>0.40</td>\n",
" <td>3334</td>\n",
" </tr>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>265</td>\n",
" <td>270</td>\n",
" <td>36</td>\n",
" <td>10</td>\n",
" <td>6</td>\n",
" <td>3</td>\n",
" <td>33</td>\n",
" <td>1.20</td>\n",
" <td>5853</td>\n",
" </tr>\n",
" <tr>\n",
" <th>42</th>\n",
" <td>157</td>\n",
" <td>163</td>\n",
" <td>18</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>12</td>\n",
" <td>1.13</td>\n",
" <td>3982</td>\n",
" </tr>\n",
" <tr>\n",
" <th>43</th>\n",
" <td>128</td>\n",
" <td>135</td>\n",
" <td>17</td>\n",
" <td>9</td>\n",
" <td>4</td>\n",
" <td>1</td>\n",
" <td>25</td>\n",
" <td>0.52</td>\n",
" <td>3374</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44</th>\n",
" <td>110</td>\n",
" <td>120</td>\n",
" <td>15</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>11</td>\n",
" <td>0.59</td>\n",
" <td>3119</td>\n",
" </tr>\n",
" <tr>\n",
" <th>45</th>\n",
" <td>123</td>\n",
" <td>130</td>\n",
" <td>18</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>43</td>\n",
" <td>0.39</td>\n",
" <td>3268</td>\n",
" </tr>\n",
" <tr>\n",
" <th>46</th>\n",
" <td>148</td>\n",
" <td>153</td>\n",
" <td>13</td>\n",
" <td>6</td>\n",
" <td>3</td>\n",
" <td>2</td>\n",
" <td>22</td>\n",
" <td>0.39</td>\n",
" <td>3950</td>\n",
" </tr>\n",
" <tr>\n",
" <th>47</th>\n",
" <td>212</td>\n",
" <td>230</td>\n",
" <td>39</td>\n",
" <td>12</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>202</td>\n",
" <td>4.29</td>\n",
" <td>3648</td>\n",
" </tr>\n",
" <tr>\n",
" <th>48</th>\n",
" <td>145</td>\n",
" <td>145</td>\n",
" <td>18</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>44</td>\n",
" <td>0.22</td>\n",
" <td>2783</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49</th>\n",
" <td>129</td>\n",
" <td>135</td>\n",
" <td>10</td>\n",
" <td>6</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>15</td>\n",
" <td>1.00</td>\n",
" <td>2438</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50</th>\n",
" <td>143</td>\n",
" <td>145</td>\n",
" <td>21</td>\n",
" <td>7</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>10</td>\n",
" <td>1.20</td>\n",
" <td>3529</td>\n",
" </tr>\n",
" <tr>\n",
" <th>51</th>\n",
" <td>247</td>\n",
" <td>252</td>\n",
" <td>29</td>\n",
" <td>9</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>4</td>\n",
" <td>1.25</td>\n",
" <td>4626</td>\n",
" </tr>\n",
" <tr>\n",
" <th>52</th>\n",
" <td>111</td>\n",
" <td>120</td>\n",
" <td>15</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>97</td>\n",
" <td>1.11</td>\n",
" <td>3205</td>\n",
" </tr>\n",
" <tr>\n",
" <th>53</th>\n",
" <td>133</td>\n",
" <td>145</td>\n",
" <td>26</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>42</td>\n",
" <td>0.36</td>\n",
" <td>3059</td>\n",
" </tr>\n",
" <tr>\n",
" <th>54</th>\n",
" <td>87</td>\n",
" <td>90</td>\n",
" <td>16</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>50</td>\n",
" <td>0.65</td>\n",
" <td>1445</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Sell List Living Rooms Beds Baths Age Acres Taxes\n",
"0 142 160 28 10 5 3 60 0.28 3167\n",
"1 175 180 18 8 4 1 12 0.43 4033\n",
"2 143 145 21 7 4 2 10 1.20 3529\n",
"3 129 132 13 6 3 1 41 0.33 1471\n",
"4 138 140 17 7 3 1 22 0.46 3204\n",
"5 232 240 25 8 4 3 5 2.05 3613\n",
"6 135 140 18 7 4 3 9 0.57 3028\n",
"7 150 160 20 8 4 3 18 4.00 3131\n",
"8 293 305 26 8 4 3 6 0.46 7088\n",
"9 207 225 22 8 4 2 16 2.22 5158\n",
"10 271 285 30 10 5 2 30 0.53 5702\n",
"11 89 90 10 5 3 1 43 0.30 2054\n",
"12 153 157 22 8 3 3 18 0.38 4127\n",
"13 87 90 16 7 3 1 50 0.65 1445\n",
"14 234 238 25 8 4 2 2 1.61 2087\n",
"15 106 116 20 8 4 1 13 0.22 2818\n",
"16 175 180 22 8 4 2 15 2.06 3917\n",
"17 165 170 17 8 4 2 33 0.46 2220\n",
"18 166 170 23 9 4 2 37 0.27 3498\n",
"19 136 140 19 7 3 1 22 0.63 3607\n",
"20 148 160 17 7 3 2 13 0.36 3648\n",
"21 151 153 19 8 4 2 24 0.34 3561\n",
"22 180 190 24 9 4 2 10 1.55 4681\n",
"23 293 305 26 8 4 3 6 0.46 7088\n",
"24 167 170 20 9 4 2 46 0.46 3482\n",
"25 190 193 22 9 5 2 37 0.48 3920\n",
"26 184 190 21 9 5 2 27 1.30 4162\n",
"27 157 165 20 8 4 2 7 0.30 3785\n",
"28 110 115 16 8 4 1 26 0.29 3103\n",
"29 135 145 18 7 4 1 35 0.43 3363\n",
"30 567 625 64 11 4 4 4 0.85 12192\n",
"31 180 185 20 8 4 2 11 1.00 3831\n",
"32 183 188 17 7 3 2 16 3.00 3564\n",
"33 185 193 20 9 3 2 56 6.49 3765\n",
"34 152 155 17 8 4 1 33 0.70 3361\n",
"35 148 153 13 6 3 2 22 0.39 3950\n",
"36 152 159 15 7 3 1 25 0.59 3055\n",
"37 146 150 16 7 3 1 31 0.36 2950\n",
"38 170 190 24 10 3 2 33 0.57 3346\n",
"39 106 116 20 8 4 1 13 0.22 2818\n",
"40 127 130 20 8 4 1 65 0.40 3334\n",
"41 265 270 36 10 6 3 33 1.20 5853\n",
"42 157 163 18 8 4 2 12 1.13 3982\n",
"43 128 135 17 9 4 1 25 0.52 3374\n",
"44 110 120 15 8 4 2 11 0.59 3119\n",
"45 123 130 18 8 4 2 43 0.39 3268\n",
"46 148 153 13 6 3 2 22 0.39 3950\n",
"47 212 230 39 12 5 3 202 4.29 3648\n",
"48 145 145 18 8 4 2 44 0.22 2783\n",
"49 129 135 10 6 3 1 15 1.00 2438\n",
"50 143 145 21 7 4 2 10 1.20 3529\n",
"51 247 252 29 9 4 2 4 1.25 4626\n",
"52 111 120 15 8 3 1 97 1.11 3205\n",
"53 133 145 26 7 3 1 42 0.36 3059\n",
"54 87 90 16 7 3 1 50 0.65 1445"
]
},
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"homes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Easy right?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 5\n",
"Find the mean selling price of the homes in `data/homes.csv`"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Sell 169.000000\n",
"List 176.836364\n",
"Living 20.945455\n",
"Rooms 7.981818\n",
"Beds 3.800000\n",
"Baths 1.854545\n",
"Age 29.309091\n",
"Acres 0.980909\n",
"Taxes 3711.545455\n",
"dtype: float64"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"homes.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `read_csv` function has a lot of optional arguments (more than 50). It's impossible to memorise all of them - it's usually best just to look up the particular functionality when you need it. \n",
"\n",
"You can search `pandas read_csv` online and find all of the documentation.\n",
"\n",
"There are also many other functions that can read textual data. Here are some of them:\n",
"\n",
"| Function | Description\n",
"| -- | -- |\n",
"| read_csv | Load delimited data from a file, URL, or file-like object. The default delimiter is a comma `,` |\n",
"| read_table | Load delimited data from a file, URL, or file-like object. The default delimiter is tab `\\t` |\n",
"| read_clipboard | Reads the last object you have copied (Ctrl-C) |\n",
"| read_excel | Read tabular data from Excel XLS or XLSX file |\n",
"| read_hdf | Read HDF5 file written by pandas |\n",
"| read_html | Read all tables found in the given HTML document |\n",
"| read_json | Read data from a JSON string representation |\n",
"| read_sql | Read the results of a SQL query |\n",
"\n",
"*Note: there are also other loading functions which are not touched upon here*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 6\n",
"There is another file in the data folder called `homes.xlsx`. Can you read it? Can you spot anything different?"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Sell</th>\n",
" <th>List</th>\n",
" <th>Living</th>\n",
" <th>Rooms</th>\n",
" <th>Beds</th>\n",
" <th>Baths</th>\n",
" <th>Age</th>\n",
" <th>Acres</th>\n",
" <th>Taxes</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>142</td>\n",
" <td>160.0</td>\n",
" <td>28.0</td>\n",
" <td>10.0</td>\n",
" <td>5.0</td>\n",
" <td>3.0</td>\n",
" <td>60.0</td>\n",
" <td>0.28</td>\n",
" <td>3167.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>175</td>\n",
" <td>180.0</td>\n",
" <td>18.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>12.0</td>\n",
" <td>0.43</td>\n",
" <td>4033.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>129</td>\n",
" <td>132.0</td>\n",
" <td>13.0</td>\n",
" <td>6.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>41.0</td>\n",
" <td>0.33</td>\n",
" <td>1471.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>138</td>\n",
" <td>140.0</td>\n",
" <td>17.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>22.0</td>\n",
" <td>0.46</td>\n",
" <td>3204.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>232</td>\n",
" <td>240.0</td>\n",
" <td>25.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>3.0</td>\n",
" <td>5.0</td>\n",
" <td>2.05</td>\n",
" <td>3613.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>135</td>\n",
" <td>140.0</td>\n",
" <td>18.0</td>\n",
" <td>7.0</td>\n",
" <td>4.0</td>\n",
" <td>3.0</td>\n",
" <td>9.0</td>\n",
" <td>0.57</td>\n",
" <td>3028.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>150</td>\n",
" <td>160.0</td>\n",
" <td>20.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>3.0</td>\n",
" <td>18.0</td>\n",
" <td>4.00</td>\n",
" <td>3131.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>207</td>\n",
" <td>225.0</td>\n",
" <td>22.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>16.0</td>\n",
" <td>2.22</td>\n",
" <td>5158.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>271</td>\n",
" <td>285.0</td>\n",
" <td>30.0</td>\n",
" <td>10.0</td>\n",
" <td>5.0</td>\n",
" <td>2.0</td>\n",
" <td>30.0</td>\n",
" <td>0.53</td>\n",
" <td>5702.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>89</td>\n",
" <td>90.0</td>\n",
" <td>10.0</td>\n",
" <td>5.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>43.0</td>\n",
" <td>0.30</td>\n",
" <td>2054.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>153</td>\n",
" <td>157.0</td>\n",
" <td>22.0</td>\n",
" <td>8.0</td>\n",
" <td>3.0</td>\n",
" <td>3.0</td>\n",
" <td>18.0</td>\n",
" <td>0.38</td>\n",
" <td>4127.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>87</td>\n",
" <td>90.0</td>\n",
" <td>16.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>50.0</td>\n",
" <td>0.65</td>\n",
" <td>1445.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>234</td>\n",
" <td>238.0</td>\n",
" <td>25.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>2.0</td>\n",
" <td>1.61</td>\n",
" <td>2087.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>106</td>\n",
" <td>116.0</td>\n",
" <td>20.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>13.0</td>\n",
" <td>0.22</td>\n",
" <td>2818.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>175</td>\n",
" <td>180.0</td>\n",
" <td>22.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>15.0</td>\n",
" <td>2.06</td>\n",
" <td>3917.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>165</td>\n",
" <td>170.0</td>\n",
" <td>17.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>33.0</td>\n",
" <td>0.46</td>\n",
" <td>2220.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>166</td>\n",
" <td>170.0</td>\n",
" <td>23.0</td>\n",
" <td>9.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>37.0</td>\n",
" <td>0.27</td>\n",
" <td>3498.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>136</td>\n",
" <td>140.0</td>\n",
" <td>19.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>22.0</td>\n",
" <td>0.63</td>\n",
" <td>3607.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>148</td>\n",
" <td>160.0</td>\n",
" <td>17.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>2.0</td>\n",
" <td>13.0</td>\n",
" <td>0.36</td>\n",
" <td>3648.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>151</td>\n",
" <td>153.0</td>\n",
" <td>19.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>24.0</td>\n",
" <td>0.34</td>\n",
" <td>3561.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>180</td>\n",
" <td>190.0</td>\n",
" <td>24.0</td>\n",
" <td>9.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>10.0</td>\n",
" <td>1.55</td>\n",
" <td>4681.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>293</td>\n",
" <td>305.0</td>\n",
" <td>26.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>3.0</td>\n",
" <td>6.0</td>\n",
" <td>0.46</td>\n",
" <td>7088.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>167</td>\n",
" <td>170.0</td>\n",
" <td>20.0</td>\n",
" <td>9.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>46.0</td>\n",
" <td>0.46</td>\n",
" <td>3482.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>190</td>\n",
" <td>193.0</td>\n",
" <td>22.0</td>\n",
" <td>9.0</td>\n",
" <td>5.0</td>\n",
" <td>2.0</td>\n",
" <td>37.0</td>\n",
" <td>0.48</td>\n",
" <td>3920.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>184</td>\n",
" <td>190.0</td>\n",
" <td>21.0</td>\n",
" <td>9.0</td>\n",
" <td>5.0</td>\n",
" <td>2.0</td>\n",
" <td>27.0</td>\n",
" <td>1.30</td>\n",
" <td>4162.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>157</td>\n",
" <td>165.0</td>\n",
" <td>20.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>7.0</td>\n",
" <td>0.30</td>\n",
" <td>3785.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>110</td>\n",
" <td>115.0</td>\n",
" <td>16.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>26.0</td>\n",
" <td>0.29</td>\n",
" <td>3103.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>135</td>\n",
" <td>145.0</td>\n",
" <td>18.0</td>\n",
" <td>7.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>35.0</td>\n",
" <td>0.43</td>\n",
" <td>3363.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>567</td>\n",
" <td>625.0</td>\n",
" <td>64.0</td>\n",
" <td>11.0</td>\n",
" <td>4.0</td>\n",
" <td>4.0</td>\n",
" <td>4.0</td>\n",
" <td>0.85</td>\n",
" <td>12192.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>180</td>\n",
" <td>185.0</td>\n",
" <td>20.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>11.0</td>\n",
" <td>1.00</td>\n",
" <td>3831.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>183</td>\n",
" <td>188.0</td>\n",
" <td>17.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>2.0</td>\n",
" <td>16.0</td>\n",
" <td>3.00</td>\n",
" <td>3564.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>185</td>\n",
" <td>193.0</td>\n",
" <td>20.0</td>\n",
" <td>9.0</td>\n",
" <td>3.0</td>\n",
" <td>2.0</td>\n",
" <td>56.0</td>\n",
" <td>6.49</td>\n",
" <td>3765.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>152</td>\n",
" <td>155.0</td>\n",
" <td>17.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>33.0</td>\n",
" <td>0.70</td>\n",
" <td>3361.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>33</th>\n",
" <td>148</td>\n",
" <td>153.0</td>\n",
" <td>13.0</td>\n",
" <td>6.0</td>\n",
" <td>3.0</td>\n",
" <td>2.0</td>\n",
" <td>22.0</td>\n",
" <td>0.39</td>\n",
" <td>3950.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>34</th>\n",
" <td>152</td>\n",
" <td>159.0</td>\n",
" <td>15.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>25.0</td>\n",
" <td>0.59</td>\n",
" <td>3055.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>35</th>\n",
" <td>146</td>\n",
" <td>150.0</td>\n",
" <td>16.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>31.0</td>\n",
" <td>0.36</td>\n",
" <td>2950.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>36</th>\n",
" <td>170</td>\n",
" <td>190.0</td>\n",
" <td>24.0</td>\n",
" <td>10.0</td>\n",
" <td>3.0</td>\n",
" <td>2.0</td>\n",
" <td>33.0</td>\n",
" <td>0.57</td>\n",
" <td>3346.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>37</th>\n",
" <td>127</td>\n",
" <td>130.0</td>\n",
" <td>20.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>65.0</td>\n",
" <td>0.40</td>\n",
" <td>3334.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38</th>\n",
" <td>265</td>\n",
" <td>270.0</td>\n",
" <td>36.0</td>\n",
" <td>10.0</td>\n",
" <td>6.0</td>\n",
" <td>3.0</td>\n",
" <td>33.0</td>\n",
" <td>1.20</td>\n",
" <td>5853.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>39</th>\n",
" <td>157</td>\n",
" <td>163.0</td>\n",
" <td>18.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>12.0</td>\n",
" <td>1.13</td>\n",
" <td>3982.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>40</th>\n",
" <td>128</td>\n",
" <td>135.0</td>\n",
" <td>17.0</td>\n",
" <td>9.0</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>25.0</td>\n",
" <td>0.52</td>\n",
" <td>3374.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>110</td>\n",
" <td>120.0</td>\n",
" <td>15.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>11.0</td>\n",
" <td>0.59</td>\n",
" <td>3119.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>42</th>\n",
" <td>123</td>\n",
" <td>130.0</td>\n",
" <td>18.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>43.0</td>\n",
" <td>0.39</td>\n",
" <td>3268.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>43</th>\n",
" <td>212</td>\n",
" <td>230.0</td>\n",
" <td>39.0</td>\n",
" <td>12.0</td>\n",
" <td>5.0</td>\n",
" <td>3.0</td>\n",
" <td>202.0</td>\n",
" <td>4.29</td>\n",
" <td>3648.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44</th>\n",
" <td>145</td>\n",
" <td>145.0</td>\n",
" <td>18.0</td>\n",
" <td>8.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>44.0</td>\n",
" <td>0.22</td>\n",
" <td>2783.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>45</th>\n",
" <td>129</td>\n",
" <td>135.0</td>\n",
" <td>10.0</td>\n",
" <td>6.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>15.0</td>\n",
" <td>1.00</td>\n",
" <td>2438.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>46</th>\n",
" <td>143</td>\n",
" <td>145.0</td>\n",
" <td>21.0</td>\n",
" <td>7.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>10.0</td>\n",
" <td>1.20</td>\n",
" <td>3529.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>47</th>\n",
" <td>247</td>\n",
" <td>252.0</td>\n",
" <td>29.0</td>\n",
" <td>9.0</td>\n",
" <td>4.0</td>\n",
" <td>2.0</td>\n",
" <td>4.0</td>\n",
" <td>1.25</td>\n",
" <td>4626.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>48</th>\n",
" <td>111</td>\n",
" <td>120.0</td>\n",
" <td>15.0</td>\n",
" <td>8.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>97.0</td>\n",
" <td>1.11</td>\n",
" <td>3205.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49</th>\n",
" <td>133</td>\n",
" <td>145.0</td>\n",
" <td>26.0</td>\n",
" <td>7.0</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>42.0</td>\n",
" <td>0.36</td>\n",
" <td>3059.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50</th>\n",
" <td></td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Sell List Living Rooms Beds Baths Age Acres Taxes\n",
"0 142 160.0 28.0 10.0 5.0 3.0 60.0 0.28 3167.0\n",
"1 175 180.0 18.0 8.0 4.0 1.0 12.0 0.43 4033.0\n",
"2 129 132.0 13.0 6.0 3.0 1.0 41.0 0.33 1471.0\n",
"3 138 140.0 17.0 7.0 3.0 1.0 22.0 0.46 3204.0\n",
"4 232 240.0 25.0 8.0 4.0 3.0 5.0 2.05 3613.0\n",
"5 135 140.0 18.0 7.0 4.0 3.0 9.0 0.57 3028.0\n",
"6 150 160.0 20.0 8.0 4.0 3.0 18.0 4.00 3131.0\n",
"7 207 225.0 22.0 8.0 4.0 2.0 16.0 2.22 5158.0\n",
"8 271 285.0 30.0 10.0 5.0 2.0 30.0 0.53 5702.0\n",
"9 89 90.0 10.0 5.0 3.0 1.0 43.0 0.30 2054.0\n",
"10 153 157.0 22.0 8.0 3.0 3.0 18.0 0.38 4127.0\n",
"11 87 90.0 16.0 7.0 3.0 1.0 50.0 0.65 1445.0\n",
"12 234 238.0 25.0 8.0 4.0 2.0 2.0 1.61 2087.0\n",
"13 106 116.0 20.0 8.0 4.0 1.0 13.0 0.22 2818.0\n",
"14 175 180.0 22.0 8.0 4.0 2.0 15.0 2.06 3917.0\n",
"15 165 170.0 17.0 8.0 4.0 2.0 33.0 0.46 2220.0\n",
"16 166 170.0 23.0 9.0 4.0 2.0 37.0 0.27 3498.0\n",
"17 136 140.0 19.0 7.0 3.0 1.0 22.0 0.63 3607.0\n",
"18 148 160.0 17.0 7.0 3.0 2.0 13.0 0.36 3648.0\n",
"19 151 153.0 19.0 8.0 4.0 2.0 24.0 0.34 3561.0\n",
"20 180 190.0 24.0 9.0 4.0 2.0 10.0 1.55 4681.0\n",
"21 293 305.0 26.0 8.0 4.0 3.0 6.0 0.46 7088.0\n",
"22 167 170.0 20.0 9.0 4.0 2.0 46.0 0.46 3482.0\n",
"23 190 193.0 22.0 9.0 5.0 2.0 37.0 0.48 3920.0\n",
"24 184 190.0 21.0 9.0 5.0 2.0 27.0 1.30 4162.0\n",
"25 157 165.0 20.0 8.0 4.0 2.0 7.0 0.30 3785.0\n",
"26 110 115.0 16.0 8.0 4.0 1.0 26.0 0.29 3103.0\n",
"27 135 145.0 18.0 7.0 4.0 1.0 35.0 0.43 3363.0\n",
"28 567 625.0 64.0 11.0 4.0 4.0 4.0 0.85 12192.0\n",
"29 180 185.0 20.0 8.0 4.0 2.0 11.0 1.00 3831.0\n",
"30 183 188.0 17.0 7.0 3.0 2.0 16.0 3.00 3564.0\n",
"31 185 193.0 20.0 9.0 3.0 2.0 56.0 6.49 3765.0\n",
"32 152 155.0 17.0 8.0 4.0 1.0 33.0 0.70 3361.0\n",
"33 148 153.0 13.0 6.0 3.0 2.0 22.0 0.39 3950.0\n",
"34 152 159.0 15.0 7.0 3.0 1.0 25.0 0.59 3055.0\n",
"35 146 150.0 16.0 7.0 3.0 1.0 31.0 0.36 2950.0\n",
"36 170 190.0 24.0 10.0 3.0 2.0 33.0 0.57 3346.0\n",
"37 127 130.0 20.0 8.0 4.0 1.0 65.0 0.40 3334.0\n",
"38 265 270.0 36.0 10.0 6.0 3.0 33.0 1.20 5853.0\n",
"39 157 163.0 18.0 8.0 4.0 2.0 12.0 1.13 3982.0\n",
"40 128 135.0 17.0 9.0 4.0 1.0 25.0 0.52 3374.0\n",
"41 110 120.0 15.0 8.0 4.0 2.0 11.0 0.59 3119.0\n",
"42 123 130.0 18.0 8.0 4.0 2.0 43.0 0.39 3268.0\n",
"43 212 230.0 39.0 12.0 5.0 3.0 202.0 4.29 3648.0\n",
"44 145 145.0 18.0 8.0 4.0 2.0 44.0 0.22 2783.0\n",
"45 129 135.0 10.0 6.0 3.0 1.0 15.0 1.00 2438.0\n",
"46 143 145.0 21.0 7.0 4.0 2.0 10.0 1.20 3529.0\n",
"47 247 252.0 29.0 9.0 4.0 2.0 4.0 1.25 4626.0\n",
"48 111 120.0 15.0 8.0 3.0 1.0 97.0 1.11 3205.0\n",
"49 133 145.0 26.0 7.0 3.0 1.0 42.0 0.36 3059.0\n",
"50 NaN NaN NaN NaN NaN NaN NaN NaN"
]
},
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_excel(\"data/homes.xlsx\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Writing CSV files\n",
"Easy!"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [],
"source": [
"homes.to_csv(\"test.csv\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 7\n",
"Create a DataFrame which consists of all numbers 0 to 1000. Reshape it into 50 rows and save it to a `.csv` file. How many columns did you end up with?"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>10</th>\n",
" <th>11</th>\n",
" <th>12</th>\n",
" <th>13</th>\n",
" <th>14</th>\n",
" <th>15</th>\n",
" <th>16</th>\n",
" <th>17</th>\n",
" <th>18</th>\n",
" <th>19</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>11</td>\n",
" <td>12</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>15</td>\n",
" <td>16</td>\n",
" <td>17</td>\n",
" <td>18</td>\n",
" <td>19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>21</td>\n",
" <td>22</td>\n",
" <td>23</td>\n",
" <td>24</td>\n",
" <td>25</td>\n",
" <td>26</td>\n",
" <td>27</td>\n",
" <td>28</td>\n",
" <td>29</td>\n",
" <td>30</td>\n",
" <td>31</td>\n",
" <td>32</td>\n",
" <td>33</td>\n",
" <td>34</td>\n",
" <td>35</td>\n",
" <td>36</td>\n",
" <td>37</td>\n",
" <td>38</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>40</td>\n",
" <td>41</td>\n",
" <td>42</td>\n",
" <td>43</td>\n",
" <td>44</td>\n",
" <td>45</td>\n",
" <td>46</td>\n",
" <td>47</td>\n",
" <td>48</td>\n",
" <td>49</td>\n",
" <td>50</td>\n",
" <td>51</td>\n",
" <td>52</td>\n",
" <td>53</td>\n",
" <td>54</td>\n",
" <td>55</td>\n",
" <td>56</td>\n",
" <td>57</td>\n",
" <td>58</td>\n",
" <td>59</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>60</td>\n",
" <td>61</td>\n",
" <td>62</td>\n",
" <td>63</td>\n",
" <td>64</td>\n",
" <td>65</td>\n",
" <td>66</td>\n",
" <td>67</td>\n",
" <td>68</td>\n",
" <td>69</td>\n",
" <td>70</td>\n",
" <td>71</td>\n",
" <td>72</td>\n",
" <td>73</td>\n",
" <td>74</td>\n",
" <td>75</td>\n",
" <td>76</td>\n",
" <td>77</td>\n",
" <td>78</td>\n",
" <td>79</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>80</td>\n",
" <td>81</td>\n",
" <td>82</td>\n",
" <td>83</td>\n",
" <td>84</td>\n",
" <td>85</td>\n",
" <td>86</td>\n",
" <td>87</td>\n",
" <td>88</td>\n",
" <td>89</td>\n",
" <td>90</td>\n",
" <td>91</td>\n",
" <td>92</td>\n",
" <td>93</td>\n",
" <td>94</td>\n",
" <td>95</td>\n",
" <td>96</td>\n",
" <td>97</td>\n",
" <td>98</td>\n",
" <td>99</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>100</td>\n",
" <td>101</td>\n",
" <td>102</td>\n",
" <td>103</td>\n",
" <td>104</td>\n",
" <td>105</td>\n",
" <td>106</td>\n",
" <td>107</td>\n",
" <td>108</td>\n",
" <td>109</td>\n",
" <td>110</td>\n",
" <td>111</td>\n",
" <td>112</td>\n",
" <td>113</td>\n",
" <td>114</td>\n",
" <td>115</td>\n",
" <td>116</td>\n",
" <td>117</td>\n",
" <td>118</td>\n",
" <td>119</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>120</td>\n",
" <td>121</td>\n",
" <td>122</td>\n",
" <td>123</td>\n",
" <td>124</td>\n",
" <td>125</td>\n",
" <td>126</td>\n",
" <td>127</td>\n",
" <td>128</td>\n",
" <td>129</td>\n",
" <td>130</td>\n",
" <td>131</td>\n",
" <td>132</td>\n",
" <td>133</td>\n",
" <td>134</td>\n",
" <td>135</td>\n",
" <td>136</td>\n",
" <td>137</td>\n",
" <td>138</td>\n",
" <td>139</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>140</td>\n",
" <td>141</td>\n",
" <td>142</td>\n",
" <td>143</td>\n",
" <td>144</td>\n",
" <td>145</td>\n",
" <td>146</td>\n",
" <td>147</td>\n",
" <td>148</td>\n",
" <td>149</td>\n",
" <td>150</td>\n",
" <td>151</td>\n",
" <td>152</td>\n",
" <td>153</td>\n",
" <td>154</td>\n",
" <td>155</td>\n",
" <td>156</td>\n",
" <td>157</td>\n",
" <td>158</td>\n",
" <td>159</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>160</td>\n",
" <td>161</td>\n",
" <td>162</td>\n",
" <td>163</td>\n",
" <td>164</td>\n",
" <td>165</td>\n",
" <td>166</td>\n",
" <td>167</td>\n",
" <td>168</td>\n",
" <td>169</td>\n",
" <td>170</td>\n",
" <td>171</td>\n",
" <td>172</td>\n",
" <td>173</td>\n",
" <td>174</td>\n",
" <td>175</td>\n",
" <td>176</td>\n",
" <td>177</td>\n",
" <td>178</td>\n",
" <td>179</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>180</td>\n",
" <td>181</td>\n",
" <td>182</td>\n",
" <td>183</td>\n",
" <td>184</td>\n",
" <td>185</td>\n",
" <td>186</td>\n",
" <td>187</td>\n",
" <td>188</td>\n",
" <td>189</td>\n",
" <td>190</td>\n",
" <td>191</td>\n",
" <td>192</td>\n",
" <td>193</td>\n",
" <td>194</td>\n",
" <td>195</td>\n",
" <td>196</td>\n",
" <td>197</td>\n",
" <td>198</td>\n",
" <td>199</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>200</td>\n",
" <td>201</td>\n",
" <td>202</td>\n",
" <td>203</td>\n",
" <td>204</td>\n",
" <td>205</td>\n",
" <td>206</td>\n",
" <td>207</td>\n",
" <td>208</td>\n",
" <td>209</td>\n",
" <td>210</td>\n",
" <td>211</td>\n",
" <td>212</td>\n",
" <td>213</td>\n",
" <td>214</td>\n",
" <td>215</td>\n",
" <td>216</td>\n",
" <td>217</td>\n",
" <td>218</td>\n",
" <td>219</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>220</td>\n",
" <td>221</td>\n",
" <td>222</td>\n",
" <td>223</td>\n",
" <td>224</td>\n",
" <td>225</td>\n",
" <td>226</td>\n",
" <td>227</td>\n",
" <td>228</td>\n",
" <td>229</td>\n",
" <td>230</td>\n",
" <td>231</td>\n",
" <td>232</td>\n",
" <td>233</td>\n",
" <td>234</td>\n",
" <td>235</td>\n",
" <td>236</td>\n",
" <td>237</td>\n",
" <td>238</td>\n",
" <td>239</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>240</td>\n",
" <td>241</td>\n",
" <td>242</td>\n",
" <td>243</td>\n",
" <td>244</td>\n",
" <td>245</td>\n",
" <td>246</td>\n",
" <td>247</td>\n",
" <td>248</td>\n",
" <td>249</td>\n",
" <td>250</td>\n",
" <td>251</td>\n",
" <td>252</td>\n",
" <td>253</td>\n",
" <td>254</td>\n",
" <td>255</td>\n",
" <td>256</td>\n",
" <td>257</td>\n",
" <td>258</td>\n",
" <td>259</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>260</td>\n",
" <td>261</td>\n",
" <td>262</td>\n",
" <td>263</td>\n",
" <td>264</td>\n",
" <td>265</td>\n",
" <td>266</td>\n",
" <td>267</td>\n",
" <td>268</td>\n",
" <td>269</td>\n",
" <td>270</td>\n",
" <td>271</td>\n",
" <td>272</td>\n",
" <td>273</td>\n",
" <td>274</td>\n",
" <td>275</td>\n",
" <td>276</td>\n",
" <td>277</td>\n",
" <td>278</td>\n",
" <td>279</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>280</td>\n",
" <td>281</td>\n",
" <td>282</td>\n",
" <td>283</td>\n",
" <td>284</td>\n",
" <td>285</td>\n",
" <td>286</td>\n",
" <td>287</td>\n",
" <td>288</td>\n",
" <td>289</td>\n",
" <td>290</td>\n",
" <td>291</td>\n",
" <td>292</td>\n",
" <td>293</td>\n",
" <td>294</td>\n",
" <td>295</td>\n",
" <td>296</td>\n",
" <td>297</td>\n",
" <td>298</td>\n",
" <td>299</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>300</td>\n",
" <td>301</td>\n",
" <td>302</td>\n",
" <td>303</td>\n",
" <td>304</td>\n",
" <td>305</td>\n",
" <td>306</td>\n",
" <td>307</td>\n",
" <td>308</td>\n",
" <td>309</td>\n",
" <td>310</td>\n",
" <td>311</td>\n",
" <td>312</td>\n",
" <td>313</td>\n",
" <td>314</td>\n",
" <td>315</td>\n",
" <td>316</td>\n",
" <td>317</td>\n",
" <td>318</td>\n",
" <td>319</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>320</td>\n",
" <td>321</td>\n",
" <td>322</td>\n",
" <td>323</td>\n",
" <td>324</td>\n",
" <td>325</td>\n",
" <td>326</td>\n",
" <td>327</td>\n",
" <td>328</td>\n",
" <td>329</td>\n",
" <td>330</td>\n",
" <td>331</td>\n",
" <td>332</td>\n",
" <td>333</td>\n",
" <td>334</td>\n",
" <td>335</td>\n",
" <td>336</td>\n",
" <td>337</td>\n",
" <td>338</td>\n",
" <td>339</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>340</td>\n",
" <td>341</td>\n",
" <td>342</td>\n",
" <td>343</td>\n",
" <td>344</td>\n",
" <td>345</td>\n",
" <td>346</td>\n",
" <td>347</td>\n",
" <td>348</td>\n",
" <td>349</td>\n",
" <td>350</td>\n",
" <td>351</td>\n",
" <td>352</td>\n",
" <td>353</td>\n",
" <td>354</td>\n",
" <td>355</td>\n",
" <td>356</td>\n",
" <td>357</td>\n",
" <td>358</td>\n",
" <td>359</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>360</td>\n",
" <td>361</td>\n",
" <td>362</td>\n",
" <td>363</td>\n",
" <td>364</td>\n",
" <td>365</td>\n",
" <td>366</td>\n",
" <td>367</td>\n",
" <td>368</td>\n",
" <td>369</td>\n",
" <td>370</td>\n",
" <td>371</td>\n",
" <td>372</td>\n",
" <td>373</td>\n",
" <td>374</td>\n",
" <td>375</td>\n",
" <td>376</td>\n",
" <td>377</td>\n",
" <td>378</td>\n",
" <td>379</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>380</td>\n",
" <td>381</td>\n",
" <td>382</td>\n",
" <td>383</td>\n",
" <td>384</td>\n",
" <td>385</td>\n",
" <td>386</td>\n",
" <td>387</td>\n",
" <td>388</td>\n",
" <td>389</td>\n",
" <td>390</td>\n",
" <td>391</td>\n",
" <td>392</td>\n",
" <td>393</td>\n",
" <td>394</td>\n",
" <td>395</td>\n",
" <td>396</td>\n",
" <td>397</td>\n",
" <td>398</td>\n",
" <td>399</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>400</td>\n",
" <td>401</td>\n",
" <td>402</td>\n",
" <td>403</td>\n",
" <td>404</td>\n",
" <td>405</td>\n",
" <td>406</td>\n",
" <td>407</td>\n",
" <td>408</td>\n",
" <td>409</td>\n",
" <td>410</td>\n",
" <td>411</td>\n",
" <td>412</td>\n",
" <td>413</td>\n",
" <td>414</td>\n",
" <td>415</td>\n",
" <td>416</td>\n",
" <td>417</td>\n",
" <td>418</td>\n",
" <td>419</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>420</td>\n",
" <td>421</td>\n",
" <td>422</td>\n",
" <td>423</td>\n",
" <td>424</td>\n",
" <td>425</td>\n",
" <td>426</td>\n",
" <td>427</td>\n",
" <td>428</td>\n",
" <td>429</td>\n",
" <td>430</td>\n",
" <td>431</td>\n",
" <td>432</td>\n",
" <td>433</td>\n",
" <td>434</td>\n",
" <td>435</td>\n",
" <td>436</td>\n",
" <td>437</td>\n",
" <td>438</td>\n",
" <td>439</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>440</td>\n",
" <td>441</td>\n",
" <td>442</td>\n",
" <td>443</td>\n",
" <td>444</td>\n",
" <td>445</td>\n",
" <td>446</td>\n",
" <td>447</td>\n",
" <td>448</td>\n",
" <td>449</td>\n",
" <td>450</td>\n",
" <td>451</td>\n",
" <td>452</td>\n",
" <td>453</td>\n",
" <td>454</td>\n",
" <td>455</td>\n",
" <td>456</td>\n",
" <td>457</td>\n",
" <td>458</td>\n",
" <td>459</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>460</td>\n",
" <td>461</td>\n",
" <td>462</td>\n",
" <td>463</td>\n",
" <td>464</td>\n",
" <td>465</td>\n",
" <td>466</td>\n",
" <td>467</td>\n",
" <td>468</td>\n",
" <td>469</td>\n",
" <td>470</td>\n",
" <td>471</td>\n",
" <td>472</td>\n",
" <td>473</td>\n",
" <td>474</td>\n",
" <td>475</td>\n",
" <td>476</td>\n",
" <td>477</td>\n",
" <td>478</td>\n",
" <td>479</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>480</td>\n",
" <td>481</td>\n",
" <td>482</td>\n",
" <td>483</td>\n",
" <td>484</td>\n",
" <td>485</td>\n",
" <td>486</td>\n",
" <td>487</td>\n",
" <td>488</td>\n",
" <td>489</td>\n",
" <td>490</td>\n",
" <td>491</td>\n",
" <td>492</td>\n",
" <td>493</td>\n",
" <td>494</td>\n",
" <td>495</td>\n",
" <td>496</td>\n",
" <td>497</td>\n",
" <td>498</td>\n",
" <td>499</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>500</td>\n",
" <td>501</td>\n",
" <td>502</td>\n",
" <td>503</td>\n",
" <td>504</td>\n",
" <td>505</td>\n",
" <td>506</td>\n",
" <td>507</td>\n",
" <td>508</td>\n",
" <td>509</td>\n",
" <td>510</td>\n",
" <td>511</td>\n",
" <td>512</td>\n",
" <td>513</td>\n",
" <td>514</td>\n",
" <td>515</td>\n",
" <td>516</td>\n",
" <td>517</td>\n",
" <td>518</td>\n",
" <td>519</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>520</td>\n",
" <td>521</td>\n",
" <td>522</td>\n",
" <td>523</td>\n",
" <td>524</td>\n",
" <td>525</td>\n",
" <td>526</td>\n",
" <td>527</td>\n",
" <td>528</td>\n",
" <td>529</td>\n",
" <td>530</td>\n",
" <td>531</td>\n",
" <td>532</td>\n",
" <td>533</td>\n",
" <td>534</td>\n",
" <td>535</td>\n",
" <td>536</td>\n",
" <td>537</td>\n",
" <td>538</td>\n",
" <td>539</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>540</td>\n",
" <td>541</td>\n",
" <td>542</td>\n",
" <td>543</td>\n",
" <td>544</td>\n",
" <td>545</td>\n",
" <td>546</td>\n",
" <td>547</td>\n",
" <td>548</td>\n",
" <td>549</td>\n",
" <td>550</td>\n",
" <td>551</td>\n",
" <td>552</td>\n",
" <td>553</td>\n",
" <td>554</td>\n",
" <td>555</td>\n",
" <td>556</td>\n",
" <td>557</td>\n",
" <td>558</td>\n",
" <td>559</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>560</td>\n",
" <td>561</td>\n",
" <td>562</td>\n",
" <td>563</td>\n",
" <td>564</td>\n",
" <td>565</td>\n",
" <td>566</td>\n",
" <td>567</td>\n",
" <td>568</td>\n",
" <td>569</td>\n",
" <td>570</td>\n",
" <td>571</td>\n",
" <td>572</td>\n",
" <td>573</td>\n",
" <td>574</td>\n",
" <td>575</td>\n",
" <td>576</td>\n",
" <td>577</td>\n",
" <td>578</td>\n",
" <td>579</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>580</td>\n",
" <td>581</td>\n",
" <td>582</td>\n",
" <td>583</td>\n",
" <td>584</td>\n",
" <td>585</td>\n",
" <td>586</td>\n",
" <td>587</td>\n",
" <td>588</td>\n",
" <td>589</td>\n",
" <td>590</td>\n",
" <td>591</td>\n",
" <td>592</td>\n",
" <td>593</td>\n",
" <td>594</td>\n",
" <td>595</td>\n",
" <td>596</td>\n",
" <td>597</td>\n",
" <td>598</td>\n",
" <td>599</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>600</td>\n",
" <td>601</td>\n",
" <td>602</td>\n",
" <td>603</td>\n",
" <td>604</td>\n",
" <td>605</td>\n",
" <td>606</td>\n",
" <td>607</td>\n",
" <td>608</td>\n",
" <td>609</td>\n",
" <td>610</td>\n",
" <td>611</td>\n",
" <td>612</td>\n",
" <td>613</td>\n",
" <td>614</td>\n",
" <td>615</td>\n",
" <td>616</td>\n",
" <td>617</td>\n",
" <td>618</td>\n",
" <td>619</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>620</td>\n",
" <td>621</td>\n",
" <td>622</td>\n",
" <td>623</td>\n",
" <td>624</td>\n",
" <td>625</td>\n",
" <td>626</td>\n",
" <td>627</td>\n",
" <td>628</td>\n",
" <td>629</td>\n",
" <td>630</td>\n",
" <td>631</td>\n",
" <td>632</td>\n",
" <td>633</td>\n",
" <td>634</td>\n",
" <td>635</td>\n",
" <td>636</td>\n",
" <td>637</td>\n",
" <td>638</td>\n",
" <td>639</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>640</td>\n",
" <td>641</td>\n",
" <td>642</td>\n",
" <td>643</td>\n",
" <td>644</td>\n",
" <td>645</td>\n",
" <td>646</td>\n",
" <td>647</td>\n",
" <td>648</td>\n",
" <td>649</td>\n",
" <td>650</td>\n",
" <td>651</td>\n",
" <td>652</td>\n",
" <td>653</td>\n",
" <td>654</td>\n",
" <td>655</td>\n",
" <td>656</td>\n",
" <td>657</td>\n",
" <td>658</td>\n",
" <td>659</td>\n",
" </tr>\n",
" <tr>\n",
" <th>33</th>\n",
" <td>660</td>\n",
" <td>661</td>\n",
" <td>662</td>\n",
" <td>663</td>\n",
" <td>664</td>\n",
" <td>665</td>\n",
" <td>666</td>\n",
" <td>667</td>\n",
" <td>668</td>\n",
" <td>669</td>\n",
" <td>670</td>\n",
" <td>671</td>\n",
" <td>672</td>\n",
" <td>673</td>\n",
" <td>674</td>\n",
" <td>675</td>\n",
" <td>676</td>\n",
" <td>677</td>\n",
" <td>678</td>\n",
" <td>679</td>\n",
" </tr>\n",
" <tr>\n",
" <th>34</th>\n",
" <td>680</td>\n",
" <td>681</td>\n",
" <td>682</td>\n",
" <td>683</td>\n",
" <td>684</td>\n",
" <td>685</td>\n",
" <td>686</td>\n",
" <td>687</td>\n",
" <td>688</td>\n",
" <td>689</td>\n",
" <td>690</td>\n",
" <td>691</td>\n",
" <td>692</td>\n",
" <td>693</td>\n",
" <td>694</td>\n",
" <td>695</td>\n",
" <td>696</td>\n",
" <td>697</td>\n",
" <td>698</td>\n",
" <td>699</td>\n",
" </tr>\n",
" <tr>\n",
" <th>35</th>\n",
" <td>700</td>\n",
" <td>701</td>\n",
" <td>702</td>\n",
" <td>703</td>\n",
" <td>704</td>\n",
" <td>705</td>\n",
" <td>706</td>\n",
" <td>707</td>\n",
" <td>708</td>\n",
" <td>709</td>\n",
" <td>710</td>\n",
" <td>711</td>\n",
" <td>712</td>\n",
" <td>713</td>\n",
" <td>714</td>\n",
" <td>715</td>\n",
" <td>716</td>\n",
" <td>717</td>\n",
" <td>718</td>\n",
" <td>719</td>\n",
" </tr>\n",
" <tr>\n",
" <th>36</th>\n",
" <td>720</td>\n",
" <td>721</td>\n",
" <td>722</td>\n",
" <td>723</td>\n",
" <td>724</td>\n",
" <td>725</td>\n",
" <td>726</td>\n",
" <td>727</td>\n",
" <td>728</td>\n",
" <td>729</td>\n",
" <td>730</td>\n",
" <td>731</td>\n",
" <td>732</td>\n",
" <td>733</td>\n",
" <td>734</td>\n",
" <td>735</td>\n",
" <td>736</td>\n",
" <td>737</td>\n",
" <td>738</td>\n",
" <td>739</td>\n",
" </tr>\n",
" <tr>\n",
" <th>37</th>\n",
" <td>740</td>\n",
" <td>741</td>\n",
" <td>742</td>\n",
" <td>743</td>\n",
" <td>744</td>\n",
" <td>745</td>\n",
" <td>746</td>\n",
" <td>747</td>\n",
" <td>748</td>\n",
" <td>749</td>\n",
" <td>750</td>\n",
" <td>751</td>\n",
" <td>752</td>\n",
" <td>753</td>\n",
" <td>754</td>\n",
" <td>755</td>\n",
" <td>756</td>\n",
" <td>757</td>\n",
" <td>758</td>\n",
" <td>759</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38</th>\n",
" <td>760</td>\n",
" <td>761</td>\n",
" <td>762</td>\n",
" <td>763</td>\n",
" <td>764</td>\n",
" <td>765</td>\n",
" <td>766</td>\n",
" <td>767</td>\n",
" <td>768</td>\n",
" <td>769</td>\n",
" <td>770</td>\n",
" <td>771</td>\n",
" <td>772</td>\n",
" <td>773</td>\n",
" <td>774</td>\n",
" <td>775</td>\n",
" <td>776</td>\n",
" <td>777</td>\n",
" <td>778</td>\n",
" <td>779</td>\n",
" </tr>\n",
" <tr>\n",
" <th>39</th>\n",
" <td>780</td>\n",
" <td>781</td>\n",
" <td>782</td>\n",
" <td>783</td>\n",
" <td>784</td>\n",
" <td>785</td>\n",
" <td>786</td>\n",
" <td>787</td>\n",
" <td>788</td>\n",
" <td>789</td>\n",
" <td>790</td>\n",
" <td>791</td>\n",
" <td>792</td>\n",
" <td>793</td>\n",
" <td>794</td>\n",
" <td>795</td>\n",
" <td>796</td>\n",
" <td>797</td>\n",
" <td>798</td>\n",
" <td>799</td>\n",
" </tr>\n",
" <tr>\n",
" <th>40</th>\n",
" <td>800</td>\n",
" <td>801</td>\n",
" <td>802</td>\n",
" <td>803</td>\n",
" <td>804</td>\n",
" <td>805</td>\n",
" <td>806</td>\n",
" <td>807</td>\n",
" <td>808</td>\n",
" <td>809</td>\n",
" <td>810</td>\n",
" <td>811</td>\n",
" <td>812</td>\n",
" <td>813</td>\n",
" <td>814</td>\n",
" <td>815</td>\n",
" <td>816</td>\n",
" <td>817</td>\n",
" <td>818</td>\n",
" <td>819</td>\n",
" </tr>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>820</td>\n",
" <td>821</td>\n",
" <td>822</td>\n",
" <td>823</td>\n",
" <td>824</td>\n",
" <td>825</td>\n",
" <td>826</td>\n",
" <td>827</td>\n",
" <td>828</td>\n",
" <td>829</td>\n",
" <td>830</td>\n",
" <td>831</td>\n",
" <td>832</td>\n",
" <td>833</td>\n",
" <td>834</td>\n",
" <td>835</td>\n",
" <td>836</td>\n",
" <td>837</td>\n",
" <td>838</td>\n",
" <td>839</td>\n",
" </tr>\n",
" <tr>\n",
" <th>42</th>\n",
" <td>840</td>\n",
" <td>841</td>\n",
" <td>842</td>\n",
" <td>843</td>\n",
" <td>844</td>\n",
" <td>845</td>\n",
" <td>846</td>\n",
" <td>847</td>\n",
" <td>848</td>\n",
" <td>849</td>\n",
" <td>850</td>\n",
" <td>851</td>\n",
" <td>852</td>\n",
" <td>853</td>\n",
" <td>854</td>\n",
" <td>855</td>\n",
" <td>856</td>\n",
" <td>857</td>\n",
" <td>858</td>\n",
" <td>859</td>\n",
" </tr>\n",
" <tr>\n",
" <th>43</th>\n",
" <td>860</td>\n",
" <td>861</td>\n",
" <td>862</td>\n",
" <td>863</td>\n",
" <td>864</td>\n",
" <td>865</td>\n",
" <td>866</td>\n",
" <td>867</td>\n",
" <td>868</td>\n",
" <td>869</td>\n",
" <td>870</td>\n",
" <td>871</td>\n",
" <td>872</td>\n",
" <td>873</td>\n",
" <td>874</td>\n",
" <td>875</td>\n",
" <td>876</td>\n",
" <td>877</td>\n",
" <td>878</td>\n",
" <td>879</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44</th>\n",
" <td>880</td>\n",
" <td>881</td>\n",
" <td>882</td>\n",
" <td>883</td>\n",
" <td>884</td>\n",
" <td>885</td>\n",
" <td>886</td>\n",
" <td>887</td>\n",
" <td>888</td>\n",
" <td>889</td>\n",
" <td>890</td>\n",
" <td>891</td>\n",
" <td>892</td>\n",
" <td>893</td>\n",
" <td>894</td>\n",
" <td>895</td>\n",
" <td>896</td>\n",
" <td>897</td>\n",
" <td>898</td>\n",
" <td>899</td>\n",
" </tr>\n",
" <tr>\n",
" <th>45</th>\n",
" <td>900</td>\n",
" <td>901</td>\n",
" <td>902</td>\n",
" <td>903</td>\n",
" <td>904</td>\n",
" <td>905</td>\n",
" <td>906</td>\n",
" <td>907</td>\n",
" <td>908</td>\n",
" <td>909</td>\n",
" <td>910</td>\n",
" <td>911</td>\n",
" <td>912</td>\n",
" <td>913</td>\n",
" <td>914</td>\n",
" <td>915</td>\n",
" <td>916</td>\n",
" <td>917</td>\n",
" <td>918</td>\n",
" <td>919</td>\n",
" </tr>\n",
" <tr>\n",
" <th>46</th>\n",
" <td>920</td>\n",
" <td>921</td>\n",
" <td>922</td>\n",
" <td>923</td>\n",
" <td>924</td>\n",
" <td>925</td>\n",
" <td>926</td>\n",
" <td>927</td>\n",
" <td>928</td>\n",
" <td>929</td>\n",
" <td>930</td>\n",
" <td>931</td>\n",
" <td>932</td>\n",
" <td>933</td>\n",
" <td>934</td>\n",
" <td>935</td>\n",
" <td>936</td>\n",
" <td>937</td>\n",
" <td>938</td>\n",
" <td>939</td>\n",
" </tr>\n",
" <tr>\n",
" <th>47</th>\n",
" <td>940</td>\n",
" <td>941</td>\n",
" <td>942</td>\n",
" <td>943</td>\n",
" <td>944</td>\n",
" <td>945</td>\n",
" <td>946</td>\n",
" <td>947</td>\n",
" <td>948</td>\n",
" <td>949</td>\n",
" <td>950</td>\n",
" <td>951</td>\n",
" <td>952</td>\n",
" <td>953</td>\n",
" <td>954</td>\n",
" <td>955</td>\n",
" <td>956</td>\n",
" <td>957</td>\n",
" <td>958</td>\n",
" <td>959</td>\n",
" </tr>\n",
" <tr>\n",
" <th>48</th>\n",
" <td>960</td>\n",
" <td>961</td>\n",
" <td>962</td>\n",
" <td>963</td>\n",
" <td>964</td>\n",
" <td>965</td>\n",
" <td>966</td>\n",
" <td>967</td>\n",
" <td>968</td>\n",
" <td>969</td>\n",
" <td>970</td>\n",
" <td>971</td>\n",
" <td>972</td>\n",
" <td>973</td>\n",
" <td>974</td>\n",
" <td>975</td>\n",
" <td>976</td>\n",
" <td>977</td>\n",
" <td>978</td>\n",
" <td>979</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49</th>\n",
" <td>980</td>\n",
" <td>981</td>\n",
" <td>982</td>\n",
" <td>983</td>\n",
" <td>984</td>\n",
" <td>985</td>\n",
" <td>986</td>\n",
" <td>987</td>\n",
" <td>988</td>\n",
" <td>989</td>\n",
" <td>990</td>\n",
" <td>991</td>\n",
" <td>992</td>\n",
" <td>993</td>\n",
" <td>994</td>\n",
" <td>995</td>\n",
" <td>996</td>\n",
" <td>997</td>\n",
" <td>998</td>\n",
" <td>999</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 \\\n",
"0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 \n",
"1 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 \n",
"2 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 \n",
"3 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 \n",
"4 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 \n",
"5 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 \n",
"6 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 \n",
"7 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 \n",
"8 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 \n",
"9 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 \n",
"10 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 \n",
"11 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 \n",
"12 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 \n",
"13 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 \n",
"14 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 \n",
"15 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 \n",
"16 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 \n",
"17 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 \n",
"18 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 \n",
"19 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 \n",
"20 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 \n",
"21 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 \n",
"22 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 \n",
"23 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 \n",
"24 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 \n",
"25 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 \n",
"26 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 \n",
"27 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 \n",
"28 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 \n",
"29 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 \n",
"30 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 \n",
"31 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 \n",
"32 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 \n",
"33 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 \n",
"34 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 \n",
"35 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 \n",
"36 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 \n",
"37 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 \n",
"38 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 \n",
"39 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 \n",
"40 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 \n",
"41 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 \n",
"42 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 \n",
"43 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 \n",
"44 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 \n",
"45 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 \n",
"46 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 \n",
"47 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 \n",
"48 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 \n",
"49 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 \n",
"\n",
" 15 16 17 18 19 \n",
"0 15 16 17 18 19 \n",
"1 35 36 37 38 39 \n",
"2 55 56 57 58 59 \n",
"3 75 76 77 78 79 \n",
"4 95 96 97 98 99 \n",
"5 115 116 117 118 119 \n",
"6 135 136 137 138 139 \n",
"7 155 156 157 158 159 \n",
"8 175 176 177 178 179 \n",
"9 195 196 197 198 199 \n",
"10 215 216 217 218 219 \n",
"11 235 236 237 238 239 \n",
"12 255 256 257 258 259 \n",
"13 275 276 277 278 279 \n",
"14 295 296 297 298 299 \n",
"15 315 316 317 318 319 \n",
"16 335 336 337 338 339 \n",
"17 355 356 357 358 359 \n",
"18 375 376 377 378 379 \n",
"19 395 396 397 398 399 \n",
"20 415 416 417 418 419 \n",
"21 435 436 437 438 439 \n",
"22 455 456 457 458 459 \n",
"23 475 476 477 478 479 \n",
"24 495 496 497 498 499 \n",
"25 515 516 517 518 519 \n",
"26 535 536 537 538 539 \n",
"27 555 556 557 558 559 \n",
"28 575 576 577 578 579 \n",
"29 595 596 597 598 599 \n",
"30 615 616 617 618 619 \n",
"31 635 636 637 638 639 \n",
"32 655 656 657 658 659 \n",
"33 675 676 677 678 679 \n",
"34 695 696 697 698 699 \n",
"35 715 716 717 718 719 \n",
"36 735 736 737 738 739 \n",
"37 755 756 757 758 759 \n",
"38 775 776 777 778 779 \n",
"39 795 796 797 798 799 \n",
"40 815 816 817 818 819 \n",
"41 835 836 837 838 839 \n",
"42 855 856 857 858 859 \n",
"43 875 876 877 878 879 \n",
"44 895 896 897 898 899 \n",
"45 915 916 917 918 919 \n",
"46 935 936 937 938 939 \n",
"47 955 956 957 958 959 \n",
"48 975 976 977 978 979 \n",
"49 995 996 997 998 999 "
]
},
"execution_count": 68,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.DataFrame(np.arange(1000).reshape(50, -1))\n",
"df.to_csv(\"ex7.csv\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 8\n",
"There is a dataset `data/yob2012.txt` which lists the number of newborns registered in 2018 with their names and sex. Open the dataset in pandas **as a csv**, explore it and derive the ratio between male and female newborns.\n",
"\n",
"*Note: The file doesn't contain a header so you will need to add your own column names with*\n",
"```python\n",
"pd.read_csv(\"...\", names=[\"Some\", \"Fun\", \"Columns\"]\n",
"```\n"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1.3694428812605515"
]
},
"execution_count": 75,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_csv(\"data/yob2012.txt\", names=[\"Name\", \"Sex\", \"idk\"])\n",
"sexes = df.Sex.value_counts()\n",
"\n",
"sexes[0] / sexes[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Web scraping <a name=\"web\"></a>\n",
"It is also very easy to scrape webpages and extract tables from them.\n",
"\n",
"For example, let's consider extracting the table of failed American banks."
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"url = \"https://www.fdic.gov/bank/individual/failed/banklist.html\"\n",
"banks = pd.read_html(url)\n",
"banks = banks[0]"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Bank Name</th>\n",
" <th>City</th>\n",
" <th>ST</th>\n",
" <th>CERT</th>\n",
" <th>Acquiring Institution</th>\n",
" <th>Closing Date</th>\n",
" <th>Updated Date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Washington Federal Bank for Savings</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>30570</td>\n",
" <td>Royal Savings Bank</td>\n",
" <td>December 15, 2017</td>\n",
" <td>February 21, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>The Farmers and Merchants State Bank of Argonia</td>\n",
" <td>Argonia</td>\n",
" <td>KS</td>\n",
" <td>17719</td>\n",
" <td>Conway Bank</td>\n",
" <td>October 13, 2017</td>\n",
" <td>February 21, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Fayette County Bank</td>\n",
" <td>Saint Elmo</td>\n",
" <td>IL</td>\n",
" <td>1802</td>\n",
" <td>United Fidelity Bank, fsb</td>\n",
" <td>May 26, 2017</td>\n",
" <td>July 26, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Guaranty Bank, (d/b/a BestBank in Georgia & Mi...</td>\n",
" <td>Milwaukee</td>\n",
" <td>WI</td>\n",
" <td>30003</td>\n",
" <td>First-Citizens Bank & Trust Company</td>\n",
" <td>May 5, 2017</td>\n",
" <td>March 22, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>First NBC Bank</td>\n",
" <td>New Orleans</td>\n",
" <td>LA</td>\n",
" <td>58302</td>\n",
" <td>Whitney Bank</td>\n",
" <td>April 28, 2017</td>\n",
" <td>December 5, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Proficio Bank</td>\n",
" <td>Cottonwood Heights</td>\n",
" <td>UT</td>\n",
" <td>35495</td>\n",
" <td>Cache Valley Bank</td>\n",
" <td>March 3, 2017</td>\n",
" <td>March 7, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Seaway Bank and Trust Company</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>19328</td>\n",
" <td>State Bank of Texas</td>\n",
" <td>January 27, 2017</td>\n",
" <td>May 18, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Harvest Community Bank</td>\n",
" <td>Pennsville</td>\n",
" <td>NJ</td>\n",
" <td>34951</td>\n",
" <td>First-Citizens Bank & Trust Company</td>\n",
" <td>January 13, 2017</td>\n",
" <td>May 18, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Allied Bank</td>\n",
" <td>Mulberry</td>\n",
" <td>AR</td>\n",
" <td>91</td>\n",
" <td>Today's Bank</td>\n",
" <td>September 23, 2016</td>\n",
" <td>September 25, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>The Woodbury Banking Company</td>\n",
" <td>Woodbury</td>\n",
" <td>GA</td>\n",
" <td>11297</td>\n",
" <td>United Bank</td>\n",
" <td>August 19, 2016</td>\n",
" <td>December 13, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>First CornerStone Bank</td>\n",
" <td>King of Prussia</td>\n",
" <td>PA</td>\n",
" <td>35312</td>\n",
" <td>First-Citizens Bank & Trust Company</td>\n",
" <td>May 6, 2016</td>\n",
" <td>November 13, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Trust Company Bank</td>\n",
" <td>Memphis</td>\n",
" <td>TN</td>\n",
" <td>9956</td>\n",
" <td>The Bank of Fayette County</td>\n",
" <td>April 29, 2016</td>\n",
" <td>September 14, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>North Milwaukee State Bank</td>\n",
" <td>Milwaukee</td>\n",
" <td>WI</td>\n",
" <td>20364</td>\n",
" <td>First-Citizens Bank & Trust Company</td>\n",
" <td>March 11, 2016</td>\n",
" <td>March 13, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Hometown National Bank</td>\n",
" <td>Longview</td>\n",
" <td>WA</td>\n",
" <td>35156</td>\n",
" <td>Twin City Bank</td>\n",
" <td>October 2, 2015</td>\n",
" <td>February 19, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>The Bank of Georgia</td>\n",
" <td>Peachtree City</td>\n",
" <td>GA</td>\n",
" <td>35259</td>\n",
" <td>Fidelity Bank</td>\n",
" <td>October 2, 2015</td>\n",
" <td>July 9, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Premier Bank</td>\n",
" <td>Denver</td>\n",
" <td>CO</td>\n",
" <td>34112</td>\n",
" <td>United Fidelity Bank, fsb</td>\n",
" <td>July 10, 2015</td>\n",
" <td>February 20, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Edgebrook Bank</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>57772</td>\n",
" <td>Republic Bank of Chicago</td>\n",
" <td>May 8, 2015</td>\n",
" <td>July 12, 2016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Doral Bank En Espanol</td>\n",
" <td>San Juan</td>\n",
" <td>PR</td>\n",
" <td>32102</td>\n",
" <td>Banco Popular de Puerto Rico</td>\n",
" <td>February 27, 2015</td>\n",
" <td>May 13, 2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Capitol City Bank & Trust Company</td>\n",
" <td>Atlanta</td>\n",
" <td>GA</td>\n",
" <td>33938</td>\n",
" <td>First-Citizens Bank & Trust Company</td>\n",
" <td>February 13, 2015</td>\n",
" <td>April 21, 2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Highland Community Bank</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>20290</td>\n",
" <td>United Fidelity Bank, fsb</td>\n",
" <td>January 23, 2015</td>\n",
" <td>November 15, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>First National Bank of Crestview</td>\n",
" <td>Crestview</td>\n",
" <td>FL</td>\n",
" <td>17557</td>\n",
" <td>First NBC Bank</td>\n",
" <td>January 16, 2015</td>\n",
" <td>November 15, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Northern Star Bank</td>\n",
" <td>Mankato</td>\n",
" <td>MN</td>\n",
" <td>34983</td>\n",
" <td>BankVista</td>\n",
" <td>December 19, 2014</td>\n",
" <td>January 3, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>Frontier Bank, FSB D/B/A El Paseo Bank</td>\n",
" <td>Palm Desert</td>\n",
" <td>CA</td>\n",
" <td>34738</td>\n",
" <td>Bank of Southern California, N.A.</td>\n",
" <td>November 7, 2014</td>\n",
" <td>November 10, 2016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>The National Republic Bank of Chicago</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>916</td>\n",
" <td>State Bank of Texas</td>\n",
" <td>October 24, 2014</td>\n",
" <td>January 6, 2016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>NBRS Financial</td>\n",
" <td>Rising Sun</td>\n",
" <td>MD</td>\n",
" <td>4862</td>\n",
" <td>Howard Bank</td>\n",
" <td>October 17, 2014</td>\n",
" <td>February 19, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>GreenChoice Bank, fsb</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>28462</td>\n",
" <td>Providence Bank, LLC</td>\n",
" <td>July 25, 2014</td>\n",
" <td>December 12, 2016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>Eastside Commercial Bank</td>\n",
" <td>Conyers</td>\n",
" <td>GA</td>\n",
" <td>58125</td>\n",
" <td>Community & Southern Bank</td>\n",
" <td>July 18, 2014</td>\n",
" <td>October 6, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>The Freedom State Bank</td>\n",
" <td>Freedom</td>\n",
" <td>OK</td>\n",
" <td>12483</td>\n",
" <td>Alva State Bank & Trust Company</td>\n",
" <td>June 27, 2014</td>\n",
" <td>February 21, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>Valley Bank</td>\n",
" <td>Fort Lauderdale</td>\n",
" <td>FL</td>\n",
" <td>21793</td>\n",
" <td>Landmark Bank, National Association</td>\n",
" <td>June 20, 2014</td>\n",
" <td>February 14, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>Valley Bank</td>\n",
" <td>Moline</td>\n",
" <td>IL</td>\n",
" <td>10450</td>\n",
" <td>Great Southern Bank</td>\n",
" <td>June 20, 2014</td>\n",
" <td>June 26, 2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>525</th>\n",
" <td>ANB Financial, NA</td>\n",
" <td>Bentonville</td>\n",
" <td>AR</td>\n",
" <td>33901</td>\n",
" <td>Pulaski Bank and Trust Company</td>\n",
" <td>May 9, 2008</td>\n",
" <td>August 28, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>526</th>\n",
" <td>Hume Bank</td>\n",
" <td>Hume</td>\n",
" <td>MO</td>\n",
" <td>1971</td>\n",
" <td>Security Bank</td>\n",
" <td>March 7, 2008</td>\n",
" <td>August 28, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>527</th>\n",
" <td>Douglass National Bank</td>\n",
" <td>Kansas City</td>\n",
" <td>MO</td>\n",
" <td>24660</td>\n",
" <td>Liberty Bank and Trust Company</td>\n",
" <td>January 25, 2008</td>\n",
" <td>October 26, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>528</th>\n",
" <td>Miami Valley Bank</td>\n",
" <td>Lakeview</td>\n",
" <td>OH</td>\n",
" <td>16848</td>\n",
" <td>The Citizens Banking Company</td>\n",
" <td>October 4, 2007</td>\n",
" <td>September 12, 2016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>529</th>\n",
" <td>NetBank</td>\n",
" <td>Alpharetta</td>\n",
" <td>GA</td>\n",
" <td>32575</td>\n",
" <td>ING DIRECT</td>\n",
" <td>September 28, 2007</td>\n",
" <td>August 28, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>530</th>\n",
" <td>Metropolitan Savings Bank</td>\n",
" <td>Pittsburgh</td>\n",
" <td>PA</td>\n",
" <td>35353</td>\n",
" <td>Allegheny Valley Bank of Pittsburgh</td>\n",
" <td>February 2, 2007</td>\n",
" <td>October 27, 2010</td>\n",
" </tr>\n",
" <tr>\n",
" <th>531</th>\n",
" <td>Bank of Ephraim</td>\n",
" <td>Ephraim</td>\n",
" <td>UT</td>\n",
" <td>1249</td>\n",
" <td>Far West Bank</td>\n",
" <td>June 25, 2004</td>\n",
" <td>April 9, 2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>532</th>\n",
" <td>Reliance Bank</td>\n",
" <td>White Plains</td>\n",
" <td>NY</td>\n",
" <td>26778</td>\n",
" <td>Union State Bank</td>\n",
" <td>March 19, 2004</td>\n",
" <td>April 9, 2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>533</th>\n",
" <td>Guaranty National Bank of Tallahassee</td>\n",
" <td>Tallahassee</td>\n",
" <td>FL</td>\n",
" <td>26838</td>\n",
" <td>Hancock Bank of Florida</td>\n",
" <td>March 12, 2004</td>\n",
" <td>April 17, 2018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>534</th>\n",
" <td>Dollar Savings Bank</td>\n",
" <td>Newark</td>\n",
" <td>NJ</td>\n",
" <td>31330</td>\n",
" <td>No Acquirer</td>\n",
" <td>February 14, 2004</td>\n",
" <td>April 9, 2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>535</th>\n",
" <td>Pulaski Savings Bank</td>\n",
" <td>Philadelphia</td>\n",
" <td>PA</td>\n",
" <td>27203</td>\n",
" <td>Earthstar Bank</td>\n",
" <td>November 14, 2003</td>\n",
" <td>October 6, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>536</th>\n",
" <td>First National Bank of Blanchardville</td>\n",
" <td>Blanchardville</td>\n",
" <td>WI</td>\n",
" <td>11639</td>\n",
" <td>The Park Bank</td>\n",
" <td>May 9, 2003</td>\n",
" <td>June 5, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>537</th>\n",
" <td>Southern Pacific Bank</td>\n",
" <td>Torrance</td>\n",
" <td>CA</td>\n",
" <td>27094</td>\n",
" <td>Beal Bank</td>\n",
" <td>February 7, 2003</td>\n",
" <td>October 20, 2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>538</th>\n",
" <td>Farmers Bank of Cheneyville</td>\n",
" <td>Cheneyville</td>\n",
" <td>LA</td>\n",
" <td>16445</td>\n",
" <td>Sabine State Bank & Trust</td>\n",
" <td>December 17, 2002</td>\n",
" <td>October 20, 2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>539</th>\n",
" <td>Bank of Alamo</td>\n",
" <td>Alamo</td>\n",
" <td>TN</td>\n",
" <td>9961</td>\n",
" <td>No Acquirer</td>\n",
" <td>November 8, 2002</td>\n",
" <td>March 18, 2005</td>\n",
" </tr>\n",
" <tr>\n",
" <th>540</th>\n",
" <td>AmTrade International Bank En Espanol</td>\n",
" <td>Atlanta</td>\n",
" <td>GA</td>\n",
" <td>33784</td>\n",
" <td>No Acquirer</td>\n",
" <td>September 30, 2002</td>\n",
" <td>September 11, 2006</td>\n",
" </tr>\n",
" <tr>\n",
" <th>541</th>\n",
" <td>Universal Federal Savings Bank</td>\n",
" <td>Chicago</td>\n",
" <td>IL</td>\n",
" <td>29355</td>\n",
" <td>Chicago Community Bank</td>\n",
" <td>June 27, 2002</td>\n",
" <td>October 6, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>542</th>\n",
" <td>Connecticut Bank of Commerce</td>\n",
" <td>Stamford</td>\n",
" <td>CT</td>\n",
" <td>19183</td>\n",
" <td>Hudson United Bank</td>\n",
" <td>June 26, 2002</td>\n",
" <td>February 14, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>543</th>\n",
" <td>New Century Bank</td>\n",
" <td>Shelby Township</td>\n",
" <td>MI</td>\n",
" <td>34979</td>\n",
" <td>No Acquirer</td>\n",
" <td>March 28, 2002</td>\n",
" <td>March 18, 2005</td>\n",
" </tr>\n",
" <tr>\n",
" <th>544</th>\n",
" <td>Net 1st National Bank</td>\n",
" <td>Boca Raton</td>\n",
" <td>FL</td>\n",
" <td>26652</td>\n",
" <td>Bank Leumi USA</td>\n",
" <td>March 1, 2002</td>\n",
" <td>April 9, 2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>545</th>\n",
" <td>NextBank, NA</td>\n",
" <td>Phoenix</td>\n",
" <td>AZ</td>\n",
" <td>22314</td>\n",
" <td>No Acquirer</td>\n",
" <td>February 7, 2002</td>\n",
" <td>February 5, 2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>546</th>\n",
" <td>Oakwood Deposit Bank Co.</td>\n",
" <td>Oakwood</td>\n",
" <td>OH</td>\n",
" <td>8966</td>\n",
" <td>The State Bank & Trust Company</td>\n",
" <td>February 1, 2002</td>\n",
" <td>October 25, 2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>547</th>\n",
" <td>Bank of Sierra Blanca</td>\n",
" <td>Sierra Blanca</td>\n",
" <td>TX</td>\n",
" <td>22002</td>\n",
" <td>The Security State Bank of Pecos</td>\n",
" <td>January 18, 2002</td>\n",
" <td>November 6, 2003</td>\n",
" </tr>\n",
" <tr>\n",
" <th>548</th>\n",
" <td>Hamilton Bank, NA En Espanol</td>\n",
" <td>Miami</td>\n",
" <td>FL</td>\n",
" <td>24382</td>\n",
" <td>Israel Discount Bank of New York</td>\n",
" <td>January 11, 2002</td>\n",
" <td>September 21, 2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>549</th>\n",
" <td>Sinclair National Bank</td>\n",
" <td>Gravette</td>\n",
" <td>AR</td>\n",
" <td>34248</td>\n",
" <td>Delta Trust & Bank</td>\n",
" <td>September 7, 2001</td>\n",
" <td>October 6, 2017</td>\n",
" </tr>\n",
" <tr>\n",
" <th>550</th>\n",
" <td>Superior Bank, FSB</td>\n",
" <td>Hinsdale</td>\n",
" <td>IL</td>\n",
" <td>32646</td>\n",
" <td>Superior Federal, FSB</td>\n",
" <td>July 27, 2001</td>\n",
" <td>August 19, 2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>551</th>\n",
" <td>Malta National Bank</td>\n",
" <td>Malta</td>\n",
" <td>OH</td>\n",
" <td>6629</td>\n",
" <td>North Valley Bank</td>\n",
" <td>May 3, 2001</td>\n",
" <td>November 18, 2002</td>\n",
" </tr>\n",
" <tr>\n",
" <th>552</th>\n",
" <td>First Alliance Bank & Trust Co.</td>\n",
" <td>Manchester</td>\n",
" <td>NH</td>\n",
" <td>34264</td>\n",
" <td>Southern New Hampshire Bank & Trust</td>\n",
" <td>February 2, 2001</td>\n",
" <td>February 18, 2003</td>\n",
" </tr>\n",
" <tr>\n",
" <th>553</th>\n",
" <td>National State Bank of Metropolis</td>\n",
" <td>Metropolis</td>\n",
" <td>IL</td>\n",
" <td>3815</td>\n",
" <td>Banterra Bank of Marion</td>\n",
" <td>December 14, 2000</td>\n",
" <td>March 17, 2005</td>\n",
" </tr>\n",
" <tr>\n",
" <th>554</th>\n",
" <td>Bank of Honolulu</td>\n",
" <td>Honolulu</td>\n",
" <td>HI</td>\n",
" <td>21029</td>\n",
" <td>Bank of the Orient</td>\n",
" <td>October 13, 2000</td>\n",
" <td>March 17, 2005</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>555 rows × 7 columns</p>\n",
"</div>"
],
"text/plain": [
" Bank Name City \\\n",
"0 Washington Federal Bank for Savings Chicago \n",
"1 The Farmers and Merchants State Bank of Argonia Argonia \n",
"2 Fayette County Bank Saint Elmo \n",
"3 Guaranty Bank, (d/b/a BestBank in Georgia & Mi... Milwaukee \n",
"4 First NBC Bank New Orleans \n",
"5 Proficio Bank Cottonwood Heights \n",
"6 Seaway Bank and Trust Company Chicago \n",
"7 Harvest Community Bank Pennsville \n",
"8 Allied Bank Mulberry \n",
"9 The Woodbury Banking Company Woodbury \n",
"10 First CornerStone Bank King of Prussia \n",
"11 Trust Company Bank Memphis \n",
"12 North Milwaukee State Bank Milwaukee \n",
"13 Hometown National Bank Longview \n",
"14 The Bank of Georgia Peachtree City \n",
"15 Premier Bank Denver \n",
"16 Edgebrook Bank Chicago \n",
"17 Doral Bank En Espanol San Juan \n",
"18 Capitol City Bank & Trust Company Atlanta \n",
"19 Highland Community Bank Chicago \n",
"20 First National Bank of Crestview Crestview \n",
"21 Northern Star Bank Mankato \n",
"22 Frontier Bank, FSB D/B/A El Paseo Bank Palm Desert \n",
"23 The National Republic Bank of Chicago Chicago \n",
"24 NBRS Financial Rising Sun \n",
"25 GreenChoice Bank, fsb Chicago \n",
"26 Eastside Commercial Bank Conyers \n",
"27 The Freedom State Bank Freedom \n",
"28 Valley Bank Fort Lauderdale \n",
"29 Valley Bank Moline \n",
".. ... ... \n",
"525 ANB Financial, NA Bentonville \n",
"526 Hume Bank Hume \n",
"527 Douglass National Bank Kansas City \n",
"528 Miami Valley Bank Lakeview \n",
"529 NetBank Alpharetta \n",
"530 Metropolitan Savings Bank Pittsburgh \n",
"531 Bank of Ephraim Ephraim \n",
"532 Reliance Bank White Plains \n",
"533 Guaranty National Bank of Tallahassee Tallahassee \n",
"534 Dollar Savings Bank Newark \n",
"535 Pulaski Savings Bank Philadelphia \n",
"536 First National Bank of Blanchardville Blanchardville \n",
"537 Southern Pacific Bank Torrance \n",
"538 Farmers Bank of Cheneyville Cheneyville \n",
"539 Bank of Alamo Alamo \n",
"540 AmTrade International Bank En Espanol Atlanta \n",
"541 Universal Federal Savings Bank Chicago \n",
"542 Connecticut Bank of Commerce Stamford \n",
"543 New Century Bank Shelby Township \n",
"544 Net 1st National Bank Boca Raton \n",
"545 NextBank, NA Phoenix \n",
"546 Oakwood Deposit Bank Co. Oakwood \n",
"547 Bank of Sierra Blanca Sierra Blanca \n",
"548 Hamilton Bank, NA En Espanol Miami \n",
"549 Sinclair National Bank Gravette \n",
"550 Superior Bank, FSB Hinsdale \n",
"551 Malta National Bank Malta \n",
"552 First Alliance Bank & Trust Co. Manchester \n",
"553 National State Bank of Metropolis Metropolis \n",
"554 Bank of Honolulu Honolulu \n",
"\n",
" ST CERT Acquiring Institution Closing Date \\\n",
"0 IL 30570 Royal Savings Bank December 15, 2017 \n",
"1 KS 17719 Conway Bank October 13, 2017 \n",
"2 IL 1802 United Fidelity Bank, fsb May 26, 2017 \n",
"3 WI 30003 First-Citizens Bank & Trust Company May 5, 2017 \n",
"4 LA 58302 Whitney Bank April 28, 2017 \n",
"5 UT 35495 Cache Valley Bank March 3, 2017 \n",
"6 IL 19328 State Bank of Texas January 27, 2017 \n",
"7 NJ 34951 First-Citizens Bank & Trust Company January 13, 2017 \n",
"8 AR 91 Today's Bank September 23, 2016 \n",
"9 GA 11297 United Bank August 19, 2016 \n",
"10 PA 35312 First-Citizens Bank & Trust Company May 6, 2016 \n",
"11 TN 9956 The Bank of Fayette County April 29, 2016 \n",
"12 WI 20364 First-Citizens Bank & Trust Company March 11, 2016 \n",
"13 WA 35156 Twin City Bank October 2, 2015 \n",
"14 GA 35259 Fidelity Bank October 2, 2015 \n",
"15 CO 34112 United Fidelity Bank, fsb July 10, 2015 \n",
"16 IL 57772 Republic Bank of Chicago May 8, 2015 \n",
"17 PR 32102 Banco Popular de Puerto Rico February 27, 2015 \n",
"18 GA 33938 First-Citizens Bank & Trust Company February 13, 2015 \n",
"19 IL 20290 United Fidelity Bank, fsb January 23, 2015 \n",
"20 FL 17557 First NBC Bank January 16, 2015 \n",
"21 MN 34983 BankVista December 19, 2014 \n",
"22 CA 34738 Bank of Southern California, N.A. November 7, 2014 \n",
"23 IL 916 State Bank of Texas October 24, 2014 \n",
"24 MD 4862 Howard Bank October 17, 2014 \n",
"25 IL 28462 Providence Bank, LLC July 25, 2014 \n",
"26 GA 58125 Community & Southern Bank July 18, 2014 \n",
"27 OK 12483 Alva State Bank & Trust Company June 27, 2014 \n",
"28 FL 21793 Landmark Bank, National Association June 20, 2014 \n",
"29 IL 10450 Great Southern Bank June 20, 2014 \n",
".. .. ... ... ... \n",
"525 AR 33901 Pulaski Bank and Trust Company May 9, 2008 \n",
"526 MO 1971 Security Bank March 7, 2008 \n",
"527 MO 24660 Liberty Bank and Trust Company January 25, 2008 \n",
"528 OH 16848 The Citizens Banking Company October 4, 2007 \n",
"529 GA 32575 ING DIRECT September 28, 2007 \n",
"530 PA 35353 Allegheny Valley Bank of Pittsburgh February 2, 2007 \n",
"531 UT 1249 Far West Bank June 25, 2004 \n",
"532 NY 26778 Union State Bank March 19, 2004 \n",
"533 FL 26838 Hancock Bank of Florida March 12, 2004 \n",
"534 NJ 31330 No Acquirer February 14, 2004 \n",
"535 PA 27203 Earthstar Bank November 14, 2003 \n",
"536 WI 11639 The Park Bank May 9, 2003 \n",
"537 CA 27094 Beal Bank February 7, 2003 \n",
"538 LA 16445 Sabine State Bank & Trust December 17, 2002 \n",
"539 TN 9961 No Acquirer November 8, 2002 \n",
"540 GA 33784 No Acquirer September 30, 2002 \n",
"541 IL 29355 Chicago Community Bank June 27, 2002 \n",
"542 CT 19183 Hudson United Bank June 26, 2002 \n",
"543 MI 34979 No Acquirer March 28, 2002 \n",
"544 FL 26652 Bank Leumi USA March 1, 2002 \n",
"545 AZ 22314 No Acquirer February 7, 2002 \n",
"546 OH 8966 The State Bank & Trust Company February 1, 2002 \n",
"547 TX 22002 The Security State Bank of Pecos January 18, 2002 \n",
"548 FL 24382 Israel Discount Bank of New York January 11, 2002 \n",
"549 AR 34248 Delta Trust & Bank September 7, 2001 \n",
"550 IL 32646 Superior Federal, FSB July 27, 2001 \n",
"551 OH 6629 North Valley Bank May 3, 2001 \n",
"552 NH 34264 Southern New Hampshire Bank & Trust February 2, 2001 \n",
"553 IL 3815 Banterra Bank of Marion December 14, 2000 \n",
"554 HI 21029 Bank of the Orient October 13, 2000 \n",
"\n",
" Updated Date \n",
"0 February 21, 2018 \n",
"1 February 21, 2018 \n",
"2 July 26, 2017 \n",
"3 March 22, 2018 \n",
"4 December 5, 2017 \n",
"5 March 7, 2018 \n",
"6 May 18, 2017 \n",
"7 May 18, 2017 \n",
"8 September 25, 2017 \n",
"9 December 13, 2018 \n",
"10 November 13, 2018 \n",
"11 September 14, 2018 \n",
"12 March 13, 2017 \n",
"13 February 19, 2018 \n",
"14 July 9, 2018 \n",
"15 February 20, 2018 \n",
"16 July 12, 2016 \n",
"17 May 13, 2015 \n",
"18 April 21, 2015 \n",
"19 November 15, 2017 \n",
"20 November 15, 2017 \n",
"21 January 3, 2018 \n",
"22 November 10, 2016 \n",
"23 January 6, 2016 \n",
"24 February 19, 2018 \n",
"25 December 12, 2016 \n",
"26 October 6, 2017 \n",
"27 February 21, 2018 \n",
"28 February 14, 2018 \n",
"29 June 26, 2015 \n",
".. ... \n",
"525 August 28, 2012 \n",
"526 August 28, 2012 \n",
"527 October 26, 2012 \n",
"528 September 12, 2016 \n",
"529 August 28, 2012 \n",
"530 October 27, 2010 \n",
"531 April 9, 2008 \n",
"532 April 9, 2008 \n",
"533 April 17, 2018 \n",
"534 April 9, 2008 \n",
"535 October 6, 2017 \n",
"536 June 5, 2012 \n",
"537 October 20, 2008 \n",
"538 October 20, 2004 \n",
"539 March 18, 2005 \n",
"540 September 11, 2006 \n",
"541 October 6, 2017 \n",
"542 February 14, 2012 \n",
"543 March 18, 2005 \n",
"544 April 9, 2008 \n",
"545 February 5, 2015 \n",
"546 October 25, 2012 \n",
"547 November 6, 2003 \n",
"548 September 21, 2015 \n",
"549 October 6, 2017 \n",
"550 August 19, 2014 \n",
"551 November 18, 2002 \n",
"552 February 18, 2003 \n",
"553 March 17, 2005 \n",
"554 March 17, 2005 \n",
"\n",
"[555 rows x 7 columns]"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"banks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Powerful no? Now let's turn that into an exercise.\n",
"\n",
"### Exercise 9\n",
"Given the data you just extracted above, can you analyse how many banks have failed per state?\n",
"\n",
"Georgia (GA) should be the state with the most failed banks!\n",
"\n",
"*Hint: try searching the web for pandas counting occurrences* "
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GA 93\n",
"FL 75\n",
"IL 69\n",
"CA 41\n",
"MN 23\n",
"Name: ST, dtype: int64"
]
},
"execution_count": 81,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"banks.ST.value_counts().head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Cleaning <a name=\"cleaning\"></a>\n",
"While doing data analysis and modeling, a significant amount of time is spent on data preparation: loading, cleaning, transforming and rearranging. Such tasks are often reported to take **up to 80%** or more of a data analyst's time. Often the way the data is stored in files isn't in the correct format and needs to be modified. Researchers usually do this on an ad-hoc basis using programming languages like Python.\n",
"\n",
"In this chapter, we will discuss tools for handling missing data, duplicate data, string manipulation, and some other analytical data transformations.\n",
"\n",
"## Handling missing data <a name=\"missing\"></a>\n",
"Mussing data occurs commonly in many data analysis applications. One of the goals of pandas is to make working with missing data as painless as possible.\n",
"\n",
"In pandas, missing numeric data is represented by `NaN` (Not a Number) and can easily be handled:"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 orange\n",
"1 tomato\n",
"2 NaN\n",
"3 avocado\n",
"dtype: object"
]
},
"execution_count": 82,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"string_data = pd.Series(['orange', 'tomato', np.nan, 'avocado'])\n",
"string_data"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 False\n",
"2 True\n",
"3 False\n",
"dtype: bool"
]
},
"execution_count": 83,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"string_data.isnull()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Furthermore, the pandas `NaN` is functionally equlevant to the standard Python type `NoneType` which can be defined with `x = None`."
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 None\n",
"1 tomato\n",
"2 NaN\n",
"3 avocado\n",
"dtype: object"
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"string_data[0] = None\n",
"string_data"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 True\n",
"1 False\n",
"2 True\n",
"3 False\n",
"dtype: bool"
]
},
"execution_count": 85,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"string_data.isnull()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are some other methods which you can find useful:\n",
" \n",
"| Method | Description |\n",
"| -- | -- |\n",
"| dropna | Filter axis labels based on whether the values of each label have missing data|\n",
"| fillna | Fill in missing data with some value |\n",
"| isnull | Return boolean values indicating which values are missing |\n",
"| notnull | Negation of isnull |\n",
"\n",
"### Exercise 10\n",
"Remove the missing data below using the appropriate method"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 1.0\n",
"2 3.0\n",
"3 4.0\n",
"5 6.0\n",
"dtype: float64"
]
},
"execution_count": 86,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.Series([1, None, 3, 4, None, 6])\n",
"data.dropna()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`dropna()` by default removes any row/column that has a missing value. What if we want to remove only rows in which all of the data is missing though?"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1.0 6.5 3.0\n",
"1 1.0 NaN NaN\n",
"2 NaN NaN NaN\n",
"3 NaN 6.5 3.0"
]
},
"execution_count": 87,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.DataFrame([[1., 6.5, 3.], [1., None, None],\n",
" [None, None, None], [None, 6.5, 3.]])\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1.0 6.5 3.0"
]
},
"execution_count": 88,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.dropna()"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1.0 6.5 3.0\n",
"1 1.0 NaN NaN\n",
"3 NaN 6.5 3.0"
]
},
"execution_count": 89,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.dropna(how=\"all\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 11\n",
"That's fine if we want to remove missing data, what if we want to fill in missing data? Do you know of a way? Try to fill in all of the missing values from the data below with **0s**"
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [],
"source": [
"data = pd.DataFrame([[1., 6.5, 3.], [2., None, None],\n",
" [None, None, None], [None, 1.5, 9.]])\n",
"\n",
"data.fillna()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"pandas also allows us to interpolate the data instead of just filling it with a constant. The easiest way to do that is shown below, but there are more complex ones that are not covered in this course."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.fillna(method=\"ffill\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you want you can explore the other capabilities of [`fillna`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Transformation <a name=\"transformation\"></a>\n",
"### Removing duplicates\n",
"Duplicate data can be a serious issue, luckily pandas offers a simple way to remove duplicates"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pd.DataFrame([1, 2, 3, 4, 3, 2, 1])\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.drop_duplicates()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also select which rows to keep"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.drop_duplicates(keep=\"last\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Replacing data\n",
"You've already seen how you can fill in missing data with `fillna()`. That is actually a special case of more general value replacement. That is done via the `replace()` method.\n",
"\n",
"Let's consider an example where the dataset given to us had `-999` as sentinel values for missing data instead of `NaN`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pd.DataFrame([1., -999., 2., -999., 3., 4., -999, -999, 7.])\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.replace(-999, np.nan)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Detection and Filtering Outliers\n",
"Filtering or transforming outliers is largely a matter of applying array operations. Consider a DataFrame with some normally distributed data:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>1000.000000</td>\n",
" <td>1000.000000</td>\n",
" <td>1000.000000</td>\n",
" <td>1000.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>-0.007632</td>\n",
" <td>0.050024</td>\n",
" <td>-0.012849</td>\n",
" <td>0.016784</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.998612</td>\n",
" <td>1.006627</td>\n",
" <td>0.964622</td>\n",
" <td>1.000706</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>-2.957364</td>\n",
" <td>-3.502528</td>\n",
" <td>-3.227869</td>\n",
" <td>-3.169486</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>-0.665692</td>\n",
" <td>-0.626164</td>\n",
" <td>-0.662087</td>\n",
" <td>-0.670290</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>0.006478</td>\n",
" <td>0.048645</td>\n",
" <td>-0.030726</td>\n",
" <td>0.042760</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>0.669843</td>\n",
" <td>0.707385</td>\n",
" <td>0.645383</td>\n",
" <td>0.703211</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>2.772749</td>\n",
" <td>3.179202</td>\n",
" <td>3.247557</td>\n",
" <td>2.902432</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"count 1000.000000 1000.000000 1000.000000 1000.000000\n",
"mean -0.007632 0.050024 -0.012849 0.016784\n",
"std 0.998612 1.006627 0.964622 1.000706\n",
"min -2.957364 -3.502528 -3.227869 -3.169486\n",
"25% -0.665692 -0.626164 -0.662087 -0.670290\n",
"50% 0.006478 0.048645 -0.030726 0.042760\n",
"75% 0.669843 0.707385 0.645383 0.703211\n",
"max 2.772749 3.179202 3.247557 2.902432"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.DataFrame(np.random.randn(1000, 4))\n",
"data.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Suppose you now want to lower all absolute values exceeding 3 from one of the columns"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"86 3.247557\n",
"211 3.133593\n",
"494 -3.227869\n",
"Name: 2, dtype: float64"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"col = data[2]\n",
"col[np.abs(col) > 3]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>1000.000000</td>\n",
" <td>1000.000000</td>\n",
" <td>1000.000000</td>\n",
" <td>1000.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>-0.007632</td>\n",
" <td>0.050358</td>\n",
" <td>-0.013003</td>\n",
" <td>0.016953</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.998612</td>\n",
" <td>1.004265</td>\n",
" <td>0.962654</td>\n",
" <td>1.000180</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>-2.957364</td>\n",
" <td>-3.000000</td>\n",
" <td>-3.000000</td>\n",
" <td>-3.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>-0.665692</td>\n",
" <td>-0.626164</td>\n",
" <td>-0.662087</td>\n",
" <td>-0.670290</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>0.006478</td>\n",
" <td>0.048645</td>\n",
" <td>-0.030726</td>\n",
" <td>0.042760</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>0.669843</td>\n",
" <td>0.707385</td>\n",
" <td>0.645383</td>\n",
" <td>0.703211</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>2.772749</td>\n",
" <td>3.000000</td>\n",
" <td>3.000000</td>\n",
" <td>2.902432</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"count 1000.000000 1000.000000 1000.000000 1000.000000\n",
"mean -0.007632 0.050358 -0.013003 0.016953\n",
"std 0.998612 1.004265 0.962654 1.000180\n",
"min -2.957364 -3.000000 -3.000000 -3.000000\n",
"25% -0.665692 -0.626164 -0.662087 -0.670290\n",
"50% 0.006478 0.048645 -0.030726 0.042760\n",
"75% 0.669843 0.707385 0.645383 0.703211\n",
"max 2.772749 3.000000 3.000000 2.902432"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data[np.abs(data) > 3] = np.sign(data) * 3\n",
"data.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 12\n",
"Let's load again our file with home prices and filter out homes based on our preference:\n",
"1. Load up the file `data/homes.csv`\n",
"2. The data contains some duplicates. Filter them out.\n",
"3. Let's say that the most we can spend on a house is £150. Keep only houses that have a **sell**ing price less than £150 and remove the rest\n",
"4. Select only houses that have 4 or more bedrooms\n",
"5. Select only houses that have 3 or more baths\n",
"\n",
"You should end up with only 2 houses"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Sell</th>\n",
" <th>List</th>\n",
" <th>Living</th>\n",
" <th>Rooms</th>\n",
" <th>Beds</th>\n",
" <th>Baths</th>\n",
" <th>Age</th>\n",
" <th>Acres</th>\n",
" <th>Taxes</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>142</td>\n",
" <td>160</td>\n",
" <td>28</td>\n",
" <td>10</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>60</td>\n",
" <td>0.28</td>\n",
" <td>3167</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>135</td>\n",
" <td>140</td>\n",
" <td>18</td>\n",
" <td>7</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>9</td>\n",
" <td>0.57</td>\n",
" <td>3028</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Sell List Living Rooms Beds Baths Age Acres Taxes\n",
"0 142 160 28 10 5 3 60 0.28 3167\n",
"6 135 140 18 7 4 3 9 0.57 3028"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.read_csv(\"data/homes.csv\")\n",
"data = data.drop_duplicates()\n",
"data = data[data[\"Sell\"] < 150]\n",
"# data[data[\"Age\"] > 2]\n",
"data = data[data[\"Beds\"] >= 4]\n",
"data[data[\"Baths\"] >= 3]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}