Skip to content
Snippets Groups Projects
python-data-5-pandas.ipynb 184 KiB
Newer Older
ignat's avatar
ignat committed
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 5 - pandas\n",
ignat's avatar
ignat committed
    "[pandas](http://pandas.pydata.org) provides high-level data structures and functions designed to make working with structured or tabular data fast, easy and expressive. The primary objects in pandas that we will be using are the `DataFrame`, a tabular, column-oriented data structure with both row and column labels, and the `Series`, a one-dimensional labeled array object.\n",
    "\n",
ignat's avatar
ignat committed
    "pandas blends the high-performance, array-computing ideas of NumPy with the flexible data manipulation capabilities of spreadsheets and relational databases. It provides sophisticated indexing functionality to make it easy to reshape, slice and perform aggregations.\n",
ignat's avatar
ignat committed
    "\n",
ignat's avatar
ignat committed
    "While pandas adopts many coding idioms from NumPy, the most significant difference is that pandas is designed for working with tabular or heterogeneous data. NumPy, by contrast, is best suited for working with homogeneous numerical array data.\n",
ignat's avatar
ignat committed
    "<br>\n",
    "\n",
    "## Table of Contents:\n",
ignat's avatar
ignat committed
    "- [Data Structures](#structures)\n",
    "    - [Series](#series)\n",
    "    - [DataFrame](#dataframe)\n",
    "- [Essential Functionality](#ess_func)\n",
    "    - [Reindexing](#reindexing)\n",
    "    - [Dropping Entries](#removing)\n",
    "    - [Indexing, Slicing and Filtering](#indexing)\n",
    "    - [Arithmetic Operations](#arithmetic)\n",
    "- [Summarizing and Computing Descriptive Statistics](#sums)\n",
    "- [Loading and storing data](#loading)\n",
    "    - [Text Format](#text) \n",
    "    - [Web Scraping](#web)\n",
    "- [Data Cleaning and preperation](#cleaning)\n",
    "    - [Handling missing data](#missing)\n",
    "    - [Data transformation](#transformation)\n",
    "- [String manipulation](#strings)\n",
    "\n",
    "The common pandas import statment is shown below:"
ignat's avatar
ignat committed
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "# Common pandas import statement\n",
    "import numpy as np\n",
ignat's avatar
ignat committed
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "# Data Structures <a name=\"structures\"></a>\n",
    "## Series <a name=\"series\"></a>\n",
    "A Series is a one-dimensional array-like object containing a sequence of values and an associated array of data labels called its index.\n",
ignat's avatar
ignat committed
    "\n",
    "The easiest way to make a Series is from an array of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.Series([4, 7, -5, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try printing out data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    4\n",
       "1    7\n",
       "2   -5\n",
       "3    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
ignat's avatar
ignat committed
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "The string representation of a Series displayed interactively shows the index on the left and the values on the right. Because we didn't specify an index, the default on is simply integers 0 through N-1.\n",
ignat's avatar
ignat committed
    "\n",
    "You can output only the values of a Series using \n",
    "```python\n",
    "data.values\n",
    "```\n",
ignat's avatar
ignat committed
    "or you can get only the indices using\n",
ignat's avatar
ignat committed
    "```python\n",
    "data.index\n",
    "```\n",
    "Try it out below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4,  7, -5,  3])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values"
   ]
ignat's avatar
ignat committed
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify custom indeces when intialising the Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "data2 = pd.Series([4, 7, -5, 3], index=[\"a\", \"b\", \"c\", \"d\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    4\n",
       "b    7\n",
       "c   -5\n",
       "d    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2"
   ]
ignat's avatar
ignat committed
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use these labels to access the data similar to a normal array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
ignat's avatar
ignat committed
   "source": [
    "data2[\"a\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "Another way to think about Series is as a fixed-length ordered dictionary. Furthermore, you can actually define a Series in a similar manner to a dictionary"
ignat's avatar
ignat committed
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "cities = {\"Glasgow\" : 599650, \"Edinburgh\" : 464990, \"Abardeen\" : 196670, \"Dundee\" : 147710}\n",
    "data3 = pd.Series(cities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Glasgow      599650\n",
       "Edinburgh    464990\n",
       "Abardeen     196670\n",
       "Dundee       147710\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
ignat's avatar
ignat committed
   "source": [
    "data3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "You can do arithmetic operations between Series similar to NumPy arrays. Even if you have 2 datasets with different data, arithmetic operations will be aligned according to their indices.\n",
ignat's avatar
ignat committed
    "\n",
    "Let's look at an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "cities_uk = {\"Birmingham\" : 1092330, \"Leeds\": 751485, \"Glasgow\" : 599650,\n",
    "             \"Manchester\" : 503127, \"Edinburgh\" : 464990}\n",
    "data4 = pd.Series(cities_uk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Abardeen            NaN\n",
       "Birmingham          NaN\n",
       "Dundee              NaN\n",
       "Edinburgh      929980.0\n",
       "Glasgow       1199300.0\n",
       "Leeds               NaN\n",
       "Manchester          NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
ignat's avatar
ignat committed
   "source": [
    "data3 + data4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "Notice how some of the results are NaN? Well, that is because there were no instances of those cities within both of the datasets. You can usually extract NaNs from a Series with\n",
ignat's avatar
ignat committed
    "```python\n",
    "data4.isnull()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "## DataFrame <a name=\"dataframe\"></a>\n",
ignat's avatar
ignat committed
    "A DataFrame represents a rectangular table of data and contains an ordered collection of columns, each of which can be a different value type. The DataFrame has both row and column index and can be thought of as a dict of Series all sharing the same index.\n",
    "\n",
    "The most common way to create a DataFrame is with dicts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\"cities\" : [\"Glasgow\", \"Edinburgh\", \"Abardeen\", \"Dundee\"],\n",
    "        \"population\" : [599650, 464990, 196670, 147710],\n",
    "        \"year\" : [2011, 2013, 2013, 2013]}\n",
    "frame = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try printing it out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cities</th>\n",
       "      <th>population</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Glasgow</td>\n",
       "      <td>599650</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Edinburgh</td>\n",
       "      <td>464990</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abardeen</td>\n",
       "      <td>196670</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dundee</td>\n",
       "      <td>147710</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cities  population  year\n",
       "0    Glasgow      599650  2011\n",
       "1  Edinburgh      464990  2013\n",
       "2   Abardeen      196670  2013\n",
       "3     Dundee      147710  2013"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
ignat's avatar
ignat committed
   "source": [
    "frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter Notebooks prints it out in a nice table but the basic version of this is also just as readable!\n",
    "\n",
    "Additionally you can also specify the order of columns during initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "frame2 = pd.DataFrame(data, columns=[\"year\", \"cities\", \"population\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can retrieve a particular column from a DataFrame with\n",
    "```python\n",
    "frame[\"cities\"]\n",
    "```\n",
    "The result is going to be a Series\n",
    "\n",
ignat's avatar
ignat committed
    "Additionally, you can retrieve a row from the dataset using\n",
ignat's avatar
ignat committed
    "```python\n",
    "frame[1]\n",
    "```\n",
    "Try it out below"
ignat's avatar
ignat committed
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-30-dd699a264eca>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-30-dd699a264eca>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    frame[index=1]\u001b[0m\n\u001b[0m               ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "frame[1]"
   ]
ignat's avatar
ignat committed
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to add and modify the columns of a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "frame2[\"size\"] = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>cities</th>\n",
       "      <th>population</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011</td>\n",
       "      <td>Glasgow</td>\n",
       "      <td>599650</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2013</td>\n",
       "      <td>Edinburgh</td>\n",
       "      <td>464990</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2013</td>\n",
       "      <td>Abardeen</td>\n",
       "      <td>196670</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2013</td>\n",
       "      <td>Dundee</td>\n",
       "      <td>147710</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year     cities  population  size\n",
       "0  2011    Glasgow      599650   100\n",
       "1  2013  Edinburgh      464990   100\n",
       "2  2013   Abardeen      196670   100\n",
       "3  2013     Dundee      147710   100"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
ignat's avatar
ignat committed
   "source": [
    "frame2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "frame2[\"size\"] = [175, 264, 65.1, 60]  # in km^2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to dicts, columns can be deleted using\n",
    "```python\n",
    "del frame2[\"size\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "del frame2[\"size\"]"
   ]
ignat's avatar
ignat committed
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another common way of creating DataFrames is from a nested dict of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cities</th>\n",
       "      <th>population</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Glasgow</td>\n",
       "      <td>599650</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Edinburgh</td>\n",
       "      <td>464990</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abardeen</td>\n",
       "      <td>196670</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dundee</td>\n",
       "      <td>147710</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cities  population  year\n",
       "0    Glasgow      599650  2011\n",
       "1  Edinburgh      464990  2013\n",
       "2   Abardeen      196670  2013\n",
       "3     Dundee      147710  2013"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
ignat's avatar
ignat committed
   "source": [
    "data2 = {\"Glasgow\": {2011: 599650},\n",
    "        \"Edinburgh\": {2013:464990},\n",
ignat's avatar
ignat committed
    "        \"Abardeen\": {2013: 196670}}\n",
ignat's avatar
ignat committed
    "\n",
ignat's avatar
ignat committed
    "frame3 = pd.DataFrame(data)\n",
    "frame3"
ignat's avatar
ignat committed
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "Here is a table of different ways of initialising a DataFrame for your reference\n",
ignat's avatar
ignat committed
    "\n",
    "| Type | Notes |\n",
    "| --- | --- |\n",
    "| 2D ndarray | A matrix of data; passing optional row and column labels |\n",
    "| dict of arrays, lists, or tuples | Each sequence becomes a column in the DataFrame; all sequences must be the same length |\n",
    "| NumPy structured/recorded array | Treated as with the \"dict of arrays, lists or tuples\" case |\n",
ignat's avatar
ignat committed
    "| dict of Series | Each value becomes a column; indexes from each Series are unioned together to<br>form the result's row index if not explicit index is passed |\n",
ignat's avatar
ignat committed
    "| dict of dicts | Each inner dict becomes a column; keys are unioned to form the row<br>index as in the \"dict of Series\" case |\n",
ignat's avatar
ignat committed
    "| List of dicts or Series | Each item becomes a row in the DataFrame; union of dict keys or<br>Series indices becomes the DataFrame's column labels |\n",
ignat's avatar
ignat committed
    "| List of lists or tuples | Treated as the \"2D ndarray\" case |\n",
    "| Another DataFrame | The DataFrame's indexes are used unless different ones are passed |\n",
    "| NumPy MaskedArray | Like the \"2D ndarray\" case except masked values become NA/missing in the DataFrame |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "# Essential Functionality <a name=\"ess_func\"></a>\n",
    "In this section, we will go through the fundamental mechanics of interacting with the data contained in a Series or DaraFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "## Reindexing <a name=\"reindexing\"></a>\n",
    "With pandas it is easy to restructure the order of your columns and rows using the `reindex` function. Let's have a look at an example:"
ignat's avatar
ignat committed
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1\n",
       "b    2\n",
       "c    3\n",
       "d    4\n",
       "e    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first define a new Series\n",
    "s = pd.Series([1, 2, 3, 4, 5], index=['a', 'b', 'c', 'd', 'e'])\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d    4\n",
       "b    2\n",
       "a    1\n",
       "c    3\n",
       "e    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now you can reshuffle the indices\n",
    "s = s.reindex(['d', 'b', 'a', 'c', 'e'])\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Easy as that! This can also be extended for DataFrames, where you can reorder both the columns and indices at the same time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Edinburgh</th>\n",
       "      <th>Glasgow</th>\n",
       "      <th>Aberdeen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Edinburgh  Glasgow  Aberdeen\n",
       "a          0        1         2\n",
       "b          3        4         5\n",
       "c          6        7         8"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first define a new Dataframe\n",
    "data = np.reshape(np.arange(9), (3,3))\n",
    "df = pd.DataFrame(data, index=[\"a\", \"b\", \"c\"],\n",
    "                  columns=[\"Edinburgh\", \"Glasgow\", \"Aberdeen\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
ignat's avatar
ignat committed
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we can restructure it with reindex\n",
    "df = df.reindex(index=[\"a\", \"d\", \"c\", \"b\"],\n",
    "          columns=[\"Aberdeen\", \"Glasgow\", \"Edinburgh\", \"Dundee\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice something interesting? We can actually add new indices and columns using the `reindex` method. This results in the new slots in our table to be filled in with `NaN` values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
ignat's avatar
ignat committed
    "## Removing columns/indices <a name=\"removing\"></a>\n",
    "Similarl to above, it is easy to remove entries. This is done with the `drop` method and can be applied to both columns and indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Edinburgh</th>\n",
       "      <th>Glasgow</th>\n",
       "      <th>Aberdeen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Edinburgh  Glasgow  Aberdeen\n",
       "a          0        1         2\n",
       "c          6        7         8"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define new DataFrame\n",
    "data = np.reshape(np.arange(9), (3,3))\n",
    "df = pd.DataFrame(data, index=[\"a\", \"b\", \"c\"],\n",
    "                  columns=[\"Edinburgh\", \"Glasgow\", \"Aberdeen\"])\n",
    "\n",
    "df.drop(\"b\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Glasgow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Glasgow\n",
       "a        1\n",
       "b        4\n",
       "c        7"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can also drop from a column\n",
    "df.drop([\"Aberdeen\", \"Edinburgh\"], axis=\"columns\")"
   ]
  },