python-data-5-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 5 - pandas\n",
    "[pandas](http://pandas.pydata.org) provides high-level data structures and functions designed to make working with structured or tabular data fast, easy and expressive. The primary objects in pandas that we will be using are the `DataFrame`, a tabular, column-oriented data structure with both row and column labels, and the `Series`, a one-dimensional labeled array object.\n",
    "\n",
    "pandas blends the high-performance, array-computing ideas of NumPy with the flexible data manipulation capabilities of spreadsheets and relational databases. It provides sophisticated indexing functionality to make it easy to reshape, slice and perform aggregations.\n",
    "\n",
    "While pandas adopts many coding idioms from NumPy, the most significant difference is that pandas is designed for working with tabular or heterogeneous data. NumPy, by contrast, is best suited for working with homogeneous numerical array data.\n",
    "<br>\n",
    "\n",
    "## Table of Contents:\n",
    "- [Data Structures](#structures)\n",
    "    - [Series](#series)\n",
    "    - [DataFrame](#dataframe)\n",
    "- [Essential Functionality](#ess_func)\n",
    "    - [Reindexing](#reindexing)\n",
    "    - [Dropping Entries](#removing)\n",
    "    - [Indexing, Slicing and Filtering](#indexing)\n",
    "    - [Arithmetic Operations](#arithmetic)\n",
    "- [Summarizing and Computing Descriptive Statistics](#sums)\n",
    "- [Loading and storing data](#loading)\n",
    "    - [Text Format](#text) \n",
    "    - [Web Scraping](#web)\n",
    "- [Data Cleaning and preperation](#cleaning)\n",
    "    - [Handling missing data](#missing)\n",
    "    - [Data transformation](#transformation)\n",
    "- [String manipulation](#strings)\n",
    "\n",
    "The common pandas import statment is shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Common pandas import statement\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Structures <a name=\"structures\"></a>\n",
    "## Series <a name=\"series\"></a>\n",
    "A Series is a one-dimensional array-like object containing a sequence of values and an associated array of data labels called its index.\n",
    "\n",
    "The easiest way to make a Series is from an array of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.Series([4, 7, -5, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try printing out data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The string representation of a Series displayed interactively shows the index on the left and the values on the right. Because we didn't specify an index, the default on is simply integers 0 through N-1.\n",
    "\n",
    "You can output only the values of a Series using \n",
    "```python\n",
    "data.values\n",
    "```\n",
    "or you can get only the indices using\n",
    "```python\n",
    "data.index\n",
    "```\n",
    "Try it out below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify custom indeces when intialising the Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data2 = pd.Series([4, 7, -5, 3], index=[\"a\", \"b\", \"c\", \"d\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use these labels to access the data similar to a normal array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data2[\"a\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to think about Series is as a fixed-length ordered dictionary. Furthermore, you can actually define a Series in a similar manner to a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cities = {\"Glasgow\" : 599650, \"Edinburgh\" : 464990, \"Abardeen\" : 196670, \"Dundee\" : 147710}\n",
    "data3 = pd.Series(cities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do arithmetic operations between Series similar to NumPy arrays. Even if you have 2 datasets with different data, arithmetic operations will be aligned according to their indices.\n",
    "\n",
    "Let's look at an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cities_uk = {\"Birmingham\" : 1092330, \"Leeds\": 751485, \"Glasgow\" : 599650,\n",
    "             \"Manchester\" : 503127, \"Edinburgh\" : 464990}\n",
    "data4 = pd.Series(cities_uk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data3 + data4"
   ]
  },
  {
   "cell_type": "markdown",