{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Notebook 3 - NumPy\n",
"[NumPy](http://numpy.org) short for Numerical Python, has long been a cornerstone of numerical computing on Python. It provides the data structures, algorithms and the glue needed for most scientific applications involving numerical data in Python. All computation is done in vectorised form - using vectors of several values at once instead of singular values at a time. NumPy contains, among other thigs:\n",
"* A fast and efficient multidimensional array object `ndarray`.\n",
"* Mathematical functions for performing element-wise computations with arrays or mathematical operations between arrays.\n",
"* Tools for reading and manipulating large array data to disk and working with memory-mapped files.\n",
"* Linear algebra, random number generation and Fourier transform capabilities.\n",
"\n",
"For the rest of the course, whenever array is mentioned it refers to the NumPy ndarray.\n",
"<br>\n",
"\n",
"## Table of contents\n",
"- [The ndarray](#ndarray)\n",
" - [Creating arrays](#creating)\n",
" - [Data Types](#data)\n",
" - [Arithmetic Operations](#arithmetic)\n",
" - [Indexing and Slicing](#indexing)\n",
" - [Transposing and Swapping Axis](#transposing)\n",
"- [Universal Functinos](#universal)\n",
"- [Other useful operations](#other)\n",
"- [File IO](#file)\n",
"- [Liear algebra](#linear)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Why NumPy?\n",
"Is the first question that anybody asks when they find out about it. \n",
"\n",
"Some people might say: *I don't care about speed, I want to spend my time researching how to cure cancer, not optimise coputer code!*\n",
"\n",
"That's perfectly reasonable, but are you willing to wait a lot longer for your experiment to finish? I definitely don't want to do that. Let's see how much faster NumPy really is!\n",
"\n",
"to show that we'll be using the magic command `%timeit` which you can read more about [here](https://ipython.readthedocs.io/en/stable/interactive/magics.html) and don't worry about the details now, they will clear up later.\n",
"\n",
"Let's have a look at generating a vector of 10M random values and then summing them all up using the Python way and using the NumPy way!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"x = np.random.randn(10000000) # generate random numbers\n",
"\n",
"print(\"Running normal python sum()\")\n",
"%timeit sum(x)\n",
"\n",
"print(\"Running numpy sum()\")\n",
"%timeit np.sum(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**WOW** that was a difference of more than a **100 times** and that was just for a single summing operation. Imagine if you had several of those running all the time!\n",
"\n",
"Are you onboard with Numpy then? Let's proceed..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The ndarray <a name=\"ndarray\"></a>\n",
"The ndarray is a backbone on Numpy. It's a fast and flexible container for N-dimensional array objects, usually used for large datasets in Python. Arrays enable you to perform mathematical operations on whole blocks of data using similar syntax to the equivalent operations between scalar elements.\n",
"\n",
"Here is a quick example of its capabilities:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# create a 2x3 array of random values\n",
"data = np.random.randn(2,3)\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data * 10 #multiply all numbers by 10"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data + data #element-wise addition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Every array has a shape, a tuple indicating the size of each dimnesion and a dtype. You can obtain these via the respective methods:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# number of dimensions of the array\n",
"data.ndim"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# the size of the array\n",
"data.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# the type of values store in the array\n",
"data.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating arrays <a name=\"creating\"></a>\n",
"The easiest and quickest way to create an array is from a normal Python list."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = [1.2, 5.2, 5, 7.8, 0.3]\n",
"arr = np.array(data)\n",
"\n",
"arr"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to create multidimensional arrays in a similar fashion. An example would be:\n",
"```python\n",
"data = [[1.2, 5.2, 5, 7.8, 0.3],\n",
" [4.1, 7.2, 4.8, 0.1, 7.7]]\n",
"```\n",
"Try creating a multidimensional array below and verify its number of dimensions:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also create an array filled with zeros"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.zeros(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, it is also possible to create a multidimensional array by passing a tuple as an argument"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.zeros((4,6))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another option is to use the `empty()` function which creates an array filled with garbage values. It is used in a similar way to `zeros()`.\n",
"\n",
"Try using it below and see what it creates!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.empty((10, 10))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"NumPy also has an equivalent to the built-in Python function `range()`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.arange(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are most of the possible ways of creating an array"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Function | Description |\n",
"|----|:--|\n",
"| array | Convert input data to an ndarray either by inferring a dtype<br>or explicitly specifying a dtype; copies the input data by default. |\n",
"| asarray | Convert input to ndarray, but do not copy if the input is already an ndarray. |\n",
"| arange | Similar to the built-in `range` function but returns an ndarray. |\n",
"| linspace | Return evenly spaced numbers over a specified interval. |\n",
"| ones | Produces an array of all 1s with the given shape and dtype. |\n",
"| ones_like | Similar to `ones` but takes another array and produces a ones array<br>of the same shape and dtype |\n",
"| zeros, zeros_like | Similar to `ones` but produces an array of 0s. |\n",
"| empty, emtpy_like | Create new array by allocating memory but without populating any values. |\n",
"| full, full_like | Produce an array of a given shape and dtype with all values set to the indicated \"fill value\". |\n",
"| eye | Create a square NxN identity matrix (1s on the diagonal and 0s elsewhere). |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Types <a name=\"data\"></a>\n",
"The data type or `dtype` is a special object containing the information the array needs to interpret a chunk of memory. We can specify it during the creation of an array "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"arr = np.array([1, 2, 3], dtype=np.float64)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"arr.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An ndarray can only hold data in **one** dtype. This makes it a little less flexible than a regular Python list, but is part of what allows NumPy to run so fast. Here is a table of all the dtypes.\n",
"There is no need to remember all of dtypes.\n",
"\n",
"| Type | Type code | Description |\n",
"|----|---|---|\n",
"| int8, uint8 | i1, u1 | Signed and unsigned 8bit integer |\n",
"| int16, uint16 | i2, u2 | Signed and unsigned 16bit integer |\n",
"| int32, uint32 | i4, u4 | Signed and unsigned 32bit integer |\n",
"| int64, uint64 | i8, u8 | Signed and unsigned 64bit integer |\n",
"| float16 | f2 | Half-precision floating point |\n",
"| float32 | f4 or f | Standard single-precision floating point |\n",
"| float64 | f8 or d | Standard double-precision floating point |\n",
"| float128 | f16 or g | Extended-precision floating point |\n",
"| complex64<br>complex128<br>complex256 | c8, c16, c32 | Complex numbers represented by two 32, 64 or 128 floats, respectively |\n",
"| bool | ? | Boolean type storing True or False |\n",
"| object | O | Python object type, a value can be any Python object |\n",
"| string_ | S | Fixed-length ASCII string type.<br>For example use `S10` to create a string dtype with length 10 |\n",
"| unicode_ | U | Fixed-length Unicode type |\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similar to normal Python, you can cast (convert) an array from one dtype to another using the `astype` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"arr = np.array([1, 2, 3])\n",
"arr.dtype"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"float_arr = arr.astype(np.float64)\n",
"float_arr.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The normal limitations to casting apply here as well. You can try creating a `float64` array and then converting it to an `int64` array below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 1\n",
"Create a 5x5 [identity matrix](https://en.wikipedia.org/wiki/Identity_matrix). Then convert it to float64 dtype. At the end confirm its properties using the appropriate attributes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Arithmetic operations <a name=\"arithmetic\"></a>\n",
"You have already gotten a taste of this in the examples above but let's try to extend that.\n",
"\n",
"Arrays are important because they enable you to express batch operations on data without having to write for loops - this is called **vectorisation**.\n",
"\n",
"Any arithmetic operation between equal-size arrays applies the operation element-wise:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.array([[1, 2, 3], [4, 5, 6]])\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A * A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A - A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Arithmetic operations with scalars propogate the scalar argument to each element in the array:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A * 5"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A ** 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Comparisons between arrays of the same size yield boolean arrays:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B = np.array([[1, 7, 4],[4, 12, 2]])\n",
"B"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A > B"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Applying arithmetic operations to differently sized arrays is called **broadcasting** but will not be covered in this course due to the limited time."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 2\n",
"Generate a vector of size 10 with values ranging from 0 to 1, both included."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indexing and slicing <a name=\"indexing\"></a>\n",
"NumPy offers many options for indexing and slicing. Coincidentally, they are very similar to Python.\n",
"\n",
"Let's see how this is done in 1D:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(10)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5:8]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5:8] = 0\n",
"A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Important:** Unlike regular Python, NumPy array slices are _views_ on the original array. This means that the data is not copied, and any modifications to the source array will be reflected in the view. Similarly, changing the slice will update the original array."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A_slice = A[5:8] #Take a slice\n",
"A_slice"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5:8] = [12, 17, 24] #Update source array\n",
"A_slice #Slice is changed"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A_slice[:] = 0 #Edit the slice\n",
"A #The array is changed"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now have a look at higher dimensional arrays:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
"C"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have 2 dimensions, we need to input 2 indices to get a specific element of the array. Alternatively, if we input only one index, then we obtain the whole row of the array:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[2]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[2][1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[2, 1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a picture to better explain indexing in 2D:\n",
"<img src=\"img/ndarray.png\" alt=\"drawing\" width=\"300\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The same concepts and techniques are extended into multidimensional arrays:\n",
"if you omit later indices, the returned object will be a lower dimensional ndarray consisting of all data along the higher dimensions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's look into **slicing**. You already saw above that slicing in 1D is done the same way as in standard Python data structures. So how do we do that in 2D? Well, it is fairly intuitive:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
"C"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[:2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This can be read as *select the first 2 rows of C*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[1, :2] # Select row 1, the first 2 columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[:, :1] # Select all rows, first 1 column (i.e. select column 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is some visual aid for what happened above:\n",
"<img src=\"img/indexing.png\" alt=\"drawing\" width=\"400\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to index arrays via booleans.\n",
"\n",
"Say we have an 1D array of 0s and 1s and then a 2D array of randomly generated data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fruits = np.array([\"banana\", \"orange\", \"mango\", \"banana\", \"tomato\", \"passionfruit\", \"cherry\"])\n",
"fruits"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = np.random.randn(7,4)\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fruits == \"banana\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data[fruits == \"banana\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Powerful right? The only caveat is that the boolean array must be of the same length as the array axis it's indexing.\n",
"\n",
"You can also mix and match boolean arrays but there is one small difference compared to Python - the typical boolean operators (`and` and `or`) do not work, and instead you must use `&`(and) and `|`(or)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mask = (fruits == \"banana\") | (fruits == \"cherry\")\n",
"data[mask]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 3\n",
"Create a 5x5 matrix of random values. Square all positive values of the matrix and set all else to 0. Attempt to do this in place - ie. without copying the matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 4\n",
"Create a 4 by 4 2D array with 1s on the border and 0s inside"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Transposing Arrays and Swapping Axes <a name=\"transposing\"></a>\n",
"We can use the method `reshape()` to convert the data from one shape into another. Later we can use the `T` attribute to obtain the transpose of the array."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(15)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B = A.reshape((3,5))\n",
"B"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B.T"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also reshape 3D arrays but how would `T` work then? Luckily, we can use the `tranpose()` method which allows us to chose the axes we want to swap:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(16)\n",
"C = A.reshape((2, 2, 4))\n",
"C.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C.transpose((1, 0, 2))\n",
"C.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plotting\n",
"You can easily plot and show images in Python via the package `matplotlib` which can be used to plot an array. \n",
"\n",
"First we have to set up our environment. Read and run the code below to do just that:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"# Import NumPy and matplotlib\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Set a greyscale colourmap (we want white for 0 and black for 1)\n",
"plt.set_cmap('Greys')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can create an array of values and plot in a canvas. The easiest way to do this is to pick values between 0 and 1 and plot grayscale images where 1 corresponds to black and 0 corresponds to white. Let's see how we can do this by creating an array of 0s:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"canvas = np.zeros((100,50))\n",
"plt.imshow(canvas, interpolation=\"none\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All is white right? let's add some black to it!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"canvas[50, 25] = 1\n",
"plt.imshow(canvas, interpolation=\"none\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 5\n",
"Use the canvas template above and create an image where the top right and bottom left pixels are set to black.\n",
"\n",
"*Note: Remember to first reset your canvas to only 0s*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 6\n",
"Draw a horizontal and vertical line across the canvas"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 7\n",
"Make the top left corner of the image black.\n",
"\n",
"*Extra challenge: do this wihtout using numbers for indexing*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Universal Functions <a name=\"universal\"></a>\n",
"or *ufunc* are functions that perform element-wise operations on data in ndarrays. You can think of them as fast vectorised wrappers for simple functions. Here is an example of `sqrt` and `exp`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(10)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.sqrt(A)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.exp(A)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other universal functions take 2 arrays as input. These are called *binary* functions.\n",
"\n",
"For example `maximum()` selects the biggest values from two input arrays"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x = np.random.randn(10)\n",
"y = np.random.randn(10)\n",
"np.maximum(x, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other functions can return multiple arrays:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.random.randn(10)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"remainder, whole = np.modf(A)\n",
"print(remainder)\n",
"print(whole)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a list of most *ufuncs* in NumyPy:\n",
"*again, you don't need to memorise them. This is just a reference*\n",
"### Unary functions (accept one argument)\n",
"\n",
"| Function | Description |\n",
"|----|----|\n",
"| abs, fabs | Compute the absolute value element-wise for integer, floating point, or complex values.<br>Use fabs as a faster alternative for non-complex-valued data |\n",
"| sqrt | Compute the square root of each element. Equivalent to arr ** 0.5 |\n",
"| square | Compute the square of each element. Equivalent to arr ** 2 |\n",
"| exp | Compute the exponent ex of each element |\n",
"| log, log10, log2, log1p | Natural logarithm (base e), log base 10, log base 2, and log(1 + x), respectively |\n",
"| sign | Compute the sign of each element: 1 (positive), 0 (zero), or -1 (negative) |\n",
"| ceil | Compute the ceiling of each element, i.e. the smallest integer greater than or equal to each element |\n",
"| floor | Compute the floor of each element, i.e. the largest integer less than or equal to each element |\n",
"| rint | Round elements to the nearest integer, preserving the dtype |\n",
"| modf | Return fractional and integral parts of array as separate array |\n",
"| isnan | Return boolean array indicating whether each value is NaN (Not a Number) |\n",
"| isfinite, isinf | Return boolean array indicating whether each element is finite (non-inf, non-NaN) or infinite, respectively |\n",
"| cos, cosh, sin, sinh, tan, tanh | Regular and hyperbolic trigonometric functions |\n",
"| arccos, arccosh, arcsin,<br>arcsinh, arctan, arctanh | Inverse trigonometric functions |\n",
"| logical_not | Compute truth value of not x element-wise. Equivalent to -arr. |\n",
"\n",
"### Binary functions (accept 2 arguments)\n",
"| Functions | Description |\n",
"| ---- | ---- |\n",
"| add | Add corresponding elements in arrays |\n",
"| subtract | Subtract elements in second array from first array |\n",
"| multiply | Multiply array elements |\n",
"| divide, floor_divide | Divide or floor divide (truncating the remainder) |\n",
"| power | Raise elements in first array to powers indicated in second array |\n",
"| maximum, fmax | Element-wise maximum. fmax ignores NaN |\n",
"| minimum, fmin | Element-wise minimum. fmin ignores NaN |\n",
"| mod | Element-wise modulus (remainder of division) |\n",
"| copysign | Copy sign of values in second argument to values in first argument |\n",
"| greater, greater_equal, less,<br>less_equal, equal, not_equal |\tPerform element-wise comparison, yielding boolean array. <br>Equivalent to infix operators >, >=, <, <=, ==, != |\n",
"| logical_and, logical_or, logical_xor | Compute element-wise truth value of logical operation. Equivalent to infix operators & |, ^ |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Other useful operations <a name=\"other\"></a>\n",
"NumPy offers a set of mathematical functions that compute statistics about an entire array:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B = np.random.randn(5, 4)\n",
"B"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B.mean()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.mean(B)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B.sum()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B.mean(axis=1) # Compute mean in column (axis 1) direction (i.e. the mean of each row)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here `mean(axis=1)` means compute the mean across the columns (axis 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a set of other similar functions:\n",
"\n",
"| Function | Description|\n",
"| --- | --- |\n",
"|sum | Sum of all the elements in the array or along an axis. Zero-length arrays have sum 0. |\n",
"| mean | Arithmetic mean. Zero-length arrays have NaN mean. |\n",
"| std, var | Standard deviation and variance, respectively, with optional<br>degrees of freedom adjustment (default denominator n). |\n",
"|min, max | Minimum and maximum. |\n",
"| argmin, argmax | Indices of minimum and maximum elements, respectively. |\n",
"| cumsum | Cumulative sum of elements starting from 0 |\n",
"| cumprod | Cumulative product of elements starting from 1 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There also are some boolean operations. `any` tests whether one or more values in an array is `True`, and `all` tests whether all values are `True`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.random.randn(100)\n",
"A_bool = A > 0\n",
"A_bool"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A_bool.any()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A_bool.all()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 8\n",
"Generate and normalise a random 5x5x5 matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 9\n",
"Create a random vector of size 30 and find its mean value"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 10\n",
"\n",
"Subtract the mean of each row of a randomly generated matrix:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sorting <a name=\"sorting\"></a>\n",
"Similar to Python's built-in list type, NumyPy arrays can be sorted in place:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.random.randn(10)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A.sort()\n",
"A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another option is `unique()` which returns the sorted unique values in an array."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Linear Algebra <a name=\"linear\"></a>\n",
"Similar to other languages like MATLAB, NumyPy offers a set of standard linear algebra operations, like matrix multiplication, decompositions, determinants and etc.. Unlike some other languages though, the default operations like `*` peform element-wise operations. To perform matrix-wise operartions we need to use special functions:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"temp = np.arange(16)\n",
"A = temp[:8]\n",
"B = temp[8:]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A.dot(B)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also extend this with the `numpy.linalg` package:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from numpy.linalg import inv, qr\n",
"A = np.random.randn(5, 5)\n",
"mat = A.T.dot(A)\n",
"mat"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"inv(mat)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mat.dot(inv(mat))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a set of commonly used numpy.linalg functions\n",
"\n",
"| Function | Description |\n",
"| --- | --- |\n",
"| diag | Return the diagonal (or off-diagonal) elements of a square matrix as a 1D array,<br>or convert a 1D array into a square matrix with zeros on the off-diagonal |\n",
"| dot | Matrix multiplication |\n",
"| trace | Compute the sum of the diagonal elements |\n",
"| det | Compute the matrix determinant |\n",
"| eig | Compute the eigenvalues and eigenvectors of a square matrix |\n",
"| inv | Compute the inverse of a square matrix |\n",
"| pinv | Compute the Moore-Penrose pseudo-inverse inverse of a square matrix |\n",
"| qr | Compute the QR decomposition |\n",
"| svd | Compute the singular value decomposition (SVD) |\n",
"| solve | Solve the linear system Ax = b for x, where A is a square matrix |\n",
"| lstsq | Compute the least-squares solution to y = Xb |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 11\n",
"Obtain the diagonal of a dot product of 2 random matrices"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## File IO <a name=\"file\"></a>\n",
"NumPy offers its own set of File IO functions.\n",
"\n",
"The most common one is `genfromtxt()` which can load the common `.csv` and `.tsv` files.\n",
"\n",
"Now let us analyse temperature data from Stockholm over the years.\n",
"\n",
"First we have to load the file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = np.genfromtxt(\"./data/stockholm_td_adj.dat\")\n",
"data.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first column of this array gives years, and the 6th gives temperature readings. We can extract these."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"yrs = data[:, 0]\n",
"temps = data[:, 5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Having read in our data, we can now work with it - for example, we could produce a plot.\n",
"We will cover plotting in more depth in notebook 4, so there's no need to get too caught up in the details right now - this is just an examle of something we might do having read in some data. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(16, 6)) # Create a 16x6 figure\n",
"plt.plot(yrs, temps) # Plot temps vs yrs\n",
"\n",
"#Set some labels\n",
"plt.title(\"Temperatures in Stockholm\")\n",
"plt.xlabel(\"year\")\n",
"plt.ylabel(\"Temperature (C)\")\n",
"\n",
"plt.show() # Show the plot"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 12\n",
"Read in the file `daily_gas_price.csv`, which lists the daily price of natural gas since 1997. Each row contains a date and a price, separated by a comma. Find the minimum, maximum, and mean gas price over the dataset.\n",
"\n",
"(Hint: you will need to use the delimiter option in `np.genfromtxt` to specify that data is separated by commas. Also, NumPy will interpret the data in float format by default - we may need to set the dtype to a string format at first, then discard the dates, before turning the gas prices back into floats to process them! Otherwise, NumPy may find it confusing to try and interpret dates formatted as YYYY-MM-DD as floats and will probably complain.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}