Newer
Older
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[NumPy](http://numpy.org) short for Numerical Python, has long been a cornerstone of numerical computing on Python. It provides the data structures, algorithms and the glue needed for most scientific applications involving numerical data in Python. All computation is done in vectorised form - using vectors of several values at once instead of singular values at a time. NumPy contains, among other thigs:\n",
"* A fast and efficient multidimensional array object `ndarray`.\n",
"* Mathematical functions for performing element-wise computations with arrays or mathematical operations between arrays.\n",
"* Tools for reading and manipulating large array data to disk and working with memory-mapped files.\n",
"* Linear algebra, random number generation and Fourier transform capabilities.\n",
"\n",
"For the rest of the course, whenever array is mentioned it refers to the NumPy ndarray.\n",
"<br>\n",
"\n",
"## Table of contents\n",
"- [The ndarray](#ndarray)\n",
" - [Creating arrays](#creating)\n",
" - [Data Types](#data)\n",
" - [Arithmetic Operations](#arithmetic)\n",
" - [Indexing and Slicing](#indexing)\n",
" - [Transposing and Swapping Axis](#transposing)\n",
"- [Universal Functinos](#universal)\n",
"- [Other useful operations](#other)\n",
"- [File IO](#file)\n",
"- [Linear algebra](#linear)"
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Why NumPy?\n",
"Is the first question that anybody asks when they find out about it. \n",
"\n",
"Some people might say: *I don't care about speed, I want to spend my time researching how to cure cancer, not optimise coputer code!*\n",
"\n",
"That's perfectly reasonable, but are you willing to wait a lot longer for your experiment to finish? I definitely don't want to do that. Let's see how much faster NumPy really is!\n",
"\n",
"to show that we'll be using the magic command `%timeit` which you can read more about [here](https://ipython.readthedocs.io/en/stable/interactive/magics.html) and don't worry about the details now, they will clear up later.\n",
"\n",
"Let's have a look at generating a vector of 10M random values and then summing them all up using the Python way and using the NumPy way!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"x = np.random.randn(10000000) # generate random numbers\n",
"\n",
"print(\"Running normal python sum()\")\n",
"%timeit sum(x)\n",
"\n",
"print(\"Running numpy sum()\")\n",
"%timeit np.sum(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**WOW** that was a difference of roughly a **100 times** and that was just for a single summing operation. Imagine if you had several of those running all the time!\n",
"Are you onboard with Numpy then? Let's proceed..."
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The ndarray <a name=\"ndarray\"></a>\n",
"The ndarray is a backbone on Numpy. It's a fast and flexible container for N-dimensional array objects, usually used for large datasets in Python. Arrays enable you to perform mathematical operations on whole blocks of data using similar syntax to the equivalent operations between scalar elements.\n",
"\n",
"Here is a quick example of its capabilities:"
]
},
{
"cell_type": "code",
"source": [
"import numpy as np\n",
"\n",
"# create a 2x3 array of random values\n",
"data = np.random.randn(2,3)\n",
"data"
]
},
{
"cell_type": "code",
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Every array has a shape, a tuple indicating the size of each dimnesion and a dtype. You can obtain these via the respective methods:"
"source": [
"# number of dimensions of the array\n",
"data.ndim"
]
},
{
"cell_type": "code",
"source": [
"# the size of the array\n",
"data.shape"
]
},
{
"cell_type": "code",
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
"source": [
"# the type of values store in the array\n",
"data.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating arrays <a name=\"creating\"></a>\n",
"The easiest and quickest way to create an array is from a normal Python list."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = [1.2, 5.2, 5, 7.8, 0.3]\n",
"arr = np.array(data)\n",
"\n",
"arr"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to create multidimensional arrays in a similar fashion. An example would be:\n",
"```python\n",
"data = [[1.2, 5.2, 5, 7.8, 0.3],\n",
" [4.1, 7.2, 4.8, 0.1, 7.7]]\n",
"```\n",
"Try creating a multidimensional array below and verify its number of dimensions:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also create an array filled with zeros"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.zeros(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, it is also possible to create a multidimensional array by passing a tuple as an argument"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.zeros((4,6))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"NumPy also has an equivalent to the built-in Python function `range()` but it's called `arange()`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.arange(0, 10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a summary of the most often used methods to create arrays. Use it as a future reference!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Function | Description |\n",
"|----|:--|\n",
"| array | Convert input data to an ndarray either by inferring a dtype<br>or explicitly specifying a dtype; copies the input data by default. |\n",
"| arange | Similar to the built-in `range` function but returns an ndarray. |\n",
"| linspace | Return evenly spaced numbers over a specified interval. |\n",
"| ones | Produces an array of all 1s with the given shape and dtype. |\n",
"| ones_like | Similar to `ones` but takes another array and produces a ones array<br>of the same shape and dtype |\n",
"| zeros, zeros_like | Similar to `ones` but produces an array of 0s. |\n",
"| eye | Create a square NxN identity matrix (1s on the diagonal and 0s elsewhere). |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Types <a name=\"data\"></a>\n",
"The data type or `dtype` is a special object containing the information the array needs to interpret a chunk of memory. We can specify it during the creation of an array "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"arr = np.array([1, 2, 3], dtype=np.float64)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# you can check the type of an array with\n",
"arr.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An ndarray can only hold data in **one** dtype. This makes it a little less flexible than a regular Python list, but is part of what allows NumPy to run so fast. \n",
"\n",
"NumPy has several types of data like int, float and bool. However, it also extends these by specifying the number of bits used per variable like 16, 32, 64 or 128.\n",
"To keep things simpe, you can use:\n",
"- `np.int64` to store integer numbers\n",
"- `np.float64` to store numbers with a fraction value\n",
"- `np.bool` to store `True` and `False` values\n",
"When creating arrays in NumPy the type is inferred (guessed) so you don't need to explicitly specify it.\n",
"\n",
"It is not necessary for this course but if you want to learn more about datatypes in NumPy you can go to https://jakevdp.github.io/PythonDataScienceHandbook/02.01-understanding-data-types.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similar to normal Python, you can cast (convert) an array from one dtype to another using the `astype` method:"
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"arr = np.array([1, 2, 3])\n",
"arr.dtype"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"float_arr = arr.astype(np.float64)\n",
"float_arr.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The normal limitations to casting apply here as well. You can try creating a `float64` array and then converting it to an `int64` array below:"
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 1\n",
"- Create a 5x5 [identity matrix](https://en.wikipedia.org/wiki/Identity_matrix).\n",
"- Confirm its properties such as dimensionality, shape and data type."
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Arithmetic operations <a name=\"arithmetic\"></a>\n",
"You have already gotten a taste of this in the examples above but let's try to extend that.\n",
"\n",
"Arrays are important because they enable you to express batch operations on data without having to write for loops - this is called **vectorisation**.\n",
"\n",
"Any arithmetic operation between equal-size arrays applies the operation element-wise:"
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.array([[1, 2, 3], [4, 5, 6]])\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A * A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A - A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Arithmetic operations with scalars propogate the scalar argument to each element in the array:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A * 5"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A ** 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Comparisons between arrays of the same size yield boolean arrays:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B = np.array([[1, 7, 4],[4, 12, 2]])\n",
"B"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A > B"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Broadcasting\n",
"\n",
"Applying arithmetic operations to differently sized arrays is called **broadcasting**.\n",
"\n",
"Let's see an example"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.ones((3,3))\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B = np.arange(3)\n",
"B"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A + B"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So what happened?\n",
"\n",
"We stretched both `B` to match the shape of `A`, the result is a two-dimensional array!\n",
"\n",
"<img src=\"img/broadcasting.png\" alt=\"drawing\" width=\"500\"/>\n",
"\n",
"The light boxes represent the broadcasted values: again, this extra memory is not actually allocated in the course of the operation, but it can be useful conceptually to imagine that it is.\n",
"\n",
"If you want to learn more about broadcasting, check [this article](https://jakevdp.github.io/PythonDataScienceHandbook/02.05-computation-on-arrays-broadcasting.html)."
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 2\n",
"Generate a vector of size 10 with values ranging from 0 to 1, both included."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indexing and slicing <a name=\"indexing\"></a>\n",
"NumPy offers many options for indexing and slicing. Coincidentally, they are very similar to Python.\n",
"\n",
"Let's see how this is done in 1D:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(10)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5:8]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5:8] = 0\n",
"A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Important:** Unlike regular Python, NumPy array slices are _views_ on the original array. This means that the data is not copied, and any modifications to the source array will be reflected in the view. Similarly, changing the slice will update the original array."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A_slice"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[5:8] = [12, 17, 24] #Update source array\n",
"A_slice #Slice is changed"
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A_slice[:] = 0 #Edit the slice\n",
"A #The array is changed"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now have a look at higher dimensional arrays:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
"C"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have 2 dimensions, we need to input 2 indices to get a specific element of the array. Alternatively, if we input only one index, then we obtain the whole row of the array:"
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[2]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[2][1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[2, 1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a picture to better explain indexing in 2D:\n",
"<img src=\"img/ndarray.png\" alt=\"drawing\" width=\"300\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The same concepts and techniques are extended into multidimensional arrays:\n",
"if you omit later indices, the returned object will be a lower dimensional ndarray consisting of all data along the higher dimensions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's look into **slicing**. You already saw above that slicing in 1D is done the same way as in standard Python data structures. So how do we do that in 2D? Well, it is fairly intuitive:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
"C"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[:2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This can be read as *select the first 2 rows of C*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[1, :2] # Select row 1, the first 2 columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C[:, :1] # Select all rows, first 1 column (i.e. select column 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is some visual aid for what happened above:\n",
"<img src=\"img/indexing.png\" alt=\"drawing\" width=\"400\"/>"
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to index arrays via booleans.\n",
"\n",
"Say we have an 1D array of 0s and 1s and then a 2D array of randomly generated data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fruits = np.array([\"banana\", \"orange\", \"mango\", \"banana\", \"tomato\", \"passionfruit\", \"cherry\"])\n",
"fruits"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = np.random.randn(7,4)\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fruits == \"banana\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data[fruits == \"banana\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Powerful right? The only caveat is that the boolean array must be of the same length as the array axis it's indexing.\n",
"You can also mix and match boolean arrays but there is one small difference compared to Python - the typical boolean operators (`and` and `or`) do not work, and instead you must use `&`(and) and `|`(or)."
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mask = (fruits == \"banana\") | (fruits == \"cherry\")\n",
"data[mask]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 3\n",
"Create a 5x5 matrix of random values. Square all positive values of the matrix and set all else to 0. Attempt to do this in place - ie. without copying the matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 4\n",
"Create a 4 by 4 2D array with 1s on the border and 0s inside"
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Transposing Arrays and Swapping Axes <a name=\"transposing\"></a>\n",
"We can use the method `reshape()` to convert the data from one shape into another. Later we can use the `T` attribute to obtain the transpose of the array."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(15)\n",
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B = A.reshape((3,5))\n",
"B"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B.T"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also reshape 3D arrays but how would `T` work then? Luckily, we can use the `tranpose()` method which allows us to chose the axes we want to swap:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.arange(16)\n",
"C = A.reshape((2, 2, 4))\n",
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C.transpose((1, 0, 2))\n",
"C.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plotting\n",
"You can easily plot and show images in Python via the package `matplotlib` which can be used to plot an array.\n",
"\n",
"Don't worry about the details of `matplotlib` for now, it will be studied fully in the next notebook.\n",
"\n",
"First we have to set up our environment. Read and run the code below to do just that:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"# Import NumPy and matplotlib\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Set a greyscale colourmap (we want white for 0 and black for 1)\n",
"plt.set_cmap('Greys')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can create an array of values and plot in a canvas. The easiest way to do this is to pick values between 0 and 1 and plot grayscale images where 1 corresponds to black and 0 corresponds to white. Let's see how we can do this by creating an array of 0s:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All is white right? let's add some black to it!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"canvas = np.zeros((100,50))\n",
"\n",
"canvas[50,25] = 1\n",
"\n",
"plt.imshow(canvas.T, origin='lower')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 5\n",
"Use the canvas template above and create an image where the top right and bottom left pixels are set to black.\n",
"\n",
"*Note: Remember to first reset your canvas to only 0s*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"canvas = np.zeros((100,50))\n",
"\n",
"# [PUT CODE BELOW]\n",
"\n",
"\n",
"plt.imshow(canvas.T, origin='lower')"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 6\n",
"Draw a horizontal and vertical line across the canvas"
"execution_count": null,
"metadata": {},
"outputs": [],