\lab{Essentials}{NumPy}{NumPy}
\label{lab:Essentials_NumPy}

\section*{Why Arrays?}
Let's begin with a simple demonstration of why arrays are important for numerical computation.
Why use arrays when Python already has a reasonably efficient list object?
In this demonstration, we will try squaring a matrix.
The matrix will be represented as a two dimensional list (i.e. a list of lists).

The following is a function that will accept two matrices (two dimensional list), $A$ and $B$, and return $AB$ following the usual rules of matrix multiplication.
\lstinputlisting[style=python,name=]{arr_mult.py}
We can initialize a $k \times k$ ``array" of integers like this:
\begin{lstlisting}
: k = 10
: A = [range(i, i+k) for i in range(0, k**2, k)]
\end{lstlisting}

\begin{problem}
Time how long this function takes to square matrices for increasing values of \li{k}.
In IPython you can time how long it takes for a line of code to execute by prefacing it with \li{\%timeit}, as in \li{\%timeit range(100)}.

Now import NumPy and create a NumPy array, \li{A}, and square it.
\li{A*A} does \emph{not} square the array, but rather multiplies \li{A} with itself element by element.
To get matrix multiplication for NumPy arrays, you must use \li{np.dot} or the \li{dot()} method of an array like this:
\begin{lstlisting}
import numpy as np
A = np.array([range(i, i+k) for i in range(0, k**2, k)])
np.dot(A, A)
\end{lstlisting}
Time how long NumPy takes to square arrays for increasing sizes of \li{k}.
What do you notice about the time needed to square a two dimensional list vs. a two dimensional NumPy array?
\end{problem}

% Below is a comparison of runtimes needed to square a matrix
% \begin{center}
% \begin{tabular}{|c|l|l|}
% \hline
%  Data Structure & Size & Time (s) \\ \hline
%  Python List & $1\times1$ & 0.0000181198 \\ \cline{2-3}
%       & $10\times10$ & 0.0002758503 \\ \cline{2-3}
%       & $100\times100$ & 0.1336028576 \\ \cline{2-3}
%       & $1000\times1000$ & 200.4009799957 \\ \hline \hline
%  NumPy Array & $1\times1$ & 0.0000298023 \\ \cline{2-3}
%       & $10\times10$ & 0.0000109673 \\ \cline{2-3}
%       & $100\times100$ & 0.0009210110 \\ \cline{2-3}
%       & $1000\times1000$ & 2.1682999134 \\ \hline
% \end{tabular}
%
% \end{center}

The reason for the drastic speed difference is that Python, as a high level interpreted language, tends to be slower than lower level compiled languages.
The algorithms implemented in NumPy are heavily optimized and are usually implemented in C or Fortran.
Instead of operating purely in Python, they use Python to run code that is written and optimized in other languages.
NumPy interfaces with some of the best known packages for doing computational linear algebra and can be used to write relatively fast programs.

Lists are still faster for anything that involves a varying lengths of data.
If you are appending to your data or deleting items, consider using lists instead of arrays since both of these operations require the creation of a new array.

\section*{NumPy}
NumPy is a fundamental package for scientific computing with Python.
It provides an efficient $n$-dimensional (arbitrary dimensional) array object for fast computations.
These arrays are commonly known as \emph{ndarrays}.
This lab will focus on how to use these powerful objects.
NumPy is commonly imported as shown below.
NumPy also provides a matrix object which is designed to behave like matrices in MATLAB.
It is strongly encouraged to use NumPy arrays instead of NumPy matrices, despite their convenience.
\begin{lstlisting}
: import numpy as np
\end{lstlisting}
First, it will be useful to explain certain concepts and terms used to describe NumPy arrays.

\subsection*{Arbitrary Dimensions}
One, two, and three dimensional arrays are easy to visualize.
But how do we visualize a four, ten, or fifteen dimensional array?
NumPy arrays are best thought of as arrays within arrays.
A one dimensional array consists of only elements.
A two dimensional array is really just an array containing arrays which contain elements.
Extending this metaphor, a three dimensional array is an array of arrays of arrays.
Can you guess what a five dimensional array is?
Let's define a random three dimensional array.
\begin{lstlisting}
: arr = np.random.randint(50, size=(5, 4, 3))
\end{lstlisting}
Arrays are objects in Python.
For those who are familiar with object oriented programming, it is worth noting that many of the functions we discuss here are also implemented as methods of \li{ndarray} objects.
Two simple attributes of arrays that describe the number of elements in an array are \li{shape} and \li{size}
\li{shape} gives information about the size of the array in each dimension.
\li{Size} gives the total number of elements in every dimension of the array.
\begin{lstlisting}
: arr.shape
(5, 4, 3)
: arr.size
60
\end{lstlisting}

If we want to know how much memory an array uses to store its elements, we can use \li{arr.nbytes}.
The number of bytes is dependent on the data type or \emph{dtype} of the array.
The data types that NumPy uses are different from Python data types.
An integer in NumPy is not the same as an integer Python.
Remembering this is vital.
NumPy uses low level data types to speed up calculations.
However, these data types are susceptible to a problem called \emph{overflow}.
A 64 bit integer has enough bits to represent integers between $-2^{63}$ (that's $-9,223,372,036,854,775,808$) and $2^{63} - 1$.
If we have an array with $-2^{63}$ and we decide to subtract 1, the integer wraps around and becomes $2^{63} - 1$!  If we wish to reduce memory usage, we can use smaller integer types.
NumPy has support for 8, 16, 32, and 64 bit integers.
\begin{lstlisting}
: arr.dtype
dtype('int64')
: arr.nbytes
480
: arr.astype('int32').nbytes
240
\end{lstlisting}

Each element of the array has a unique address that describes its location.
Indexing always starts at $0$.
Also, like Python lists, negative indices are valid and count backwards from the end of the array.
We will discuss indexing in more detail later in this lab.
\begin{lstlisting}
: arr[0, 0, 0] #returns the first element of arr
: arr[-1, -1, -1] #returns the last element of arr
\end{lstlisting}

\section*{Creating Arrays}
NumPy has several functions for creating and initializing arrays.
When creating an array, we can, optionally, specify the data type that is stored in the array.
NumPy arrays are homogeneous, meaning that all elements in an array must have the same datatype.
The array order dictates how the array is laid out in memory.
Two common layouts are C order and Fortran order.
C ordered arrays are also known as row-major arrays.
This means that the fastest changing index corresponds to the rows of the array.
This is because rows are stored in contiguous blocks in memory.
Fortran ordered arrays are column-major.
Another common way of saying this is that an array is C contiguous or Fortran contiguous.
Not all arrays are C contiguous or Fortran contiguous, but it is useful to understand the distinction.
Let's look at a few of the ways we can create arrays in NumPy.
\begin{itemize}
\item \li{np.array}: Makes an array from a Python list or tuple.
Can also be used to make a new array based on an old one, possibly with a changed datatype.
\item \li{np.empty}: Allocates an array of a specific size without initializing the elements.
\item \li{np.empty_like}: Allocates a new uninitialized array with the same shape and type as the input array.
\item \li{np.ones}: Allocates and array and initializes each element to $1$.
\item \li{np.ones_like}: Allocates a new array of ones with the same shape and type as the input array.
\item \li{np.zeros}: Allocates an array and initializes each element to $0$.
\item \li{np.zeros_like}: Allocates a new array of zeros with the same shape and type as the input array.
\item \li{np.identity}: Allocates a 2D array with ones along the main diagonal and zeros everywhere else.
\item \li{np.random.rand}: Allocates an array of random floating point values between 0 and 1.
\end{itemize}

\subsection*{Universal Functions (ufuncs)}
NumPy and SciPy include a wide variety of functions that are designed to operate on arrays.
There are simple examples like \li{sin}, \li{cos}, \li{sqrt}, \li{exp}, and \li{log} all the way to special functions like \li{polygamma} in the \li{scipy.misc} submodule.
There are far more functions available in NumPy than could possibly be included here, so you will want to become familiar with the NumPy and SciPy documentation at \url{docs.scipy.org/doc/}.
If you need to do any sort of simple operation on an array, there is very often a function there to do it.
These functions are almost always faster and more convenient than iterating through the whole array.

Most of these functions also allow you to specify an array for the output.
It is useful in cases where you would like to avoid unnecessary memory allocation.
The output array does need to be the correct shape to store the output.
For example:
\begin{lstlisting}
: np.exp(A, out=A) #take exp(A) and store result in A
\end{lstlisting}

Other useful examples are \li{max}, \li{min}, \li{absolute}, and \li{average}.
Each of these operations also allows you to specify whether you want to operate across a particular axis or over the whole array.
For example:
\begin{lstlisting}
: np.max(A, axis=0) #max along axis 0
\end{lstlisting}
The above example returns a row of A which represents the maximum of all the rows of A.
If we had set \li{axis=1}, it would have taken the maximum of all the columns.
If, for purposes of broadcasting (discussed later) you need the output of one of these functions to have the same number of dimensions as the original array, you can also include the argument \li{keepdims=True}.

\subsection*{Indexing Arrays}
\subsubsection*{Array Views and Copies}
Before we begin accessing arrays, it is important to understand that NumPy has two ways of slicing an array.
Slice operations always return a \emph{view} and fancy indexing always returns a \emph{copy}.
Understand that even though they may look the same, views and copies are very different.

Views are special arrays that reference other arrays.
Changing elements in a view changes the array it references.
Below, we demonstrate the behavior of a view.
Notice that \li{c} looks like a copy of \li{b} even though it is not.
\begin{lstlisting}
: b = np.reshape(np.arange(25), (5,5))
: b
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
: c = b[:] #looks like c is a copy of b
: c
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
: id(c) == id(b) #We have unique objects
False
: c[2] = 500
: c
array([[  0,   1,   2,   3,   4],
       [  5,   6,   7,   8,   9],
       [500, 500, 500, 500, 500],
       [ 15,  16,  17,  18,  19],
       [ 20,  21,  22,  23,  24]])
: b #changing c also changed b!
array([[  0,   1,   2,   3,   4],
       [  5,   6,   7,   8,   9],
       [500, 500, 500, 500, 500],
       [ 15,  16,  17,  18,  19],
       [ 20,  21,  22,  23,  24]])
\end{lstlisting}
The reason that changing the array \li{c} also changed the array \li{b} is because \li{c} and \li{b} share the same memory, even though they are different Python objects.
Views reduce the overhead of making copies of arrays and are useful when we want to change certain parts of the array.

A copy of an array is a separate array that is allocated separately.
An array can be copied using the \li{copy()} function.
\begin{lstlisting}
: b = np.reshape(np.arange(25), (5, 5))
: b
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
: c = np.copy(b)
: c is b #we still have separate objects
False
: c[2] = 500
: c
array([[  0,   1,   2,   3,   4],
       [  5,   6,   7,   8,   9],
       [500, 500, 500, 500, 500],
       [ 15,  16,  17,  18,  19],
       [ 20,  21,  22,  23,  24]])
: b
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
\end{lstlisting}
Changing the data in a copy of an array, doesn't change the data in the original array.
The two arrays address different locations in memory.

\subsubsection*{Slices}
Each element of an array has a unique address that we can use to retrieve that element.
Indexing and slicing of NumPy arrays follows syntax that is similar to the syntax used with Python lists.
We will demonstrate on a simple 2D array.
Remember that slicing arrays always return a view of an array.
\begin{lstlisting}
: arr = np.reshape(np.arange(25), (5,5))
: arr
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
: arr[0, 0] #access the first element
0
: arr[-1, -1] #access the last element
24
: arr[0] #access the first row
array([0, 1, 2, 3, 4])
\end{lstlisting}
We can access ranges of elements using Python lists.
We can also more concisely select ranges using the \li{arr[start:stop:step]} range notation.
\begin{lstlisting}
: arr[::2] #get every other row, equivalent to arr[range(0, len(arr), 2)]
array([[ 0,  1,  2,  3,  4],
       [10, 11, 12, 13, 14],
       [20, 21, 22, 23, 24]])
: arr[::2, ::2] #get every other row and every other column
array([[ 0,  2,  4],
       [10, 12, 14],
       [20, 22, 24]])
: arr[3:, 3:] #extract lower right 2x2 subarray
array([[18, 19],
       [23, 24]])
: arr[:, 1] #extract second column
array([ 1,  6, 11, 16, 21])
\end{lstlisting}
Operations like those above are called array slicing.
Array slices are views of portions of the data of the original array.
They do not copy any data.

\subsubsection*{Fancy Indexing}
When the indexing object is a list or an array, NumPy behaves a little differently.
This feature is commonly referred to as fancy indexing.
One difference is that fancy indices always return a copy of an array instead of a view.
There are two types of fancy indexing: boolean and integer.
Boolean indexing returns an array of \li{True} or \li{False} values depending on some evaluating condition.
\begin{lstlisting}
: bmask = (arr > 15) & (arr < 23)
: bmask
array([[False, False, False, False, False],
       [False, False, False, False, False],
       [False, False, False, False, False],
       [False,  True,  True,  True,  True],
       [ True,  True,  True,  False,  False]], dtype=bool)
: arr[bmask]
array([16, 17, 18, 19, 20, 21, 22])
: arr[(arr > 15) & (arr < 23)] #this is the shortened form
array([16, 17, 18, 19, 20, 21, 22])
: arr[~bmask] #invert the mask
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 23, 24])
\end{lstlisting}
\begin{lstlisting}
: arr[(0, 2, 4), (0, 2, 4)] #grab every other element of diagonal
array([ 0, 12, 24])
: arr[range(0, 5, 2), range(0, 5, 2)] #same as above, but with ranges
array([ 0, 12, 24])
: arr[:, [0, -1]] #grab first and last column
array([[ 0,  4],
       [ 5,  9],
       [10, 14],
       [15, 19],
       [20, 24]])
\end{lstlisting}

Though fancy indexing does not return a view of the array, it \emph{can} be used for assignment.
For example, we will set all values of an array that are less than .5 to 0 as follows:
\begin{lstlisting}
: from numpy.random import rand
: A = rand(10, 10)
: A[A<.5] = 0.
\end{lstlisting}

\begin{problem}
Generate a random $1000 \times 1000$ array \li{A}.
Now create an uninitialized array \li{B} with all the same attributes as \li{A}.
Now do the following 100 times:
\begin{itemize}
\item Overwrite \li{B} so that it is an array of new random values like \li{A}.
This can be done like this: \li{B[:] = rand(1000,1000)}
\item Use fancy indexing to make \li{A} the maximum of \li{A} and \li{B}.
\end{itemize}
Now take \li{exp(A)} and have NumPy store the output directly in \li{A}.
Take the maximum along the vertical axis and average the result.
The final number should be very close to $e$.
\end{problem}

\section*{The Transpose and Other Useful Operations}
You may have noticed that we already used the \li{reshape()} function.
It allows us to make a new view of our array with a desired shape.
The new shape must make an array of the same size as the original array.
If we need to take the transpose of an array \li{A}, we can use the transpose attribute of the array, \li{A.T}, which will give us a new view of the same array with the order of the dimensions reversed.
The NumPy \li{transpose()} function will also do this.
Remember that both of these methods will return views of the original array and not new arrays.

The functions \li{flatten}, \li{vstack}, \li{hstack}, and \li{copy} are examples of functions that will return new arrays.
Given a $3 \times 3$ array, \li{flatten} will return a new copy of the array with shape \li{(9,)}, \li{vstack} will return a new copy of the array with shape \li{(9,1)}, \li{hstack} will return a new copy of the array  with shape \li{(1,9)}, and \li{copy} will return an exact copy of the array.

The syntax \li{A[:]} will return a new view of \li{A} that is identical to \li{A}.
This doesn't sound all that useful, but if you want to override the values in an array this syntax can be extremely useful.
For example:
\begin{lstlisting}
: import numpy as np
: from numpy.random import rand
: A = rand(100)
: B = rand(100)
: A[:] = 0 #sets each entry in A to 0
: A[:] = B #copies the data from B into A
\end{lstlisting}

\begin{problem}
Operations that create completely new arrays are generally slower than operations that create views because they have to copy all of the data from the original array.
Create a $1000 \times 1000$ array \li{A} of floating point values.
Compare the speed of the operations \li{A.reshape(A.size)} and \li{A.flatten()} (here we are calling the methods of the arrays, these are the same as \li{np.reshape(A, A.size)}, and \li{np.flatten(A)} respectively).
Why is there such a difference in speed?
What is the difference between their output?
What about \li{A.reshape((1,A.size))}?

Now let \li{A} be the an array with shape \li{(1,1000000)}.
What is the difference in the output between \li{np.vstack(A)} and \li{A.T}?
Which of the two is faster?
\end{problem}

\begin{problem}
One good application of array slicing is the Jacobi method for solving Laplace's equation on a square.
This is an example of a simple iterative method.
In this case we will modify our array in place.
Make a function that accepts an array and a tolerance as input and does the following:
\begin{itemize}
\item Copy the array
\item Make a variable representing the difference between the arrays.
Initialize it as the tolerance.
\item While the difference is less than or equal to the tolerance
\begin{itemize}
	\item set all points that are not on an edge of the new array equal to the average of their 4 immediate neighbors.
Use the values from the old array for this computation.
This should only take one line and should be based entirely on array slicing.
	\item update the difference to be the maximum of the absolute value of the new array minus the old one.
	\item copy the values from the new array into the old one (without creating a new array).
\end{itemize}
\end{itemize}

Now use the following code to generate a plot of your results
\begin{lstlisting}
: from matplotlib import pyplot as plt
: from mpl_toolkits.mplot3d import Axes3D
: n = 100
: tol = .0001
: U = np.ones((n, n))
: U[:,0] = 100
: U[:,-1] = 100
: U[0] = 0
: U[-1] = 0
: laplace(U, tol)
: X = np.linspace(0, 1, n)
: Y = np.linspace(0, 1, n)
: X, Y = np.meshgrid(X, Y)
: fig = plt.figure()
: ax = fig.gca(projection='3d')
: ax.plot_surface(X, Y, U, rstride=5)
: plt.show()
\end{lstlisting}
\end{problem}

\section*{Array Broadcasting}
Array broadcasting allows NumPy to work effectively with arrays of sizes that don't match exactly.
There are four basic rules to determine the behavior of broadcasted arrays.
\begin{remunerate}
\item All input arrays of lesser dimension than the input array with largest dimension have 1's prepended to their shapes.
\item The size in each dimension of the output shape is the maximum of all the input sizes in that dimension.
\item An input can be used in the calculation if its size in a particular dimension either matches the output size in that dimension, or has a value exactly 1.
\item If an input has a dimension size of 1 in its shape, the first data entry in that dimension will be used for all calculations along that dimension.
\end{remunerate}
To broadcast arrays, at least one of the following must be true.
\begin{remunerate}
\item All input arrays have exactly the same shape.
\item All input arrays are of the same dimension and the length of corresponding dimensions match or is 1.
\item All input arrays of fewer dimension can have 1 prepended to their shapes to satisfy the second criterion.
\end{remunerate}

One simple example is multiplying a two dimensional array by a set of numbers along its rows or columns.
Run the following lines of code and consider their output:
\begin{lstlisting}
: import numpy as np
: A = np.ones((3, 3))
: B = np.vstack(np.array([1, 2, 3,]))
: A * B #multiplies the rows of A by each entry of B
array([[ 1.,  1.,  1.],
       [ 2.,  2.,  2.],
       [ 3.,  3.,  3.]])
: A * B.T #multiplies the columns of A by each entry of B
array([[ 1.,  2.,  3.],
       [ 1.,  2.,  3.],
       [ 1.,  2.,  3.]])
\end{lstlisting}

For a more detailed description of array broadcasting rules, see \url{http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html}.

When working with multi-dimensional arrays, it can be useful to create new views of your arrays that change the order of the dimensions.
This will give you greater control over how different arrays are broadcast together.
This can be done by taking a transpose, using the \li{rollaxis} function, or using the \li{swapaxes} function.
The \li{rollaxis} function will take a given axis and change the ordering of the axes of the array so that that particular axis appears at a new place.
The order of the other axes will not be changed.
The \li{swapaxes} function will swap the placement of two axes.
Note that all three of these options create views of the previous array.
There is no copying of data involved.

\begin{problem}
Explore array broadcasting.
Create an example for each of the three cases where arrays are broadcasted.
Don't use the example already given.
\end{problem}

\section*{Saving Arrays}
It is often useful to save an array as a file.
NumPy provides several easy methods for saving and loading array data.
\begin{table*}[h]
\begin{tabular}{|l|l|}
\hline
\li{np.save(file, arr)} & Save an array to a binary file \\
\li{np.savez(file, *arrs)} & Save multiple arrays to a binary file \\
\li{np.savetxt(file, arr)} & Save an array to a text file \\
\hline
\end{tabular}
\end{table*}

\begin{table*}[h]
\begin{tabular}{|l|l|}
\hline
\li{np.load(file)} & Load and return an array from a binary file \\
\li{np.loadtxt(file)} & Load and return an array from text file \\
\hline
\end{tabular}
\end{table*}

Let's practice saving an array to a file and loading it again.
Note that, when saving an array, NumPy automatically appends the extension \li{.npy} if it is not already present.
\begin{lstlisting}
a = np.arange(30)
np.save('test_arr', a)
new_a = np.load('test_arr.npy')
np.savez('test_multi', a=a, new_a=new_a)
arrs = np.load('test_multi.npz')
\end{lstlisting}
The variable \li{arrs} points to a dictionary object with the keys \li{a} and \li{new_a} which reference the arrays that have been saved.
The \li{.npz} file extension is the file type used to store multiple arrays.

\section*{SciPy}
SciPy is a Python library that provides many easy-to-use, efficient algorithms for all types of scientific computing.
It is designed to work with NumPy arrays.
It is therefore important to master the use of NumPy arrays.
Many of the algorithms needed to solve problems in these lab manuals are implemented in SciPy.
Scipy is organized into packages grouped by problem type.
There are packages for statistics, optimization, signal processing, linear algebra, etc.
\begin{table}[h]
\centering
\begin{tabular}{|l|l|}
\hline
cluster & Clustering package \\
constants & Physical and mathematical constants and units \\
fftpack & Fast Fourier transforms \\
integrate & Integration package \\
interpolate & Interpolation package \\
io & Data input/output from various formats \\
linalg & Linear algebra functions \\
misc & Various utilities that can't be classified elsewhere \\
ndimage & Multi-dimensional image processing \\
odr & Orthogonal distance regression \\
optimize & Optimization and root finding \\
signal & Signal processing package \\
sparse & Two dimensional sparse matrices \\
sparse.linalg & Sparse matrix linear algebra functions \\
sparse.csgraph & Fast graph algorithms based on sparse matrices \\
spatial & Spatial algorithms and data structures \\
special & Special functions \\
stats & Statistical functions \\
mstats & Statistical functions for masked arrays \\
\hline
\end{tabular}
\caption{SciPy subpackages}
\end{table}

One of the primary packages that we will be using is \li{scipy.linalg}.
Even though NumPy also has \li{numpy.linalg}, using \li{scipy.linalg} is preferred.
Reasons for this include the fact that \li{scipy.linalg} is more fully featured and is always compiled with BLAS/LAPACK support (this support is optional in NumPy).
NumPy's linear algebra library should only be used to avoid adding scipy as a dependency.
Let's look at a few of the routines in SciPy's linear algebra module.
\begin{lstlisting}
: import numpy as np
: from scipy import linalg as la
: arr = np.array([[2, 3], [5, 7]])
: la.det(arr) #find the determinant of arr
-1.0000000000000018
: la.inv(arr) #find the inverse of arr
array([[-7.,  3.],
       [ 5., -2.]])
: la.eig(arr) #return eigenvalues and corresponding eigenvectors
(array([-0.10977223+0.j,  9.10977223+0.j]),
 array([[-0.81797819, -0.38876264],
       [ 0.57524923, -0.92133794]]))
\end{lstlisting}

There are also many extension packages for more specialized problems such as machine learning and natural language processing.
These packages are called \emph{SciKits}.
These kits augment the functionality of SciPy.
They are designed to work efficiently with NumPy arrays.
Some portions of these packages may someday be included in SciPy itself, but many of these packages are still incomplete or in development.
However, they can still be useful.
The following are some of the packages currently included as scikits.
\begin{table}[h]
\centering
\begin{tabular}{|l|l|}
\hline
scikit-aero & Aeronautical engineering routines \\
scikit-commpy & Digital communications in Python \\
scikit-fmm & Fast marching method implementation \\
scikit-image & Image processing functions \\
scikit-learn & Machine learning and data mining routines \\
scikit-rf & RF/Microwave engineering routines \\
ann & Approximate nearest neighbor search using KDTrees \\
audiolab & Make noise from NumPy arrays \\
bootstrap & Bootstrap confidence interval estimation \\
bvp1lg & Multi-point boundary value problem solver \\
bvp\_solver & Two-point boundary value problem solver \\
cuda & Python interface to CUDA libraries \\
datasmooth & Data smoothing routines \\
eartho & Earth observation functions \\
fitting & Data fitting framework \\
hydroclimpy & Environmental time series for Python \\
odes & Ordinary differential equation solver for Python \\
samplerate & High quality audio resampling \\
scattpy & Light scattering methods for Python \\
sparse & Sparse matrix package \\
statsmodels & Statistical computations and models \\
talkbox & Set of routines for speech/signal processing \\
timeseries & Time series manipulation \\
vectorplot & Vector fields plotting algorithms \\
\hline
\end{tabular}
\caption{SciKits extend the functionality of SciPy.}
\end{table}


\section{Properties of Norms}

\section{$L^p$ Lebesgue Vector Norms}
\label{sec:lpnorms}

Let $p\geq1$ be a real number, then the $p$-norm or $L^p$ norm of a vector $\mathbf{x}\in\mathbb{C}^n$ is defined as:

\begin{equation}
||\mathbf{x}||^p = \left[ \sum^n_i |x_i|^p \right]^{1/p}
\end{equation}

The expression can still be useful for $0<p<1$, but in that case the result is not a proper norm, because it is not subadditive (does not satisfy $f(x+y) \leq f(x) + f(y)$). $p$-norms are closely related to expressions for the generalized mean.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{unitsets.png}
    \caption{Unit Circles: Level sets $\{\mathbf{x}: ||\mathbf{x}||_q = 1\}$ for different $L^q$ norms.}
    \label{fig:unitsets}
\end{figure}

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{dispersionmetrics.png}
    \caption{Distance metrics base for different $L^q$ norms.}
    \label{fig:unitsets}
\end{figure}

\subsection{$L^{1}$ Taxicab / Manhattan Norm}
\label{sec:l1norm}
\begin{equation}
||\mathbf{x}||^1 = \sum_i |x_i|
\end{equation}

In the context of regression, $L^1$ loss gives the maximum likelihood estimator under the assumption of Laplacian (double exponential) distributed errors. For a dataset $\mathbf{x}\in S$, minimizing $\argmin_{\mathbf{s}} \sum_{\mathbf{x}\in S}||\mathbf{x}-\mathbf{s}||_{1}$ gives the median.

\subsection{$L^{2}$ Euclidian Norm}
\label{sec:l2norm}
\begin{equation}
||\mathbf{x}||^2 = \sum_i |x_i|^2
\end{equation}

In the context of regression, $L^2$ loss gives the maximum likelihood estimator under the assumption of normally distributed errors. For a dataset $\mathbf{x}\in S$, minimizing $\argmin_{\mathbf{s}} \sum_{\mathbf{x}\in S}||\mathbf{x}-\mathbf{s}||_{2}$ gives the mean.

I believe that $L^2$ norm should be the only norm that preserves the distance between two points under rotations of the coordinate system.

\subsection{$L^{\infty}$ Maximum Norm}

\begin{equation}
||\mathbf{x}||^{\infty} = \max(x_1,x_2,...,x_n)
\end{equation}

For a dataset $\mathbf{x}\in S$, minimizing $\argmin_{\mathbf{s}} \sum_{\mathbf{x}\in S}||\mathbf{x}-\mathbf{s}||_{\infty}$ gives the average of the maximum and the minimum value of the dataset.

\subsection{$L^{-\infty}$ Minimum Norm}

Formally, I only came across values $0<p$, but it is my opinion that $-\infty$ picks out the minimum value:

\begin{equation}
||\mathbf{x}||^{-\infty} = \min(x_1,x_2,...,x_n)
\end{equation}

\section{Operator and Matrix Norms}
\label{sec:norms}

Matrix norms are functions $||\cdot||: K^{m\times n} \rightarrow \mathbb{R}$ where $K$ is a field of real or complex numbers. They satisfy:

\begin{itemize}
\item $||\alpha A|| = |a| ||A||$ (absolutely homogenous)
\item $||A+B|| \leq ||A|| + ||B||$ (triangle inequality, subadditivity)
\item $||A||\geq 0$ (positive valued)
\item $||A||=0 \implies A_{n,m}=0$ (definiteness)
\end{itemize}

A norm is submultiplicative if it satisfies $||AB||\leq||A||||B||$, which \citeasnoun{rgeraNotes} calls a requirement of "useful matrix norms".

\subparagraph{}
The main risk of confusion is that norms for operators and vectors are different animals. Norms for operators normally measure some relationship between input and output. Norms for vectors are normally some kind of size, length or distance metric. In as far as matrices can be thought of as both operators and multidimensional vectors, norms of either type may be applied to them. People's notation and language is all over the place. Below I've used $||\cdot||_{(\alpha)}$ to denote norms in the operator sense and $||\cdot||_{\alpha}$ in the vector sense.

\subsection{$||\mathbf{A}||_{(\alpha)}$ Operator Norm}
\label{sec:operatornorm}
The operator norm describes the largest change in size that it may impart on any of its inputs. That means that the operator norm is defined with respect to a definition of size in both domain and codomain. I.e., for an operator $\mathbf{A}$ and a given way of measuring size $||\cdot||_{\alpha}$:

\begin{equation}
||\mathbf{A}||_{(\alpha)} = \sup\left\{\frac{||\mathbf{A}\mathbf{v}||_{\alpha}}{||\mathbf{x}||_{\alpha}}: \mathbf{v} \in V\right\}
\end{equation}

When the operator is given by a matrix $\mathbf{A}$, and the length of the vector $\mathbf{x}$ is measured using the usual euclidian 2-norm ($||\cdot||_{2}$), then the operator norm is given by the square root of the largest eigenvalue of $\mathbf{A^T A}$. In that case, the operator norm is the same as the 2-norm (cf. section \ref{sec:2norm}).

To re-emphasize, $||A||_{(q)}$ and $||A||_q$ are two different things. The former measures the change in input size, where the size of the input is measured according to the latter. That is the reason for why the 1-Norm and 2-Norms are so different from the vector norms $L_1$ and $L_2$.

Operators that preserve the length of a vector with respect to some norm $||\cdot||_{\alpha}$ satisfy $||\mathbf{A}||_{(\alpha)} = 1$ and are called isometries (cf. section \ref{sec:isometric}).

% q-norm
\subsection{$||\mathbf{A}||_q$ $q$-Norms}
\label{sec:qnorms}

The $q$ norms for a matrix $\mathbf{A} \in \mathbb{R}^{m\times n}$ with entries $a_{i,j}$ in row $i$ and column $j$ are defined:

\begin{equation}
||\mathbf{A}||_q = \left(\sum_{i}\sum_{j} a^q_{i,j}\right)^{1/q}
\end{equation}

For $q=2$, this becomes the Frobenius norm (section \ref{sec:frobenius}). For vectors $\mathbf{v}\in\mathbb{R}^{n}$, the $q$-norm is more known as $p$-norm or $L^p$ norm (cf. section \ref{sec:lpnorms}).

% frobenius
\subsection{$||\mathbf{A}||_F$ Frobenius Norm}
\label{sec:frobenius}
The Frobenius Norm is the sum of the squares of all entries of a matrix. Let $\mathbf{A} \in \mathbb{R}^{m\times n}$ be a matrix with entires $a_{i,j}$ in row $i$ and column $j$, then:

\begin{equation}
||\mathbf{A}||_F = \sqrt{\sum_{i}\sum_{j} a^2_{i,j}}
\end{equation}

The Frobenius norm is invariant under rotations, and $||\mathbf{A}||_F = \sqrt{\sum_i \sigma_i^2}$ where $\sigma_i$ are the singular values of $\mathbf{A}$.

\subsection{$||\mathbf{A}||_{(1)}$ (1)-Norm}
Let $\mathbf{A}$ be a matrix with entires $a_{i,j}$ in row $i$ and column $j$, then:

\begin{equation}
||\mathbf{A}||_{(1)} = \max_{1\leq j \leq n} \sum^m_{i=1} |a_{i,j} |
\end{equation}

That is, it is the maximum of the sums of the absolute values of any of the columns of $\mathbf{A}$.

\subsection{$||\mathbf{A}||_{(\infty)}$ ($\infty$)-Norm}

Let $\mathbf{A}$ be a matrix with entires $a_{i,j}$ in row $i$ and column $j$, then:

\begin{equation}
||\mathbf{A}||_{(\infty)} = \max_{1\leq i \leq m} \sum^n_{j=1} |a_{i,j} |
\end{equation}

That is, it is the maximum of the sums of the absolute values of any of the rows of $\mathbf{A}$.

\subsection{$||\mathbf{A}||_{(2)}$ (2)-Norm}
\label{sec:2norm}

Let $\mathbf{A} \in \mathbb{R}^{m\times n}$ be a matrix with entires $a_{i,j}$ in row $i$ and column $j$, then:

\begin{equation}
||A||_{(2)} = \max_{\mathbf{x}\neq 0} \frac{||\mathbf{Ax}||_2}{||\mathbf{x}||_2}
\end{equation}

Which is the square root of the largest eigenvalue of $A^T A$. Or, equivalently, $||A||_{(2)} = \sigma_1$, where $\sigma_1$ is the largest singular value of the SVD of $\mathbf{A} = \mathbf{U\Sigma V}^T$. That means that $||A^{-1}|| = \frac{1}{\sigma_n}$, where $\sigma_n$ is the smallest singular value of the SVD of $\mathbf{A}$.

% nuclear norm
\subsection{$||\mathbf{A}||_{*}$ Nuclear Norm}
\label{sec:nuclearnorm}
The nuclear norm of a matrix is the sum of its singular values. This is equivalent to the Schatten-1 Norm. Optimization problems that involve rank restriction tend to be non-convex. The nuclear norm is a convex alternative that can make such problems tractable.

\begin{equation}
||\mathbf{A}||_{*} = \sum_i \sigma_i
\end{equation}


\subsection{Draw}

$draw(f,x)$ draws a graph of the function $f$ of $x$.
The second argument can be omitted when the dependent variable
is literally $x$ or $t$.
The vectors $xrange$ and $yrange$ control the scale of the graph.

{\color{blue}
\begin{verbatim}
draw(x^2)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.2]{parabola.png}
\end{center}

{\color{blue}
\begin{verbatim}
xrange = (-1,1)
yrange = (0,2)
draw(x^2)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.2]{parabola2.png}
\end{center}

\noindent
Parametric drawing occurs when a function returns a vector.
The vector $trange$ controls the parametric range.
The default is $trange=(-\pi,\pi)$.
In the following example, $draw$ varies $theta$
over the default range $-\pi$ to $+\pi$.

{\color{blue}
\begin{verbatim}
xrange = (-10,10)
yrange = (-10,10)
f = (cos(theta),sin(theta))
draw(5 f,theta)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.2]{circle.png}
\end{center}

\noindent
In the following example, $trange$ is reduced
to draw a quarter circle instead of a full circle.

{\color{blue}
\begin{verbatim}
trange = (0,pi/2)
f = (cos(theta),sin(theta))
draw(5 f,theta)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.2]{circle2.png}
\end{center}

\noindent
Here are a couple of interesting curves and the code for drawing them.
First is a lemniscate.

{\color{blue}
\begin{verbatim}
trange = (-pi,pi)
X = cos(t)/(1 + sin(t)^2)
Y = sin(t) cos(t)/(1 + sin(t)^2)
f = (X,Y)
draw(5 f,t)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.2]{lemniscate.png}
\end{center}

\noindent
Next is a cardioid.

{\color{blue}
\begin{verbatim}
r = (1 + cos(t))/2
u = (cos(t),sin(t))
xrange = (-1,1)
yrange = (-1,1)
trange = (0,2 pi)
draw(r u,t)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.2]{cardioid.png}
\end{center}


\chapter{Solving Quadratics}

A quadratic function has three terms: $ax^2 + bx + c$. $a$, $b$, and
$c$ are known as the \textit{coefficents}. The coefficients can be any
constant, except that $a$ can never be zero. (If $a$ is zero, it is a linear
function, not a quadratic.)

When you have an equation with a quadratic function on one side and a
zero on the other, you have a quadratic equation. For example:

$$72x^2 - 12x + 1.2 = 0$$

How can you find the values of $x$ that will make this equation true?

You can always reduce a quadratic equation so that the first
coefficient is 1, so that your equation looks like this:

$$x^2 +bx + c = 0$$

For example, if you are asked to solve $4x^2 + 8x - 19 = -2x^2 - 7$
\begin{multline*}
  4x^2 + 8x - 19 = -2x^2 - 7 \\
  6x^2 + 8x -12 = 0 \\
  x^2 + \frac{4}{3}x - 2 = 0
\end{multline*}
Here, $b = \frac{4}{3}$ and $c = -2$.

\begin{mdframed}[style=important]
$x^2 + bx + c = 0$ when
\begin{equation*}
x = -\frac{b}{2} \pm \frac{\sqrt{b^2 - 4c}}{2}
\end{equation*}
\end{mdframed}

What does this mean?

For any $b$ and $c$, the graph of $x^2 + bx + c$ is a parabola
that goes up on each end. Its low point is at $x = -\frac{b}{2}$.

If there are no real roots ($b^2 - 4c < 0$), that means the
parabola never gets low enough to cross the $x$-axis:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-0.5,xmax=2.75,
        ymin=-1,ymax=5,
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
      ]
      \addplot[no marks,sdkblue] expression[domain=-0.25:2.5,samples=100]{x^2 - 2* x + 3} node[above, xshift=-1cm] {$x^2 - 2x + 3$};
      \addplot[dashed,gray] coordinates {(1,-1)(1,3)};
    \end{axis}
\end{tikzpicture}

If there is one real root ($b^2 - 4c = 0$), it means that the parabola just touches the x-axis.

\begin{tikzpicture}
    \begin{axis}[
        xmin=-0.5,xmax=3.75,
        ymin=-0.5,ymax=5,
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
      ]
      \addplot[no marks,sdkblue] expression[domain=-0.25:3.5,samples=100]{x^2 - 4*x + 4} node[above, xshift=-1cm] {$x^2 - 4x + 4$};
      \addplot[dashed,gray] coordinates {(2,-0.5)(2,1)};
    \end{axis}
\end{tikzpicture}

If there are two real roots ($b^2 - 4c > 0$), it means that the parabola crosses the x-axis twice as it dips below and then returns:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-0.5,xmax=3.75,
        ymin=-2,ymax=5,
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
      ]
      \addplot[no marks,sdkblue] expression[domain=-0.25:3.5,samples=100]{x^2 - 3*x + 1} node[above, xshift=-1cm] {$x^2 - 4x + 4$};
      \addplot[dashed,gray] coordinates {(1.5,-2)(1.5,1)};
    \end{axis}
\end{tikzpicture}

\begin{Exercise}[title={Roots of a Quadratic}, label=solve_quadratic]

  In the last chapter, you found that the function for the height of your flying hammer is:

  $$p = -\frac{1}{2}9.8 t^2 + 12t + 2$$

  At what time will the hammer hit the ground?

\end{Exercise}
\begin{Answer}[ref=solve_quadratic]

  For what $t$ is  $-4.9 t^2 + 12t + 2 = 0$?  Start by dividing both sides of the equation by -4.9.

  $$t^2 - 2.45 t - 0.408 = 0$$

  The roots of this are at

  $$x = -\frac{b}{2} \pm \frac{\sqrt{b^2 - 4c}}{2} = -\frac{-2.45}{2} \pm \frac{\sqrt{(-2.45)^2 - 4(-0.408)}}{2} = 1.22 \pm 1.36$$

  We only care about the root after we release the hammer ($t > 0$).

  $1.22 + 1.36 = 2.58$ seconds after releasing the hammer, it will hit the ground.

\end{Answer}

\section{The Traditional Quadratic Formula}

I like the approach I just showed you. I find it easier to remember
and easier to prove than the traditional quadratic fomula, but you
should probably know the traditional quadratic formula.

\begin{mdframed}[style=important, frametitle={The Quadratic Formula}]

$ax^2 + bx + c = 0$ when
\begin{equation*}
  x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}
\end{equation*}

\end{mdframed}


\section*{Exercises}

\begin{ex}
  For each of the following matrices, find the eigenvectors and
  eigenvalues. Use this to visualize the linear transformation
  $T:\R^2\to\R^2$ that is described by the matrix.
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      1 & 2 \\
      2 & 1 \\
    \end{mymatrix},
    \quad
    B = \begin{mymatrix}{rr}
      1 & 0 \\
      2 & -1 \\
    \end{mymatrix},
    \quad
    C = \begin{mymatrix}{rr}
      1 & 1 \\
      0 & 2 \\
    \end{mymatrix},
    \quad
    D = \begin{mymatrix}{rr}
      1 & 1 \\
      1 & 1 \\
    \end{mymatrix},
    \quad
    E = \begin{mymatrix}{rr}
      1 & 0 \\
      -1 & 2 \\
    \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  If $A$ is the matrix of a linear transformation that rotates all
  vectors in $\R^2$ by $60^{\circ}$, explain why $A$ cannot have any
  real eigenvalues. Is there an angle such that rotation by this
  angle would have a real eigenvalue? What eigenvalues would be
  obtainable in this way?
  \begin{sol}
    A rotation by $60^{\circ}$ cannot map any non-zero vector to a
    scalar multiple of itself. The only rotations of $\R^2$ that have
    real eigenvalues are rotations by $180^{\circ}$ and $0^{\circ}$.
    The former has eigenvalue $-1$, and the latter has eigenvalue $1$
    (with all vectors being eigenvectors).
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be the linear transformation that reflects vectors about
  the $x$-axis. Find a matrix for $T$ and then find its eigenvalues and
  eigenvectors.
  \begin{sol}
    The matrix of $T$ is $\begin{mymatrix}{rr}
      1 & 0 \\
      0 & -1
    \end{mymatrix}$. The eigenvectors and eigenvalues are:
    \begin{equation*}
      \set{\begin{mymatrix}{c}
          0 \\
          1
        \end{mymatrix}} ~\mbox{for eigenvalue}~ {-1},
      \quad\set{\begin{mymatrix}{c}
          1 \\
          0
        \end{mymatrix}} ~\mbox{for eigenvalue}~ {1}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be the linear transformation that reflects vectors about the
  line $x=y$. Find a matrix of $T$ and then find eigenvalues and
  eigenvectors.
  \begin{sol}
    The matrix of $T$ is $\begin{mymatrix}{rr}
      0 & 1 \\
      1 & 0
    \end{mymatrix}$. The eigenvectors and eigenvalues are:
    \begin{equation*}
      \set{\begin{mymatrix}{c}
          1 \\
          1
        \end{mymatrix}} ~\mbox{for eigenvalue}~ {-1},
      \quad\set{\begin{mymatrix}{c}
          -1 \\
          1
        \end{mymatrix}} ~\mbox{for eigenvalue}~ {1}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be the linear transformation that reflects all vectors in
  $\R^3$ about the $xy$-plane. Find a matrix for $T$ and then
  obtain its eigenvalues and eigenvectors.
  \begin{sol}
    The matrix of $T$ is $\begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & -1
    \end{mymatrix}$
    The eigenvectors and eigenvalues are:
    \begin{equation*}
      \set{\begin{mymatrix}{c}
          0 \\
          0 \\
          1
        \end{mymatrix}} ~\mbox{for eigenvalue $-1$},
      \quad\set{\begin{mymatrix}{c}
          1 \\
          0 \\
          0
        \end{mymatrix},
        \begin{mymatrix}{c}
          0 \\
          1 \\
          0
        \end{mymatrix}} ~\mbox{for eigenvalue $1$}.
    \end{equation*}
  \end{sol}
\end{ex}


\textbf{Consider the first order linear initial boundary value problem
\begin{align*}
u_t=u_x, ~~~~x\in[-1,1],~~t>0,~~u(\pm 1,t)=u(x,0)=0,
\end{align*}
with initial data $u(x,0) = \exp(-60(x-1/2)^2)$. Write a program to solve this problem by a matrix-based Chebyshev spectral discretization in $x$ coupled with the third order Adams-Bashforth formula in $t$,
\begin{align*}
u^{(n+3)} = u^{(n+2)}+\frac{1}{12}\Delta t\left(23f^{(n+2)}-16f^{(n+1)}+5f^{(n)}\right).
\end{align*}
Initial values can be supplied from the exact solution. Take $N=50$ and \newline $\Delta t=\nu N^{-2}$, where $\nu$ is a parameter. For each of the two choices $\nu = 7$ and $\nu = 8$, produce one plot of the computed solution at $t=1$ and another that superimposes the stability region in the $\lambda\Delta t$-plane, the eigenvalues of the spatial discretization matrix, and its $\epsilon$-pseudospectra fpr $\epsilon=10^{-2},10^{-3},\dots,10^{-6}$. Comment on the results.
}
\newline

We start by finding the exact solution using characteristics,

\begin{align*}
u(x,t) =
\begin{cases}
       e^{-60(x+t-1/2)^2},& x\geq -t,\\
       e^{-60(x+t+1/2)^2},& x\leq -t.
\end{cases}
\end{align*}

Further, we use Chebyshev differentiation matrices,
\begin{align*}
u_t &= u_x,\\
u_t &= Du = f,
\end{align*}
and the AB time-stepping method:
\begin{align*}
u^{(n+3)} &= u^{(n+2)}+\frac{1}{12}\Delta t\left(23f^{(n+2)}-16f^{(n+1)}+5f^{(n)}\right),\\
&= u^{(n+2)}+\frac{1}{12}\Delta tD\left(23u^{(n+2)}-16u^{(n+1)}+5u^{(n)}\right).
\end{align*}
To start the simulation we use the exact solution. The results obtained are shown in the following figure. In the top two figures we see the exact solution (red) superposed with the numberical solution (blue). As we can see, for $\nu=7$ (left) we have a correct numerical solution while for $\nu=8$ (right) the numerical solution blows up. This can be further understood looking at the bottom figures. As we can see in the bottom left graph, the eigenvalues near the border of the stability region are susceptible to perturbations. Hence, by just increasing $\nu$ to 8, those eigenvalues are now outside of the stability region (bottom-right figure), giving us an understanding of why the numerical simulation is unstable.
\begin{figure}[H]
\centering
\includegraphics[scale=1.2]{P2.png}\caption{Numerical and analytical solution of the PDE for $\nu = 7$ and $\nu = 8$ (top) and pseudo-spectra with stability regions and eigenvalues in the $\lambda\Delta t$-plane for those two values of $\nu$ (bottom).}
\end{figure}

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Homework 4, Problem 2 - Francisco Castillo'
clear all; close all; clc;
labelfontsize = 14;
markersize = 4;
linewidth = 2;

N = 50;
V = [7,8];
levels = log10([1e-2 1e-3 1e-4 1e-5 1e-6]);
% figure('units','normalized','outerposition',[0 0 1 1])
for j=1:length(V)
    v = V(j);
    dt = v/N^2;
    tf = 1;

    [D,x] = cheb(N); % D:(N+1)x(N+1), x:(N+1)x1
    D = D (2:end,2:end);
    u_n = analytic_sol(x(2:end),0);
    u_n1 = analytic_sol(x(2:end),dt);
    u_n2 = analytic_sol(x(2:end),2*dt);

    t = 2*dt;
    while t<tf
        if (t+dt>tf)
            dt=tf-t;
            t=t+dt;
        else
            t=t+dt;
        end

        u_n3 = u_n2+dt*D*(23*u_n2-16*u_n1+5*u_n)/12;
        u = [analytic_sol(1,t);u_n3];

        u_n = u_n1;
        u_n1 = u_n2;
        u_n2 = u_n3;

        res = analytic_sol(x,t);

        subplot(2,2,j)
        plot(x,u,'b*')
        hold on
        plot(x,u,'b')
        plot(x,res,'r*')
        plot(x,res,'r')
        grid on
        axis([-1 1 0 1])
        xlabel('$x$','interpreter','latex','fontsize',labelfontsize)
        ylabel('$u(x,t)$','interpreter','latex','fontsize',labelfontsize)
        hold off
        shg
    end
    %%
    % Adams-Bashforth stability region and pseudo-spectra of this problem
    dt = v/N^2;
    ee = eig(D);
    dtee = dt*ee;
    xr = 6.5*linspace(min(real(dtee)),0.1,50);
    yr = 2*linspace(min(imag(dtee)),max(imag(dtee)),50);
    [xx,yy] = meshgrid(xr,yr);
    zz = xx+1i*yy;
    ps = 0*zz;
    for k=1:numel(zz)
        ps(k) = min(svd(eye(size(D))*zz(k)-dt*D));
    end

    z = exp(1i*pi*(0:200)/100); r = z-1;
    s = (23-16./z+5./z.^2)/12;

    subplot(2,2,j+2)
    contourf(xx,yy,log10(ps),levels)
    c = colorbar;
    hold on
    plot([-8,8],[0,0],'k')
    plot([0,0],[-8,8],'k')
    plot(real(dtee),imag(dtee),'r*','markersize',markersize)
    plot(r./s,'b','linewidth',linewidth)
    xlabel('$\Re(\lambda)$','interpreter','latex','fontsize',labelfontsize)
    ylabel('$\Im(\lambda)$','interpreter','latex','fontsize',labelfontsize)
    axis([-1 0.1 -1 1])
    grid on
end

saveas(gcf,'Latex/FIGURES/P2','png')

function res = analytic_sol(x,t)
    res = zeros(size(x));
    res(x>=-t) = exp(-60*(x(x>=-t)+t-0.5).^2);
    res(x<-t) = exp(-60*(x(x<-t)+t+0.5).^2);
end
\end{verbatim}


\subsubsection{Dimension Reduction using PCA}
\label{sec:reducpca}
Principal Component Analysis (PCA) is a mathematical tool used in data analysis to simplify and extract the most relevant features in a complex data set. This technique is used to create entries composed of linearly independent `principal components'. These are the eigenvectors of the covariance matrix $\Sigma$ applied to the centered data, eq.\eqref{eq:covariance}, ordered from the largest to the smallest eigenvalue, $\lambda_1 \ge \lambda_2 \ge ... \ge \lambda_n$, where $\overline{\boldsymbol{X}}$ is the mean value of each original feature, eq.\eqref{eq:xmean}. The projection of the data onto the principal component space ensures a maximal variance on the direction of the first component. Each subsequent principal component is orthogonal to the previous ones and points in the direction of maximal variance in the residual sub-space \citep{Shlens2014}.

\begin{align}
\overline{\boldsymbol{X}} & = \frac{1}{m} \sum_{i=1}^{m} \boldsymbol{X}_i \label{eq:xmean} \\
\Sigma & = \frac{1}{m} \sum_{i=1}^{m} \left( \boldsymbol{X}_i - \overline{\boldsymbol{X}} \right)\left( \boldsymbol{X}_i - \overline{\boldsymbol{X}} \right) \label{eq:covariance}
\end{align}

The PCA transformation creates as many components in the transformed space, $\boldsymbol{\tilde{X}}$, as features in the original data space $\boldsymbol{X}$. However, components with small eigenvalues belong to a dimension where the variance is so small that it is impossible to separate points in the data. It is a general practice in data reduction to keep only the first $k$ components that explain at least a significant portion of the total variance of the data, $\lambda_{i=1..k}/\text{Tr}(\Sigma) > \epsilon$. This allows for a selection of information that will effectively differentiate data points, and for a reduction of the amount of data to process during analysis. Many techniques have been suggested for the selection of the values of $k$ and the cut-off $\epsilon$ \citep{Rea2016}. We use the value of $k=3$ to simplify the comparison among the different models, but for a detailed study of the solar wind, if a PCA transformation is applied, it is important to use a fixed criteria for the selection of the cut-offs.

Fig. \ref{fig:dimreduc} (A) is a 3D scatter plot of all the data points, colored by the Xu classification, projected on the first three PCA components. The features used to create this figure are presented in section \ref{sec:fourmodels}. Panel (B) contains the same data colored by the Zhao classification. These projections show that the Xu and Zhao classification are defined by hyper-planes separating the points, even if the data has been linearly transformed by the PCA. Class 2, ICMEs-ejecta, is restricted to a small domain in this coordinate system (for both the Xu and Zhao classification). The lateral plots on panel (A) are 2D histograms of the point distribution on the three main PCA planes. They show that the concentration of points is not homogeneous and different zones can be isolated using unsupervised classification techniques. There is a clear segregation of points in the (1st,2nd)-component plane: as we will see in subsequent sections, one of the features of the solar wind presents a strong bimodal distribution that is prioritized by the PCA.

\subsubsection{Dimension Reduction Using Autoencoders}
\label{sec:reducae}
PCA has a limitation: the principal components are a linear combination of the original properties of the solar wind. An alternative to data reduction is the use of autoencoders (AE). These are machine learning techniques that can create non-linear combinations of the original features projected on a latent space with less dimensions \citep{Hinton2006}. This is accomplished by creating a system where an encoding function, $\phi$, maps the original data $\boldsymbol{X}$ to a latent space, $\boldsymbol{\mathcal{F}}$, eq.\eqref{eq:encoder}. A decoder function, $\psi$, then maps the latent space back to the original input space, eq.\eqref{eq:decoder}. The objective of the autoencoder is to minimize the error between the original data and the data produced by the compression-decompression procedure as shown in eq.\eqref{eq:aeminimization}.

\begin{align}
\phi: & \boldsymbol{X} \rightarrow \boldsymbol{\mathcal{F}} \label{eq:encoder}\\
\psi: & \boldsymbol{\mathcal{F}} \rightarrow \boldsymbol{X} \label{eq:decoder} \\
\phi,\psi = & \underset{\phi,\psi}{\arg \min} \left\lVert \boldsymbol{X} - (\phi \circ \psi) \boldsymbol{X} \right\rVert^2 \label{eq:aeminimization}
\end{align}

Autoencoders can be represented as feed-forward neural networks, where fully connected layers lead to a central bottleneck layer with few nodes and then expands to reach again the input layer size. An encoded element, $\boldsymbol{z} \in \boldsymbol{\mathcal{F}}$, can be obtained from a data entry, $\boldsymbol{x} \in \boldsymbol{X}$, following the standard neural network function, eq.\eqref{eq:encodex}, where $\boldsymbol{W}$ is the weights matrix, $\boldsymbol{b}$ is the bias, and $\sigma$ is the non-linear activation function.

\begin{align}
\boldsymbol{z} & = \sigma \left( \boldsymbol{W}\boldsymbol{x} + \boldsymbol{b} \right) \label{eq:encodex} \\
\boldsymbol{\hat{x}} & = \sigma' \left( \boldsymbol{W'}\boldsymbol{z} + \boldsymbol{b'} \right) \label{eq:decodez} \\
\mathcal{L}(\boldsymbol{x}, \boldsymbol{\hat{x}}) & =  \left\lVert \boldsymbol{x}- \boldsymbol{\hat{x}} \right\rVert^2 \label{eq:aeloss}
\end{align}

The decoding procedure, shown in eq.\eqref{eq:decodez}, transforms $\boldsymbol{z}\rightarrow\boldsymbol{\hat{x}}$, where the prime quantities are associated with the decoder. The loss function, $\mathcal{L}(\boldsymbol{x}, \boldsymbol{\hat{x}})$, is the objective to be minimized by the training of the neural network using gradient descent. Once training is completed, the vector $\boldsymbol{z}$ is a projection of the input vector $\boldsymbol{x}$ onto the lower dimensional space $\boldsymbol{\mathcal{F}}$.

Additional enhancements and variations of this simple autoencoder setup exist in the literature, including multiple regularization techniques to minimize overfitting \citep{7407967}, Variational Autoencoders (VAE) that produce encoded Gaussian distribution functions \citep{Kingma2013}, and Generative Adversarial Networks (GAN) that produce new (unseen) data \citep{Goodfellow2014}. In this work we use the most basic form of autoencoders, presented above.

The second column of Fig.\ref{fig:dimreduc}, panels (C) and (
D), contains the same information as the first column, but with the data set encoded in the three dimensional latent space $\boldsymbol{\mathcal{F}}$. Panel (C) shows that all the classes in the Xu and Zhao classification are easy to distinguish, including ICMEs-ejecta (class 2) that is difficult to discern in the PCA. This projection also shows that class 4 from the Zhao classification in the bottom panels, overlaps with class 3 (sector reversal origin), and partially with class 2 (ejecta) in the Xu classification on the top panels. Panel (C) shows, on the side planes, 2D histograms of the density of points. These can be seen as the volume integral of the point density in each direction. Here again it is possible to observe multiple zones of high concentration, suggesting that multiple types of solar wind are present in the data and that they can be differentiated using an unsupervised classification technique.

\subsubsection{Clustering Techniques}
\label{sec:clustering}
The goal of unsupervised machine learning is to group data points in a limited number of clusters in the N-dimensional space $\Omega\in\Bbb R^N$, where N is the number of features (components or properties) in the data set. Multiple techniques can be used to perform multi-dimensional clustering. We present in Fig. \ref{fig:clustering} the three clustering techniques used to classify our 3D reduced data. The panels in the first column show the data projected in the PCA reduced space, $\boldsymbol{\tilde{X}}$, while the second column shows the data in the latent AE space, $\boldsymbol{\mathcal{F}}$. Each row corresponds to a different clustering method. The colors in the top panels (A) and (D) were obtained using the $k$-means method \citep{1056489}, the colors in the middle panels (B) and (E) were obtained using the Gaussian Mixture Model (GMM) \citep{bishop2006machine}. The bottom panels are colored by the classes from the Self-Organizing Maps described later in section \ref{sec:som}.

The $k$-means technique has already been used in a recent publication for the determination of solar wind states \citep{Roberts2020}. To our knowledge other clustering methods have never been used in the literature to classify the solar wind, but \citep{Dupuis2020} has used the GMM to characterize magnetic reconnection regions in simulations using their velocity distribution information.

The colors used in Fig.\ref{fig:clustering} are assigned randomly by each clustering technique. The most glaring issue with them is that different methods can lead to different clusters of points. The GMM and the $k$-means agree on their classification in the PCA space, but show dissimilar results in the AE space. Moreover, for a single method, e.g. $k$-means, slight modifications of the clustering parameters, e.g. using a different seed for the random number generator, can lead to very different results. We address this last issue using an algorithm that launches the $k$-means and GMM algorithms 500 times until the methods converge to a quasi-steady set of clusters. But we warn that the results are implementation dependent.

In the present data set, the cloud of points is convex and well distributed in all three components. This raises one additional issue, observed more clearly in the first column of Fig.\ref{fig:clustering}: when classical clustering methods are applied to relatively homogeneously dense data, it divides the feature space in Vorono\"i regions with linear hyper-plane boundaries. This is an issue with all clustering techniques based on discrimination of groups using their relative distances (to a centroid or to the mean of the distribution). To avoid this problem density-based techniques, such as DBSCAN \citep{ester1996density}, and agglomeration clustering methods, use a different approach. However, we can not apply them here because in such homogeneous cloud of points these techniques lead to a trivial solution where all data points are assigned to a single class.

There is no guarantee that a single classification method, with a particular set of parameters will converge to a physically meaningful classification of the data if the points in the data do not have some level of separability, or have multiple zones of high density. This is also true for other classification methods based on `supervised learning'. In those applications same issues will be observed if the training data uses target classes derived from dense data clouds using simple hyper-plane boundaries, as done for the Zhao and Xu classes. An example of such application was published by \citep{Camporeale2017b}. The authors used the Xu classification to train a Gaussian Process classifier.

\subsubsection{Self-Organizing Maps}
\label{sec:som}

\paragraph{Classical SOM}

Following the definitions and notations by \citep{Villmann2006}, a class can be defined as $C_i\overset{\text{def}}{=} \{x\in\Omega | \Phi(x)=\boldsymbol{w}_i\}$, where $\Phi$ is a function from $\Omega$ to a finite subset of $k$ points $\{\boldsymbol{w}_i\in\Bbb R^N\}_{i=1..k}$. A cluster $C_i$ is then a partition of $\Omega$, and $\{\boldsymbol{w}_i\}$ are the code words (also known as nodes, weights or centroids) associated. The mapping from the data space to the code word set, $\Phi: \Omega\rightarrow\mathcal{W}$, is obtained by finding the closest neighbor between the points $\boldsymbol{x}$ and the code words $\boldsymbol{w}$, eq.\eqref{eq:winner}. The code word $\boldsymbol{w_s}$, the closest node to the input $\boldsymbol{x_s}$, is called the `winning element'. The class $C_i$ corresponds to a Vorono\"i region of $\Omega$ with center in $\boldsymbol{w}_i$.

\begin{equation}
\Phi: x \rightarrow  \underset{i\in\mathcal{N}}{\arg\min}\left( \left\lVert x - \boldsymbol{w}_i \right\rVert \right) \label{eq:winner}
\end{equation}

A Self-Organizing Map (SOM) also composed of structured nodes arranged in a lattice, and assigned to a fixed position $\boldsymbol{p}_i$ in $\Bbb R^q$, where $q$ is the dimension of the lattice (generally $q=2$). The map nodes are characterized by their associated code words. The SOM learns by adjusting the code words $\boldsymbol{w}_i$ as input data $x$ is presented.

The SOM is the ensemble of code words and nodes $A_i = \{ \boldsymbol{w}_i, \boldsymbol{p}_i\} \in (\Omega \times \Bbb R^q)$. For a particular entry $\boldsymbol{x_s}$, the code word $s \in \mathcal{N}$ is associated to the winning node $\boldsymbol{p_s}$ if the closest word to $\boldsymbol{x_s}$ is $\boldsymbol{w_s}$. At every iteration of the method, all code words of the SOM are shifted towards $x$ following the rule:

\begin{equation}
\Delta \boldsymbol{w}_i = \epsilon(t)h_\sigma(t,i,s)(x-\boldsymbol{w}_i) \label{eq:learnsom}
\end{equation}

with $h_\sigma(t,i,j)$ defined as the lattice neighbor function:

\begin{equation}
h_\sigma(t,i,j) = e^{-\frac{\left\lVert \boldsymbol{p}_i - \boldsymbol{p}_j \right\rVert^2}{2\sigma^2(t)}} \label{eq:neigsom}
\end{equation}

where $\epsilon(t)$ is the time dependent learning rate, eq.\eqref{eq:epsilon}, and $\sigma(t)$ is the time dependent lattice neighbor width, eq.\eqref{eq:sigma}. The training of the SOM is an iterative process where each data point in the data set is presented to the algorithm multiple times $t={0, 1,..,t_f}$. In these equations the subscript $0$ refers to initial values at $t=0$ and the subscript $f$ to values at $t=t_f$.

\begin{align}
\epsilon(t) & = \epsilon_0 \left(\frac{\epsilon_f}{\epsilon_0}\right)^{t/t_f} \label{eq:epsilon} \\
\sigma(t) & = \sigma_0 \left(\frac{\sigma_f}{\sigma_0}\right)^{t/t_f} \label{eq:sigma}
\end{align}

This procedure places the code words in the data space $\Omega$ in such a way that neighboring nodes in the lattice are also neighbors in the data space. The lattice can be presented as a $q$-dimensional image, called map, where nodes sharing similar properties are organized in close proximity.

The main metric for the evaluation of the performance of the SOM is called the quantization error:

\begin{equation}
Q_E = \frac{1}{M} \sum_{i=1}^M \left\lVert x_i - w_{x_i} \right\rVert
\end{equation}

where $M$, is the total number of entries in the data set.

Once the training of the SOM is finished, the code words $\boldsymbol{w}_i$ can be grouped together using any clustering technique, e.g. k-means. The nodes of the SOM with close properties will be made part of the same class. The classes thus created are an ensemble of Vorono\"i subspaces, allowing a complex non-linear partitioning of the data space $\Omega$.

The final number of clusters is an input of the algorithm, but can also be calculated autonomously. The Within Cluster Sum of Squares (WCSS) can be used as a metric of the compactness of the clustered nodes. As its name implies the WCSS is the sum of the squred distances from each node to their cluster point. If only one class is selected, the large spread of the nodes would produce a high WCSS. The lowest possible value of the WCSS is obtained for a very high number of classes, when the number of classes is equal to the number of nodes. But such extreme solution is also unpractical. The optimal number of clusters can be obtained using the Kneedle class number determination \citep{5961514}. In the present work we do not use this technique: we will perform comparisons with previous publications that propose a fixed number of solar wind types. We will explore the use of an automatic class number selection in a future publication.

\paragraph{Dynamic SOM}
The time dependence of the SOM training allows the code words $\boldsymbol{w}_i$ to reach steady coordinates by slowing down their movement over the iterations. Due to the minimization of the distance in eq.\eqref{eq:winner} code words tend to agglomerate around high density zones of the feature space. The Dynamic Self-Organizing Map (DSOM), introduced by \citep{Rougier2011}, eliminate the time dependence and allows to cover larger zones of the space outside of the high density regions.

The DSOM is a variation of the SOM where the learning function \eqref{eq:learnsom} and the neighbor function \eqref{eq:neigsom} are replaced by eqs. \eqref{eq:learndsom} and \eqref{eq:neigdsom} respectively:

\begin{align}
\Delta \boldsymbol{w}_i & = \epsilon \left\lVert x - \boldsymbol{w}_i \right\rVert_\Omega h_\eta(i,s,x)(x-\boldsymbol{w}_i) \label{eq:learndsom} \\
h_\eta(i,s,x) & = e^{-\frac{1}{\eta^2}\frac{\left\lVert \boldsymbol{p}_i - \boldsymbol{p}_j \right\rVert^2}{\left\lVert x - \boldsymbol{w}_s \right\rVert_\Omega^2}} \label{eq:neigdsom}
\end{align}

where $\epsilon$ is a constant learning rate, $h_\eta(i,s,x)$ is defined as the new lattice neighbor function, and $\eta$ is the `elasticity' parameter. In their work \citep{Rougier2011} show that DSOM can be used to better sample the feature space $\Omega$, reducing the agglomeration of code words around high density zones. The DSOM does not converge to a steady solution, due to the lack of a temporal damping factor.

\paragraph{Visualization of SOM and DSOM}
Clustering techniques do not necessarily convergence to a steady immutable solution. Differences in the training parameters or slight changes in the data can have an important impact on the final classification. These tools can be used for statistical analysis, comparisons, data visualization and training of supervised methods. But it will be practically impossible to claim the existence of a general objective set of states discovered only by the use of these basic clustering techniques.

However, SOMs and DSOMs provide an important tool for the study of the solar wind: the maps are composed of nodes that share similar properties with its immediate neighbors. This allows for visual identification of patterns and targeted statistical analysis.

Fig.\ref{fig:maps} shows the basic types of plots and maps that can be generated using the SOM/DSOM techniques. This figure uses data from the model Amaya-21 which has been encoded, using AE, into a set of entries, $\boldsymbol{z}_i$, each one composed of three components. Panel (A) shows a histogram of the first two components of the feature space $\Omega$, with dots marking the position of the code words $\boldsymbol{w}_i$. The colors of the dots represent their SOM classification. The red lines connect a single code word $\boldsymbol{w}_s$ with its six closest neighbors. The panel (B) shows the `hit map' of the SOM. It contains the lattice nodes $\boldsymbol{p}_i$ associated to the code words $\boldsymbol{w}_i$. They are depicted as hexagons with sizes representing the number of data points connected to each node and colored by their SOM class. The thickness of the lines between lattice nodes represent the relative distance to its neighbors in the feature space $\Omega$. Red lines connect the node $\boldsymbol{p}_s$, associated to the code word $\boldsymbol{w}_s$ in panel (A), to its closest neighbors.

Panel (C) of Fig.\ref{fig:maps} corresponds to the value of a single feature associated to each node; as an example we use the ionized oxygen ratio $O^{7+}/O^{7+}$ (`O7to6'). To improve visualization all hexagon sizes have been set to their maximum and the inter-node distance line has been colored white. In order to obtain the correct value for each node, we must first perform a decoding of the data from the latent space, $\Omega=\boldsymbol{\mathcal{F}}$, to the original data set space, $\boldsymbol{X}$.

Panel (D) of Fig.\ref{fig:maps} shows that the nodes of the lattice can also be used to present data that has not been used in the training of the SOM. The method keeps track of the points in the data set associated to each lattice node, it is then possible to perform independent statistical operations on those points alone. Moreover, it is possible to activate the SOMs with just a subset of the data, i.e. with points that feature a specific solar wind type. In this case, as an example, we have colored the map using the average oxygen charge state $\left<Q_{O}\right>$ (`avqO'), and we have set the size of the nodes to represent the frequency of points with solar wind type Xu=2 (ejecta). The dark line between the lattice nodes designate the boundaries between different SOM classes.

These four representations are only a few examples of the variety of data that can be represented using SOMs. The most important aspect of the SOMs is that data is represented in simple 2D lattices where the nodes share properties with their neighbors. Here we also decided to use hexagonal nodes, connecting 6 equidistant nodes, but other types of representations are also valid, e.g. squared or triangular nodes.

The bottom row of Fig.\ref{fig:maps} displays all three components of the code words $\boldsymbol{w}_i$ associated with each one of the $\boldsymbol{p}_i$ nodes. In the first panel they have been mapped to the basic colors Red, Green and Blue (RGB). The remaining panels have been colored using each individual component. The first panel is then the RGB composition of the three remaining ones where the boundaries between the SOM classes have been highlighted.

\subsubsection{The Full Architecture}
\label{sec:fullarchi}
The previous sections introduced all the individual pieces that we use for the present work. Here we give a global view of the full model. Fig.\ref{fig:architecture} shows how all the components are interconnected. The data set is composed of clean and processed entries. We tested the PCA transformation in cases Amaya-21 and Roberts-8, keeping only the first three principal components. This possible setup is presented on the left of the figure. It is also possible to perform an unsupervised clustering directly on the un-processed data, as shown on the top of the figure, but it is not recommended. For the remaining of this manuscript we present only the cases where the non-linear AE encoding, shown at the right of Fig.\ref{fig:architecture}. The bottleneck of the AE network is three nodes, i.e. the data is encoded in three components. The transformed data is then used to train the SOM.

After training, the code words of the SOM are then clustered to group together nodes that share similar properties. This second level classification is done using the $k$-means++ algorithm with 500 re-initializations (it is in general recommended to use between 100 and 1000 iterations). The total number of classes selected is an input of the model and has been set to 8. This arbitrary choice was made following the results presented by \citep{Roberts2020}. All the software was implemented in Python using as main libraries PyTroch, Scikit-learn and Pandas.

\paragraph{Autoencoder architecture}
We use a basic, fully connected feed-forward neural network for the encoding-decoding process. The bottleneck of the network has been fixed to three neurons in order to simplify the visualization. This arbitrary choice is another parameter of the models that need further investigation. The neural network is symmetric in size but the weights of the encoder, $\boldsymbol{W}$, and the decoder, $\boldsymbol{W'}$, are not synchronized (see eqs.\eqref{eq:encodex}, \eqref{eq:decodez}). We use multiple fully connected hidden layers, where the central layer is the size of the bottleneck. Each layer is composed of a linear regressor, followed by batch normalization and a ReLU activation function. The output layer of the network contains a linear regressor followed by a hyperbolic tangent activation function. The autoencoder has been coded in python using the PyTorch framework.

We use an Adam optimizer \citep{Kingma2014} for the gradient descent with a learning rate of 0.001 and a weight decay of 0.0001 for regularization. The loss function is the Mean Squared Error (MSE). We train the network for 30 epochs, after which we see no additional improvement in the loss function. The full data set was randomly divided 50\%/50\% between training and testing sets.

\paragraph{Two Models of Solar Wind Classification}
\label{sec:fourmodels}
We have tested the two models presented in Table \ref{tab:features}. The models are inspired by the work of \citep{Roberts2020}. We call these cases Amaya-21 and Roberts-8.  The table lists all the features used in each model. A detailed description of each feature can be found in the \href{http://www.srl.caltech.edu/cgi-bin/dib/rundibviewmultil2/ACE/ASC/DATA/level2/multi}{ACE Level 2 documentation}. To spread the data over a larger range of values in each component, we have used the logarithm of all the quantities, except of those marked with an asterisk in the table.

Features 15 to 20 contain an additional suffix, corresponding to a statistical operation performed on the corresponding feature. The operations include the mean, the range, the standard deviation and the auto-correlation of quantities over a window of time of 6 hours. This window allows to capture temporal (spatial) fluctuations in some of the solar wind parameters.

On the lower part of Table \ref{tab:features} we present the range of dates used for each model. For Amaya-21 we use the full data set, while for the model Roberts-8 we try to replicate as much as possible the choices made in \citep{Roberts2020}. The same table also contains the hyper-parameters selected to run the two models. The number of neurons per layer in the encoding half of the neural network is listed in the table and was manually selected to minimize the final loss value of the AE.

All the figures presented until now correspond to the processing of data from model Amaya-21. The amount of data and figures produced in this work is very large and is not possible to include all of them in the present document. We will present in the next section some highlights, but more detailed analysis of each one of the cases will be presented in future publications.

\paragraph{Budget}
Machine learning models require fine tuning of different parameters, from the selection and testing of multiple methods, to the parameterization of the final architecture. \citep{Dodge2019} suggests that every publication in machine learning should include a section on the budget used for the development and training of the method. The budget is the amount of resources used in the data processing, the selection of the model hyper-parameters (HP), and its training.

The most time-consuming task in the present work has been the data preparation, the model setup and debugging and the writing of the SOM visualization routines. All the techniques described in the previous sections have been coded in python and are freely accessible in the repositories listed in section \ref{sec:repos}. We estimate the effort to bring this work from scratch to a total of 2 persons month. Of these, one person week was dedicated to the manual testing an selection of different model HPs (autoencoder architecture, feature selection, learning rates, initialization methods, number of epochs for training, selection of data compression method, size of the time windows, etc.).

All classic clustering techniques presented in section \ref{sec:clustering} require only a few lines of code and can be trained in minutes on a mid-range workstation (e.g. Dell Precission T5600, featuring two Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz with four cores and eight threads each). The most time consuming tasks of our models are the training of the autoencoder (5\% od the total run time), the multiple passages of the clustering algorithms (15\% of the run time), and the optimization of the SOM hyper-parameters (80\% of the run time). The training of the SOM is performed in less than a minute.

For reference, the total run-time for each one of the models used in this work are: 60 minutes for the Amaya-21 model and 20 minutes for the Roberts-8 model.

\paragraph{Hyper-Parameter Optimization}
Our main goal in this manuscript is to introduce the use of the SOMs for the classification of solar wind data. SOMs require the selection of four main Hyper-Parameters (HPs): the size of the lattice, $(m\times n)$, the initial learning rate, $\epsilon_0$, and the initial neighbor radius, $\sigma_0$. In the case of the DSOM algorithm, these two last HPs are replaced by the constant learning rate, $\epsilon$, and the elasticity, $\eta$. The automatic selection of the best HP for machine learning model is called Hyper-Parameter Optimization (HPO).

We use the library `optuna' \citep{Akiba2019} to perform an automatic optimization of the four HPs. The optimization is based on a technique called Tree-structured Parzen Estimator (TPE) \citep{pmlr-v28-bergstra13}. The objective function of the optimizer was set to minimize the expression:

\begin{equation}
\mathcal{H}\left(\sigma, \eta, m, n\right) = Q_E(\sigma, \eta, m, n) + \alpha \frac{m}{m_{max}} + \beta \frac{n}{n_{max}} + \gamma m n \label{eq:hpo}
\end{equation}

where $Q_E$ is the SOM quantization error, $m$ and $n$ are the number of lattice nodes in each dimension, and $m_{max}$ and $n_{max}$ are the given maximum number of possible nodes. The weight factors $\alpha$, $\beta$ and $\gamma$ are used to impose more or less restictions on each term. This optimization objective allows to minimize the quantization error, but also the number of nodes in the SOM. The squaring term $\gamma m n$ forces the map to be as squared as possible.

After a total of 100 trial runs of the model using different HPs, the optimizer selected the parameters presented in the lower section of table \ref{tab:features}. The total run time for the HPO of the case Amaya-21 is about 40 minutes. HPO is, understandably, one of the most expensive procedures in all our setup.


\section{Logistic Regression}
We can extend this learning to classification problems, where we have binary
labels $y$ that are either $0$ or $1$.
\subsection{The Logistic Function}
For logistic regression, our new hypothesis for estimating the class of a
sample $x$ is:
\begin{equation}
  h( x ) = g \left( \theta ^ { T } x \right) = \frac { 1 } { 1 + e ^ { - \theta ^ { T } x } }
\end{equation}
where $g(z)$ is the logistic or sigmoid function:
\begin{equation}
  g ( z ) = \frac { 1 } { 1 + e ^ { - z } }
\end{equation}
The sigmoid function is bounded between $0$ and $1$, and tends towards $1$ as
$z \rightarrow \infty$. It tends towards $0$ when $z \rightarrow -\infty$.
A useful property of the sigmoid function is the form of its derivative:
\begin{equation}
  \begin{aligned} g ^ { \prime } ( z ) \quad & = \quad \frac { d } { d z } \frac { 1 } { 1 + e ^ { - z } } \\
    & = \quad \frac { 1 } { \left( 1 + e ^ { - z } \right) ^ { 2 } } \left( e ^ { - z } \right) \\
    & = \quad \frac { 1 } { \left( 1 + e ^ { - z } \right) } \cdot \left( 1 - \frac { 1 } { \left( 1 + e ^ { - z } \right) } \right) \\
    & = \quad g ( z ) ( 1 - g ( z ) )
  \end{aligned}
\end{equation}
\subsection{Cost Function}
To fit $\theta$ for a set of training examples, we assume that:
\begin{equation}
  \begin{aligned}
    P ( y = 1 | x ; \theta ) & = h( x ) \\
    P ( y = 0 | x ; \theta ) & = 1 - h( x )
  \end{aligned}
\end{equation}
This can be written more compactly as:
\begin{equation}
  p ( y | x ; \theta ) = \left( h ( x ) \right) ^ { y } \left( 1 - h ( x ) \right) ^ { 1 - y }
\end{equation}
Assume $m$ training examples generated independently, we define the likelihood
function of the parameters as:
\begin{equation}
  \begin{aligned}
    L ( \theta ) \quad & = \quad p \left( \vec{y} | X ; \theta \right) \\
    & = \quad \prod _ { i = 1 } ^ { m } p \left( y ^ { ( i ) } | x ^ { ( i ) } ; \theta \right) \\
    & = \quad \prod _ { i = 1 } ^ { m } \left( h \left( x ^ { ( i ) } \right) \right) ^ { y ^ { ( i ) } } \left( 1 - h  \left( x ^ { ( i ) } \right) \right) ^ { 1 - y ^ { ( i ) } }
  \end{aligned}
\end{equation}
And taking the negative log likelihood to minimize:
\begin{equation}
  \begin{aligned}
    \ell ( \theta ) \quad & = \quad -\log L ( \theta ) \\
    & = \quad -\sum _ { i = 1 } ^ { m } y ^ { ( i ) } \log h \left( x ^ { ( i ) } \right) + \left( 1 - y ^ { ( i ) } \right) \log \left( 1 - h \left( x ^ { ( i ) } \right) \right) \\
    & = \quad -\sum_{i=1}^{m} y^{(i)} \theta^T x^{(i)} - \log \left( 1 + e^{\theta^T x^{(i)}} \right)
  \end{aligned}
\end{equation}
This is known as the binary cross-entropy loss function.
\subsection{Gradient Descent}
Let’s start by working with just one training example (x,y), and take
derivatives to derive the stochastic gradient ascent rule:
\begin{equation}
  \begin{aligned}
    \frac { \partial } { \partial \theta _ { j } } \ell ( \theta ) \quad & = \quad - \left( y \frac { 1 } { g \left( \theta ^ { T } x \right) } - ( 1 - y ) \frac { 1 } { 1 - g \left( \theta ^ { T } x \right) } \right) \frac { \partial } { \partial \theta _ { j } } g \left( \theta ^ { T } x \right) \\
    & = \quad - \left( y \frac { 1 } { g \left( \theta ^ { T } x \right) } - ( 1 - y ) \frac { 1 } { 1 - g \left( \theta ^ { T } x \right) } \right) g \left( \theta ^ { T } x \right) \left( 1 - g \left( \theta ^ { T } x \right) \right) \frac { \partial } { \partial \theta _ { j } } \theta ^ { T } x \\
    & = \quad - \left( y \left( 1 - g \left( \theta ^ { T } x \right) \right) - ( 1 - y ) g \left( \theta ^ { T } x \right) \right) x _ { j } \\
    & = \quad - \left( y - h ( x ) \right) x _ { j }
  \end{aligned}
\end{equation}
This therefore gives us the stochastic gradient ascent rule:
\begin{equation}
  \theta _ { j } : = \theta _ { j } + \alpha \left( y ^ { ( i ) } - h \left( x ^ { ( i ) } \right) \right) x _ { j } ^ { ( i ) }
\end{equation}
We must use gradient descent for logistic regression since there is no closed
form solution for this problem.
\subsection{Newton-Raphson Algorithm}
The Hessian matrix for logistic regression is:
\begin{equation}
  \frac{\partial^2 \ell(\theta)}{\partial \theta \partial \theta} = -\sum_{i=1}^m x^{(i)} x^{(i) T} \cdot h\left(x^{(i)}\right) \cdot \left(1-h\left(x^{(i)}\right)\right)
\end{equation}
Hence the second order update is:
\begin{equation}
  \theta := \theta - \left( \frac{\partial^2 \ell(\theta)}{\partial \theta \partial \theta} \right)^{-1} \frac{\partial \ell(\theta)}{\partial \theta}
\end{equation}


\subsection{Recap}

Previously we defined addition and multiplication in terms of successive use of the sucessor function. That is, the definition of addition was:

\(\forall a \in \mathbb{N} (a+0=a)\)

\(\forall a b \in \mathbb{N} (a+s(b)=s(a+b))\)

And similarly for multiplication:

\(\forall a \in \mathbb{N} (a.0=0)\)

\(\forall a b \in \mathbb{N} (a.s(b)=a.b+a)\)

Additional functions could also be defined, following the same pattern:

\(\forall a \in \mathbb{N} (a\oplus _n 0=c)\)

\(\forall a b \in \mathbb{N} (a\oplus _{n} s(b)=(a\oplus_{n} b)\oplus_{n-1}a)\)

\subsection{Powers}

Powers can also be defined:

\(\forall a \in \mathbb{N} a^0=1\)

\(\forall a b \in \mathbb{N} a^{s(b)}=a^b.a\)

\subsection{Example}

So \(2^2\) can be calculated like:

\(2^2=2^{s(1)}\)

\(2^{s(1)}=2.2^1\)

\(2.2^1=2.2.2^0\)

\(2.2.2^0=2.2.1\)

\(2.2.1=4\)

Unlike addition and multiplication, exponention is not commutative. That is

\(a^b\ne b^a\)

\subsection{Exponential rules}

\(a^ba^c=a^{b+c}\)

\((a^b)^c=a^{bc}\)

\((ab)^c=a^cb^c\)


\subsection{Intermediate value theorem}

Take a real function \(f(x)\) on closed interval \([a,b]\), continuous on \([a,b,]\).

IVT says that for all numbers \(u\) between \(f(a)\) and \(f(b)\), there is a corresponding value \(c\) in \([a,b]\) such that \(f(c)=u\).

That is:

\(\forall u \in [min(f(a),f(b)),max(f(a),f(b))] \exists c \in [a,b] (f(c)=u)\)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Decide whether each of the following statements is true or false.
Prove that your conclusion is correct.
\begin{enumerate}[label=(\alph*)]
\item $2^{n+1} = O(2^n)$
\item $f(n) = O(g(n))$ implies $2^{f(n)} = O(2^{g(n)})$
\end{enumerate}

\subsection*{Solution}
\begin{enumerate}[label=(\alph*)]
\item \textit{true}.
For all sufficiently large $n$ ($n \geq 0$), there exists a constant $c = 2$ such that

\begin{equation}
2 \times 2^n = f(n) \leq c \times g(n) = c \times 2^n
\end{equation}

\item \textit{true}.
Since $f(n) = O(g(n))$, there exists constants $c$ and $x_0$ such that $f(n) \leq c \times g(n)$ for any $n \geq x_0$.
Since $h(x) = 2^x$ is an ever-increasing function, we can write

\begin{equation}
f(n) \leq c \times g(n) \Rightarrow 2^{f(n)} \leq 2^{c \times g(n)} = 2^c \times 2^{g(n)}
\end{equation}

Thus $2^{f(n)} \leq d \times 2^{g(n)}$ always holds true for all $n \geq x_1 = x_0$ when choosing $d \geq 2^c$.

\end{enumerate}


\chapter{Rank of a Tensor Product Space}
The \emph{rank} of a tensor product space, along with all the tensors which live in it, is the number of vector spaces and dual spaces involved in creating it.
This is quite simple when the tensors are simple tensors (there are only vector spaces or only dual spaces involved).
In that instance, we just count the number of spaces, and put that number where the indices on the component would normally go.
Consider these two tensors,
\begin{align*}
    &T^\mu e_\mu \in V : V^* \to \mathbb{F}, \\
    &T_\nu e^\nu \in V^* : V \to \mathbb{F}.
\end{align*}
Both are known as rank 1 tensors, since only a single space is involved.

Rank gets slightly more complicated if the tensors are mixed.
In this instance, we just put number in both indices, so a tensor with a single vector space and a single dual space would be a rank $(1,1)$. We say such a tensor would live in the tensor product space $\tps{T}^1_1$. The problem comes when trying to identify the ordering of these two spaces; should $\tps{T}^1_1$ mean $V \otimes V^*$ or $V^* \otimes V$? The answer is simply a matter of convection. Tensor product spaces \emph{always} begin with vector spaces, and then move on to dual spaces.
This means that a rank $(p,q)$ tensor product space $\tps{T}^p_q$ will be as follows:
\[ \tps{T}^p_q = \underbrace{V \otimes V \otimes \cdots}_{\text{$p$ times}} \underbrace{V^* \otimes V^* \otimes \cdots}_{\text{$q$ times}}. \]
While tensor product spaces with the vector and dual spaces in different orders (even all shuffled about) still transform in exactly the same way, we insist on only defining tensors to have the proper order of $V \otimes V^*$ to rectify the ambiguity of ordering.


\subsection{Matrix powers}

For a square matrix \(M\) we can calculate \(MMMM...\), or \(M^n\) where \(n\in \mathbb{N}\).

\subsection{Powers of diagonal matrices}

Generally, calculating a matrix to an integer power can be complicated. For diagonal matrices it is trivial.

For a diagonal matrix \(M=D^n\), \(m_{ij}=d_{ij}^n\).


\subsubsection{Circulations}
\begin{definition}
	If $C$ is a simple, closed curve, then $\int\limits_{C}{\vec{F} \cdot \mathrm{d}\vec{r}}$ is the circulation of $\vec{F}$ on $C$.
\end{definition}
\noindent
We notate that we are taking a circulation as $\oint\limits_{C}{\vec{F} \cdot \mathrm{d}\vec{r}}$.

[INSERT IMAGE]

\noindent
For example, let's find the circulation of $\vec{F}(x,y,z) = \langle yz, xz, xy \rangle$ on the circle of radius 1 centered at $(0,0,1)$ in the $z = 1$ plane in the counter-clockwise direction.
\begin{equation*}
	\vec{r}(t) = \langle \cos{t}, \sin{t}, 1 \rangle, 0 \leq t \leq 2\pi
\end{equation*}
\begin{equation*}
	\vec{F}\circ\vec{r} = \langle \sin{t}, \cos{t}, \sin{t}\cos{t} \rangle
\end{equation*}
\begin{equation*}
	\left(\vec{F}\circ\vec{r}\right) \cdot \vec{r^\prime}(t) = \cos{(2t)}
\end{equation*}
\begin{equation*}
	\oint\limits_{C}{\vec{F}\cdot\mathrm{d}\vec{r}} = \int_{0}^{2\pi}{\cos{(2t)}\mathrm{d}t} = \frac{1}{2}\sin{(2t)}\rvert_{0}^{2\pi} = 0
\end{equation*}


\chapter{Division from plus, times, and norm}

\begin{abstract}
    Dividing \(n\) by \(d\) is finding
    \((q,r)\) that satisfies \(n = q \times d + r\)
    and minimizes the norm of \(r\).
    Integer division and set division are special cases of that division.
\end{abstract}

\section{Terminology}

In \(n / d = q\),
we call \(n\) the dividend,
\(d\) the divisor,
and \(q\) the quotient.
The dividend is the thing that is divided.
The divisor is the thing that divides.

In \(a \times b = c\),
we call \(a\) the multiplier, \(b\) the multiplicand, and \(c\) the product.
The multiplicand is the thing that is repeated.
The multiplier is how many times the multiplicand is repeated.
Therefore \(2 \times 3 = 3+3\) and \(3 \times 2 = 2+2+2\).

The word `quotient' comes from the Latin word `quotiens' that means `how many times'.
The quotient describes \emph{how many times} we must repeat the divisor to obtain the dividend.
Therefore \(6/2 = 3\) because \(2+2+2 = 6\)
and \(6/3 = 2\) because \(3+3 = 6\).

The divisor in a division
becomes the multiplicand in the related multiplication.
Formally, \(n / d = q\) means the same thing as \(n = q \times d\).
To obtain the related multiplication of a division,
\emph{right}-multiply each side of the equation by the divisor.
Right multiplication cancels right division.

When we write \(p \times q\), we say that we \emph{left-multiply \(q\) by \(p\)}
or \emph{right-multiply \(p\) by \(q\)}.
The `left' and `right' tell us where the multiplier is.
In a left multiplication, the multiplier (the thing that multiplies)
is on the left side of the multiplicand (the thing that is multiplied).
We say that we left-multiply or right-multiply a multiplicand \emph{by} a multiplier.

In the case of left division and right division,
the `left' and `right' tell us where the divisor is.
Left division means the divisor is on the left side of the dividend.

If \(p \times (q \times r) = r\), we call \(p\)
the \emph{left inverse} of \(q\) because \(p\)
cancels a multiplication where \(q\) is on the \emph{left} side.
We also say that we \emph{left-divide} \(q \times r\) by \(q\).
We also write \(q \backslash (q \times r) = r\).

Similarly, if \((p \times q) \times r = p\), we call \(r\)
the \emph{right inverse} of \(q\) because it
cancels a multiplication where \(q\) is on the \emph{right} side.
We write \((p \times q) / q = p\).

To \emph{left-divide} a dividend by a divisor is
to \emph{left-multiply} the dividend by the \emph{left-inverse} of the divisor.
To \emph{right-divide} a dividend by a divisor is
to \emph{right-multiply} the dividend by the \emph{right-inverse} of the divisor.

The consequence of those definitions is that
we have \(6 / 2 = 3\)
because \(3 \times 2 = 6\),
not because \(2 \times 3 = 6\).

A problem with this division is that
it is not always defined for every non-zero divisor.
That's where the remainder comes into play.
It allows us to extend the definition of division.
The remainder is the smallest thing that we must add to satisfy the equation
after we find the quotient.
If the remainder is zero, we say that the divisor \emph{divides} the dividend.

We define \(n/d = q\) where \(n = q \times d + r\)
such that \(r\) is \emph{minimal} in a way.

\section{Structure}

Let there be a ring-like structure.
Let there be a type \(R\),
an addition \(+\),
a multiplication \(\times\),
and a norm \(m : R \to [ 0,\infty )\).
The operations do not have to have identities.
They do not have to be invertible.
They do not have to commute.

Dividing \(n\) by \(d\) is finding \((q,r)\) such that \(n = q \times d + r\)
and making sure that such \((q,r)\) is unique.
However, there may be many \((q,r)\) that satisfies that equation.
We want at most one pair.
Which pair should we choose as the result of the division?

If \(\R = \Nat\), we choose the smallest \(r\).
If \(R = \Int\), we choose the \(r\) that minimizes \(|r|\),
the absolute value of \(r\).
If \(R\) is the type of sets, we choose the \(r\)
that minimizes \(|r|\), the size of \(r\).
The common thing here is that we can define a \emph{norm}.
The division then picks the pair that minimizes the norm of the remainder.
Here we write \(|r|\) (the norm of \(r\))
to mean the distance between zero (the additive identity) and \(r\).
The norm of a natural number is the number itself.

That equation is a special case of the linear equation
\( b = a_1 \cdot x_1 + \ldots + a_p \cdot x_p \)
where \(b = n\), \(p = 2\), \(a = (d,1)\), \(x = (q,r)\).

\section{Generalization}

To understand division,
first we look at following example about the natural numbers.
Later we will generalize the division to also work on sets.

Suppose that we want to divide \(n\) by \(d\).
Whenever we divide \(n\) by \(d\),
what we actually do is we
find \(q\) and \(m\) that satisfy
\(n = q \times d + m\)
where \(q\) is \emph{maximal}
and \(m\) is \emph{minimal}
in the sense that there are no bigger \(q\) and smaller \(m\)
that satisfy the equation.
We call \(n\) the \emph{dividend} (the thing that is divided),
\(d\) the \emph{divisor} (the thing that divides),
\(q\) the \emph{quotient},
and \(m\) the \emph{modulus} or the \emph{remainder}.

Now consider the integers.
Given \(n \neq 0\) and \(d \neq 0\),
there are infinitely many \((q,m)\) pairs that satisfy \( n = q \times d + m \).
Which one should we choose?

We have just defined natural number division in terms of multiplication, addition, and natural ordering.
If we can define set multiplication, set addition, and set ordering, then we can define set division.

We can define the multiplication of two sets as their Cartesian product.
We can define the addition of two sets as their union.
We can define that a set is less than another iff the former is a proper subset of the latter.
We then define the remainder as the smallest set that we must add to satisfy the equation.
Formally,
\begin{align}
    Q \times D &= \{ (q,d) ~|~ q : Q, ~ d : D \}
    \\
    A + B &= \{ x ~|~ x : A \vee x : B \}
    \\
    A \le B &\iff A \subseteq B
    \\
    N &= Q \times D + R \text{ such that \(R\) is minimal.}
\end{align}

To compute the quotient and the modulus,
we solve \(N = Q \times D + R\)
with the constraint that \(R\) is \emph{minimal}
in the sense that there are no smaller \(R\)
that satisfies that equation.

Similar to \(n/d = q\), we define \( N / D = Q \).
It seems that all we did was rewrite everything using capital letters.
It's just that now the variables represent sets instead of numbers.
\emph{Indeed if we can define multiplication, addition, and maximum,
then we can define division.}

Note that unlike natural number multiplication,
set multiplication does not commute.

\section{Example}

Let's say we have these:
\begin{align*}
    N &= \{ a, b, c, aa, ab, ac, ba, bb, bc, ca, cb, cc \}
    \\
    D &= \{ a, b \}
\end{align*}
We compute:
\begin{align*}
    N &= Q \times D + M
    \\
    N &= \{ a, b, c \} \times \{ a, b \} + \{ c, ac, bc, cc \}
\end{align*}
We can't shrink \(M\) anymore
so we claim:
\begin{align*}
    Q &= \{ a, b, c \}
    \\
    M &= \{ a, b, c, ac, bc, cc \}
\end{align*}

Let's say we have a polynomial \(x^2 + 2x + 3\) and we want to divide it with \(x + 1\).
\begin{align*}
    n &= x^2 + 2x + 3
    \\
    d &= x + 1
    \\
    q &= x + 1
    \\
    r &= 1
\end{align*}
We define \(p \le q\) iff \(\deg~p \le \deg~q\) or \(p_1 \le q_1 \wedge \ldots\).

The left quotient (Brzozowsky derivative)
of a language \(L\) with respect to a string \(p\) is \(\{p\} \backslash L\).
It is the special case of left-division of languages
where the divisor has only one element.
\begin{align}
    \{p\} \backslash L = \{ x ~|~ px : L \}
\end{align}

\begin{align}
    px : L &\vdash x : \{p\} \backslash L
\end{align}

The expression \(2 \times 3\) means \(3+3\), not \(2+2+2\).
They both evaluates to \(6\), but they mean different things.
For natural numbers this sounds pedantic,
but this is important when the multiplication does not commute,
such as set multiplication that we shall see later.