\subsection{Solving systems using $LU$ factorization}

One reason people care about the $LU$ factorization
\index{LU factorization!solving systems} is it allows the quick solution of
systems of equations. Here is an example.

\begin{example}{$LU$ factorization to solve equations}{}
Suppose you want to find the solutions to
\begin{equation*}
\begin{mymatrix}{rrrr}
1 & 2 & 3 & 2 \\
4 & 3 & 1 & 1 \\
1 & 2 & 3 & 0
\end{mymatrix} \begin{mymatrix}{c}
x \\
y \\
z \\
w
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
2 \\
3
\end{mymatrix} .
\end{equation*}
\end{example}

\begin{solution}

Of course one way is to write the augmented matrix and grind away. However,
this involves more row operations than the computation of the $LU$
factorization and it turns out that the $LU$ factorization can give the
solution quickly. Here is how. The following is an $LU$ factorization for
the matrix.
\begin{equation*}
\begin{mymatrix}{rrrr}
1 & 2 & 3 & 2 \\
4 & 3 & 1 & 1 \\
1 & 2 & 3 & 0
\end{mymatrix} \allowbreak =\allowbreak \begin{mymatrix}{rrr}
1 & 0 & 0 \\
4 & 1 & 0 \\
1 & 0 & 1
\end{mymatrix} \begin{mymatrix}{rrrr}
1 & 2 & 3 & 2 \\
0 & -5 & -11 & -7 \\
0 & 0 & 0 & -2
\end{mymatrix} .
\end{equation*}

Let $UX=Y$ and consider $LY=B$ where in this case, $B=\mat{1,2,3} ^{T}$. Thus
\begin{equation*}
\allowbreak \begin{mymatrix}{rrr}
1 & 0 & 0 \\
4 & 1 & 0 \\
1 & 0 & 1
\end{mymatrix} \begin{mymatrix}{c}
y_{1} \\
y_{2} \\
y_{3}
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
2 \\
3
\end{mymatrix}
\end{equation*}
which yields very quickly that $Y=\begin{mymatrix}{r}
1 \\
-2 \\
2
\end{mymatrix} \allowbreak$.

Now you can find $X$ by solving $UX=Y$. Thus in this case,
\begin{equation*}
\begin{mymatrix}{rrrr}
1 & 2 & 3 & 2 \\
0 & -5 & -11 & -7 \\
0 & 0 & 0 & -2
\end{mymatrix} \begin{mymatrix}{c}
x \\
y \\
z \\
w
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
-2 \\
2
\end{mymatrix}
\end{equation*}
which yields
\begin{equation*}
X=\begin{mymatrix}{c}
-
\vspace{0.05in}\frac{3}{5}+\vspace{0.05in}\frac{7}{5}t \\
\vspace{0.05in}\frac{9}{5}-\vspace{0.05in}\frac{11}{5}t \\
t \\
-1
\end{mymatrix} ,\enspace t\in \R\text{.}
\end{equation*}

\end{solution}


\chapter{Matrices}\label{app:matrices}
This appendix aims to provide some fundamental knowledge on matrices and linear algebra used throughout this thesis.

A matrix is a rectangular array with numerical elements and its dimensions are denoted using ``$row \times column$''. A $3\times 5$ matrix, for example, thus has $3$ rows and $5$ columns (see Figure \ref{fig:matrixA}). Along those lines, a \textit{row vector} is a matrix with $1$ row and more than $1$ column and a \textit{column vector} is a matrix with $1$ column and more than $1$ row.

In this document, matrices and vectors are written using bold symbols. %\SWcomment[Many notations exist, blabla $\bar a$ $\vec a$]
A matrix is denoted by a capital letter -- such as $\A$ -- whereas vectors are decapitalised -- such as $\u$. An element in a matrix is denoted with a non-bold, decapitalised variable, where the subscripts indicate the indices of the row and column. For example, the element in the 2\textsuperscript{nd} row and the 4\th column of a matrix $\A$ is denoted as $a_{24}$. An element in a vector only has one subscript, regardless of whether it is a row or a column vector.

\section{Operations}
Multiplying and dividing a matrix by a scalar (a single number) is valid and happens on an element-by-element basis. For a $2\times 2$ matrix $\A$ and scalar $p$ the following operations hold
\begin{equation*}
        p \A = \A p = \begin{bmatrix}
        p\cdot a_{11}& p\cdot a_{12}\\
        p\cdot a_{21} & p\cdot a_{22}
    \end{bmatrix}, \qaq \A / p= \begin{bmatrix}
            a_{11}/p& a_{12}/p\\
            a_{21}/p & a_{22}/p
    \end{bmatrix}.
\end{equation*}
Notice that although a matrix can be divided by a scalar, a scalar can not necessarily be divided by a matrix. See Section \ref{sec:inverse} for more information.

\subsubsection{Matrix transpose}
A matrix or vector can be \textit{transposed}, which is indicated by the $T$ operator. Transposing a matrix $\A$ is denoted by $\A^T$. This means that the elements in the $i$\th row and the $j$\th column of the original matrix become the elements in the $j$\th row and the $i$\th column of the transposed matrix. Essentially the row and column indices of the elements inside the matrix get switched according to
\begin{equation}\label{eq:matrixTransposition}
    a_{ij} = a_{ji}.
\end{equation}
Also see Figure \ref{fig:matrixTransp}. For a row vector, the transpose operation simply changes it to a column vector and vice versa. Another way of seeing a transpose is that all the elements get flipped over the \textit{main diagonal} of the matrix. The main diagonal comprises the elements $a_{ij}$ where $i=j$ and a transpose does not affect the location of these elements.

\begin{figure}[h]
    \centering
    \subfloat[A $3\times 5$ matrix $\A$.\label{fig:matrixA}]{\includegraphics[width=0.45\textwidth]{figures/analysis/matrixA1.pdf}}\hspace{0.03cm}
    \subfloat[A transposed matrix $\A^T$ of size $5\times 3$.\label{fig:matrixAT}]{\includegraphics[width=0.45\textwidth]{figures/analysis/matrixAT1.pdf}}
    \caption{A matrix $\A$ and its transpose $\A^T$. The elements get flipped along the main diagonal of the matrix according to Eq. \eqref{eq:matrixTransposition}. \label{fig:matrixTransp}}
\end{figure}

\subsubsection{Matrix multiplication}
Matrix multiplication (this includes matrix-vector multiplication) is different from regular multiplication in that it needs to abide several extra rules. The multiplication of two matrices $\A$ and $\B$ to a resulting matrix $\C$ is defined as
\begin{equation}\label{eq:matrixMult}
    c_{ij} = \sum_{k=1}^K a_{ik} b_{kj},
\end{equation}
where $K$ is both the number columns of matrix $\A$ and the number of rows in matrix $\B$. It thus follows that, in order for a matrix multiplication to be valid, the number of columns of the first matrix needs to be equal to the number of rows in the second matrix. The result will then be a matrix with a number of rows equal to that of the first matrix and a number of columns equal to that of the second matrix. See Figure \ref{fig:matrixVector} for reference.

As an example, consider the $L\times M$ matrix $
\A$ and a $M\times N$ matrix $\B$ with $L\neq N$. The multiplication $\A\B$ is defined as the number of columns of matrix $\A$ ($M$) is equal to the number of rows of matrix $\B$ (also $M$). The result, $\C$, is a $L \times N$ matrix. The multiplication $\B\A$ is undefined as the number of columns of the first matrix does not match the number of rows in the second matrix. A valid multiplication of two matrices written in their dimensions is
\begin{equation}
    \overbrace{(L\times M)}^{\A}\cdot \overbrace{(M\times N)}^{\B} = \overbrace{(L
    \times N)}^{\C}.
\end{equation}

\begin{figure}[h]
    \includegraphics[width=\textwidth]{figures/analysis/matrixVector.pdf}
    \caption{Visualisation of valid matrix multiplications (see Eq. \eqref{eq:matrixMult}). The ``inner'' dimensions (columns of the left matrix and rows of the right) must match and result in a matrix with a size of ``outer'' dimensions (rows of the left matrix and columns of the right). \label{fig:matrixVector}}
\end{figure}

\section{Matrix inverse}\label{sec:inverse}
If a matrix has the same number of rows as columns, it is called a \textit{square matrix}. Square matrices have special properties, one of which is that it (usually) can be \textit{inverted}. A square matrix $\A$ is invertable if there exists a matrix $\B$ such that
\begin{equation}
    \A \B = \B \A = \I.
\end{equation}
This matrix $\B$ is then called the \textit{inverse} of $\A$ and can be written as $\A^{-1}$. Not all square matrices have an inverse, in which case it is called \textit{singular}. Rather than going through manually inverting a matrix, or determining whether it is singular, the following function in \texttt{MATLAB} will provide the inverse of a matrix \texttt{A}:
\begin{center}
    \texttt{A\_inverted = inv(A);}
\end{center}

The inverse of a \textit{diagonal matrix} (a matrix with non-zero elements on its main diagonal and the rest zeros) is obtained by replacing the diagonal elements by their reciprocal. So for a diagonal $3\times 3$ matrix, the following holds:
\begin{equation*}
    \xbracketMatrixstack{
        a_{11} & 0 & 0\\
        0 & a_{12} & 0\\
        0 & 0 & a_{33}
    }^{-1} =\quad
    \xbracketMatrixstack{
        \frac{1}{a_{11}} & 0 & 0\\
        0 & \frac{1}{a_{12}} & 0\\
        0 & 0 & \frac{1}{a_{33}}
    }.
\end{equation*}

\section{Systems of linear equations}\label{sec:linearEquations}
Matrices can be conveniently used to solve \textit{systems of linear equations}, a set of linear equations containing the same set of variables.

For example, take the system of linear equations
\begin{align*}
    x + z &= 6,\\
    z - 3y &= 7,\\
    2x + y + 3z &= 15,
\end{align*}
with independent variables $x$, $y$ and $z$. The goal is to find a solution for these variables that satisfy all three equations. This system could be solved by hand using algebraic methods, but alternatively, the system can be written in matrix form:
\begin{equation}
    \A \u = \w.
\end{equation}
Here, column vector $\u$ contains the independent variables $x$, $y$, and $z$, matrix $\A$ contains the coefficients multiplied onto these variables and $\w$ contains the right-hand side, i.e., the coefficients not multiplied onto any of the variables:
\begin{equation*}
    \underbrace{\xbracketMatrixstack{
        1& 0& 1\\
        0& -3& 1\\
        2& 1& 3}}_{\A}
    \underbrace{\xbracketMatrixstack{
        x\\
        y\\
        z
    }}_{\u} = \underbrace{\xbracketMatrixstack{
        6\\
        7\\
        15
    }}_{\w}
\end{equation*}
This can be solved for $\u$ by taking the inverse of $\A$ (see Section \ref{sec:inverse}) and multiplying this onto $\w$
\begin{equation}
    \u = \A^{-1}\w.
\end{equation}
Generally, if X unknowns are described by X equations, the unknowns can be solved for, by using this method.

Solving a system of linear equations can be implemented in \texttt{MATLAB} by using the code given in Section \ref{sec:inverse} and multiplying this onto a vector \texttt{w}
\begin{center}
    \texttt{u = inv(A) * w;}
\end{center}
or more compactly, by using the `\texttt{\textbackslash}' operator:
\begin{center}
    \texttt{u = A\textbackslash w;}
\end{center}

\section{Eigenvalue problems}\label{sec:eigenValueProblems}
A square matrix $\A$ is characterised by its \textit{eigenvalues} and corresponding \textit{eigenvectors}. In a FDTD context, these are usually associated with the modes of a system, where the eigenvalues relate to the modal frequencies and the eigenvectors to the modal shapes. Section \ref{sec:modalAnalysis} provides more information on this.

To find these characteristic values for a $p\times p$ matrix $\A$, an equation of the following form must be solved
\begin{equation}
    \A \boldPhi = \lambda \boldPhi.
\end{equation}
This is called is an \textit{eigenvalue problem} and has $p$ solutions (corresponding to the dimensions of $\A$). These are the $p$\th eigenvector $\boldPhi_p$ and the corresponding eigenvalue $\lambda_p$ which is calculated using
\begin{equation}
    \lambda_p = \eig_p(\A),
\end{equation}
where $\eig_p(\cdot)$ denotes the `$p$\th eigenvalue of'. Instead of delving too deep into eigenvalue problems and the process of how to solve them, an easy way to obtain the solutions using \texttt{MATLAB} is provided here:

\begin{center}
    \texttt{[phi, lambda] = eig(A, {\color[HTML]{A100F4}'vector'});}
\end{center}
The $p$\th eigenvector appears in the $p$\th column of $p\times p$ matrix \texttt{phi} and the corresponsding eigenvalues are given in a $p \times 1$ column vector \texttt{lambda}. %Note that the outcome is not necessarily sorted! To do this, do
%
% \begin{center}
%     \begin{tabular}{l}
%     \texttt{[lambda, order] = sort(lambda);}\\
%     \texttt{phi = phi(:, order);}
%     \end{tabular}
% \end{center}


We assume that we want to find nearest neighbors in a space $X$ with a distance measure $\text{dist}\colon X\times X \rightarrow \mathbb{R}$, for example the $d$-dimensional
Euclidean space $\mathbb{R}^d$ under Euclidean distance ($l_2$ norm), or %$d$-dimensional
Hamming space $\{0,1\}^d$ under Hamming distance.

An algorithm $\mathcal{A}$ for nearest neighbor search builds a data structure DS$_\mathcal{A}$ for a data set $S \subset X$ of $n$ points. In a preprocessing phase, it creates DS$_\mathcal{A}$
to support the following type of queries: For a query point $q \in X$ and an integer $k$, return a \emph{result tuple} $\pi = (p_1, \ldots, p_{k'})$ of $k' \leq k$ distinct points from $S$
that are ``close'' to the query $q$. Nearest neighbor search algorithms generate $\pi$ by refining a set $C \subseteq S$ of
candidate points with respect to $q$ by choosing the $k$ closest points among those using distance computations. The size of $C$ (and thus the number of distance computations) is denoted by $N$. We let $\pi^\ast = (p^\ast_1, \ldots, p^\ast_k)$ denote the tuple containing the true $k$ nearest neighbors for $q$ in $S$ (where ties are broken arbitrarily). We
assume in the following that all tuples are sorted according to their distance to $q$.

\subsection{Quality Measures}

We use different notions of ``recall'' as a measure of the quality of the result returned by the algorithm.
Intuitively, recall is the ratio of the number of points in the result tuple that are true nearest neighbors to the number $k$ of true nearest neighbors. However, this intuitive
definition is fragile when distances are not distinct or when we try to add a notion of approximation to it. To avoid these issues, we use
the following distance-based definitions of recall and $(1+\varepsilon)$-approximative recall, that take the distance
of the $k$-th  true nearest neighbor as threshold distance.
%
\begin{align*}
\text{recall}(q, \pi, p^\ast_k) &= \frac{|\{p \text{ contained in $\pi$} \mid \text{dist}(p,q) \leq \text{dist}(p^\ast_k,q)\}|}{k}\\
\text{recall}_\varepsilon(q, \pi, p^\ast_k) &= \frac{|\{p \text{ contained in $\pi$} \mid \text{dist}(p,q) \leq (1 + \varepsilon) \text{dist}(p^\ast_k,q)\}|}{k}, \quad\text{for $\varepsilon > 0$.}
\end{align*}
(If all distances are distinct, $\text{recall}(q, \pi, p^\ast_k)$ matches the intuitive notion of recall.)

We note that (approximate) recall in high dimensions is sometimes criticised; see, for example, \cite[Section 2.1]{nmslib}. We investigate the impact of approximation as part of the evaluation in Section~\ref{sec:evaluation}, and plan to include other quality measures such as position-related measures \cite{ZezulaSAR98} in future work.

\begin{comment}
Given a query point $q \in X$ and the tuple $\pi^\ast$ of true nearest neighbors, we let $d_1 < \dots < d_t$ denote the $t \leq k$ distinct distances of the points in $\pi^\ast$ to
$q$. Let $f_i$ be the number of points in $\pi$ that have distance $d_i$ to $q$, and let $f'_t$ be the number of points in $\pi^\ast$ at distance $d_t$.  We define:
\begin{align*}
    \text{recall}(\pi, \pi^\ast) &= \frac{\sum_{i = 1}^{t - 1} f_i + \min\{f_t, f'_t\}}{k}.
\end{align*}
Note the minimum for the last class of distances. Otherwise, the recall of a result with two points at distance $2$, where the true nearest neighbors have distance $1$ and $2$ would yield a recall of $1$.

For an approximation factor $\varepsilon > 0$ we define an approximative version of recall as follows: We say that a point $p$ is \emph{$\varepsilon$-close} to $d_i$ if dist$(p, q) \leq (1+\varepsilon)d_i$ but dist$(p,q) > (1+\varepsilon)d_{i - 1}$. (For sake of the definition, we set $d_0 = -1$.) Now, we let $f^\varepsilon_i$ be the number of points in $\pi$ that
are $\varepsilon$-close to $d_i$, for each $i \in \{1,\ldots, t\}$.  Furthermore, let $f'_t$ be defined as above. Then we define

\begin{align*}
\text{recall}_\varepsilon(\pi, \pi^\ast) &= \frac{\sum_{i = 1}^{t - 1} f^\varepsilon_i + \min\{f^\varepsilon_t, f'_t\}}{k}.
\end{align*}

To estimate the quality of the result without having an approximation factor $\varepsilon$, \cite{ZezulaSAR98} propose to look at the \emph{relative error}
of $\pi$ and $\pi^\ast$.
\begin{align*}
\text{rel-error}(\pi, \pi^\ast) &=
\begin{cases}
\frac{\sum_{i = 1}^{k}\text{dist}(q, p^\ast_i)}{\sum_{i = 1}^{k}\text{dist}(q, p_i)}, & \text{if $k = k'$, and} \\
0, & \text{otherwise}.
\end{cases}
\end{align*}
%
Here, we set the relative distance of a result having less than $k$ elements to $0$, which corresponds to filling up the left-out spots by points at distance $\infty$ to the query.

We note that recall and relative error are up to criticism in high dimensions, see, for example the discussion in~\cite[Section 2.1]{nmslib}. In our experiments, we did not observe that approximative recall or relative error yielded unexpected results. Nevertheless, we plan to include other quality measures such as position-related measures \cite{ZezulaSAR98} in future work.
\end{comment}
%\matodo{Should we keep the comment about criticism of metrics? -Alec}

\begin{table}[t]
%    \setlength{\tabcolsep}{0.5em} % for the horizontal padding
%    {\renewcommand{\arraystretch}{1.2}% for the vertical padding
    \begin{tabular}{l l}
        \textbf{Name of Measure} & \textbf{Computation of Measure} \\ \hline \hline
        Index size of DS & Size of DS after preprocessing finished (in kB) \\
        Index build time DS & Time it took to build DS  (in seconds) \\ \hline
        Number of distance computations &  $N$ \\
        Time of a query & Time it took to run the query and generate result tuple $\pi$ \\[1em]
\end{tabular}%}
    \caption{Performance measures used in the framework.}
    \label{tab:performance:measures}
\end{table}
%
\subsection{Performance Measures}

With regard to the performance, we use the performance measures defined
in Table~\ref{tab:performance:measures}, which are divided
into measures of the performance of the preprocessing step,
i.e., generation of the data structure, and measures of the performance
of the query algorithm. With respect to the query performance, different
communities are interested in different cost values. Some rely on
actual timings of query times, where others rely on the number of distance
computations. The framework can take both of
these measures into account. However, none of the currently included
algorithms report the number of distance computations.


\chapter{Model Selection}
Goal: determine the optimum parameters or the model for a given approximation or classification problem

\section{Performance Estimation}
Mean-square error:
$$MSE=\frac{1}{n} \sum_{i=1}^{n}(x_i - \bar{x}_i)^{2}$$
Root-mean square error:
$$RMSE=\sqrt{\frac{1}{n} \sum_{i=1}^{n}(x_i - \bar{x}_i)^{2}}$$

\begin{description}
\item[Apparent error] training error, the error on the training data
\item[True error] the error that will be obtained in use
\item[Test error] an estimate of the true error obtained by testing the network on some independent data
\end{description}

\noindent Validation Methods:
\begin{itemize}
\item Holdout Method: split the entire dataset into training set and testing set
\item Random Sampling Methods: performs $K$ data splits randomly and retrain the classifier from scratch for each split
\item K-fold Cross Validation: For each of $K$ experiments, use K-1 folds for training and the remaining one fold for testing
\item Leave one out Cross-Validation: $K=N$ in k-fold cross validation
\end{itemize}

\section{Three-Way Data Splits Methods}
\begin{description}
\item[Training set] examples for learning to fit the parameters
\item[Validation set] examples to determine the error of the classifiers
\item[Training + Validation set] combine examples used to re-train classifier
\item[Test set] examples used only to assess the performance of trained classifier
\end{description}

\section{Model Complexity}
Simple model might not able to solve a task (underfitting) \\ \\
Complex model might not able to generalize from small and noisy dataset (overfitting)


\lab{Vectorization}{Vectorization}
\label{lab:Python_Vectorization}

In Python, the cost of looping over a sequence is sometimes prohibitive.
One method for avoiding these explicit for-loops is to \emph{vectorize} our code.
NumPy allows operation that normally apply only to scalars to also work seamlessly with NumPy arrays.
Some of the benefits of allowing these types of operations are concise code and fast execution.

\section*{Some Simple Examples}

Vectorization is a term that refers generally to eliminating for-loops in higher level languages as well as taking advantage of the capabilities of modern processors to perform the same operation on multiple numbers at once.
We'll start with a very simple example.
The following code takes the sum of the elements of a NumPy array.
\begin{lstlisting}
def pysum(A):
    total = 0.
    for i in xrange(A.size):
        total += A[i]
    return total
\end{lstlisting}
At each stage it accesses the array which involves the creation of a new Python object to contain the floating point value in that portion of the array.
This slows down the operations significantly.
We can avoid creating and deleting a Python object by doing the operations in a lower level language like C or FORTRAN.
NumPy has already implemented many of these basic operations for us in these lower level languages.
Many of these operations in NumPy also take advantage of multithreading, which allows for the processing of multiple array elements at once.
In this case, we can just use the \li{sum()} method of arrays.
Timing the results, we see a significant speedup.
\begin{lstlisting}
>>> import numpy as np
>>> from numpy.random import rand
>>> A = rand(1000000)
>>> %timeit pysum(A)
1 loops, best of 3: 3.25 s per loop
>>> %timeit A.sum()
100 loops, best of 3: 12.9 ms per loop
\end{lstlisting}
This is a remarkable speedup.
Even for such a simple operation we see a speedup of a factor of 250!

We now consider an example of an operation that is not built in to NumPy.
Consider a two dimensional array where each row represents a point in $\mathbb{R}^n$.
To make computation nice, lets say that the norm of each point is neither extremely large nor extremely small.
The following code normalizes the vectors represented in the array by modifying it in place:
\begin{lstlisting}
from math import sqrt
def normalize1(A):
    # for each row (point) in A
    for i in xrange(A.shape[0]):
        # Compute norm
        # Start by summing
        # squares of entries in this row.
        total = 0.
        for j in xrange(A.shape[1]):
            total += A[i,j]**2
        # take square root to get norm
        total = sqrt(total)
        # divide row by its norm
        for j in xrange(A.shape[1]):
            A[i,j] /= total
\end{lstlisting}
This problem can be vectorized in stages.
Here we will proceed by eliminating the inner for-loops, then eliminate the outer one.
The inner for-loops can be eliminated like this:
\begin{lstlisting}
def normalize2(A):
    # for each row (point) in A
    for i in xrange(A.shape[0]):
        # compute norm of row
        # then divide it by its norm
        A[i] /= sqrt((A[i]**2).sum())
\end{lstlisting}
This is much shorter, much faster, and reads quite nicely.
To eliminate the last for-loop we do the computation for all the rows at once.
\begin{lstlisting}
def normalize3(A):
    # compute the norm of each row
    # then divide each row by its norm
    A /= np.sqrt((A**2).sum(axis=1,keepdims=True))
\end{lstlisting}
At a glance, what this line does may be a little less clear.
It essentially takes the square of \li{A}, sums its columns (i.e. sums along its rows), takes the square root, then divides \li{A} by the result by broadcasting along rows.
The \li{keepdims} option is used to align the axes for broadcasting.
An equivalent line of code would be \li{A /= np.sqrt((A**2).sum(axis=1).reshape((A.shape[0],1)))}.
Timing our results we see that eliminating these loops and allowing NumPy to perform some of these operations in parallel results in a significant speedup.
\begin{lstlisting}
>>> A = rand(1000000, 2) # 10^6 2D points
>>> Ac = A.copy()
>>> %time normalize1(A)
Wall time: 8.6 s
>>> A[:] = Ac
>>> %time normalize2(A)
Wall time: 14.3 s
>>> A[:] = Ac
>>> %time normalize3(A)
Wall time: 68 ms
>>> A = rand(1000000, 20) # 10^6 20D points
>>> Ac = A.copy()
>>> %time normalize1(A)
Wall time: 58.5 s
>>> A[:] = Ac
>>> %time normalize2(A)
Wall time: 14.6 s
>>> A[:] = Ac
>>> %time normalize3(A)
Wall time: 250 ms
>>> A = rand(50, 10000000) # 50 (10^7)D points
>>> Ac = A.copy()
>>> %time normalize2(A)
Wall time: 5.38 s
>>> A[:] = Ac
>>> %time normalize3(A)
Wall time: 5.4 s
\end{lstlisting}
Notice how much the dimensions of \li{A} affect the performance of each algorithm.
The first one is slowed down by its extensive use of Python objects for basic computations.
The second one is slowed down by the fact that it makes a new array slice for each row.
The third is slowed down by the fact that it must allocate a significant amount of temporary storage in memory to perform all its computations.
A faster and more memory-efficient solution to this problem can be obtained, but it requires using another language (like C, C++, Cython, or FORTRAN) that better optimizes for-loops.
When optimizing, keep in mind that it takes extra time to speed up your code.
A partial optimization is often good enough.
In general, only optimize when you are sure it will be worth your time to do so.

\section*{Vectorization Techniques}

Here are a few specific techniqes that can be useful in vectorization.

\begin{itemize}

\item Try to manipulate your computations mathematically.
Changing the order or way operations are applied can (but does not always) improve speed significantly.
A very simple example is $\sin^2 x + \cos^2 x = 1$.
Another example is that, for an orthogonal matrix, $A^{-1} = A^T$.
A transpose is a constant time operation because all that is required is creating a new view of the same memory.
A matrix inversion is much more expensive.

\item Try to avoid allocating unnecessary temporary arrays.
Long algebraic expressions in NumPy often involve the creation of large amounts of temporary arrays.
Excessive memory use limits the size of the problems your code can handle and slows it down.
This will be discussed further in a later lab, but be careful.

\item Eliminate the longer for-loops in Python.
Python's loops are much slower than the loops that are run in other languages like C and FORTRAN.
Many simple loops involving basic numeric operations can be performed in these languages through NumPy's built in functions.
This is an easy place to start optimizing.

\item Watch for times when broadcasting might be convenient.
This will save memory and let your program avoid unnecessary computations.

\item Use NumPy's universal functions.
It is much faster to evaluate something like \li{np.sin(x)} than something like
\begin{lstlisting}
from math import sin
Asin = np.empty_like(A)
for i in xrange(A.size):
    Asin[i] = sin(A[i])
\end{lstlisting}

\item In general, use common sense.
Think about the operations required to perform each computation.
Try out different implementations and compare their strengths and weaknesses, then use your best judgement about what the most common uses will be.
When optimizing, make absolutely sure to test your code to verify that you are getting the expected results.
Also be sure to comment your code carefully so that you (or whoever has to read your code) will be able to tell what you are doing.
Often it is more important to make code readable than it is to make it fast.
Use your best judgment.

\end{itemize}

To demonstrate the above principles we'll consider another example.
We will write a function which, given two arrays \li{A} and \li{B} will return the distance from \emph{each} row (considered as a vector) of \li{A} to \emph{every} row of \li{B}.
This can be done naively using for-loops as follows:
\begin{lstlisting}
from math import sqrt
import numpy as np
def dist1(A, B):
    # Preallocate output array.
    # Start with everything set to 0.
    D = np.zeros((A.shape[0], B.shape[0]))
    # For each row in A
    for i in xrange(A.shape[0]):
        # For each row in B
        for j in xrange(B.shape[0]):
            # For corresponding entries of
            # the rows in A and B
            for k in xrange(A.shape[1]):
                # add the squared difference
                D[i,j] += (A[i,k] - B[j,k])**2
            # take square root after finishing sum
            D[i,j] = sqrt(D[i,j])
    return D
\end{lstlisting}
We can eliminate the innermost for-loop easily
\begin{lstlisting}
def dist2(A, B):
    # Preallocate output array.
    # This time we'll be overwriting it,
    # so there's no need to assign it values.
    D = np.empty((A.shape[0], B.shape[0]))
    # For each row in A
    for i in xrange(A.shape[0]):
        # For each row in B
        for j in xrange(B.shape[0]):
            # Take the distance between the rows
            D[i,j] = sqrt(((A[i] - B[j])**2).sum())
    return D
\end{lstlisting}
Using the \li{axis} argument of the \li{sum()} method of arrays we can eliminate the next for-loop like this:
\begin{lstlisting}
def dist3(A, B):
    # Preallocate output array
    D = np.empty((A.shape[0], B.shape[0]))
    # For each row in A
    for i in xrange(A.shape[0]):
        # Take distance from this row to each row in B.
        # Use np.sqrt to operate on the entire array at once.
        D[i] = np.sqrt(((A[i] - B)**2).sum(axis=1))
    return D
\end{lstlisting}
This is a fairly nice solution.
It can be improved further.
Our final solution will take advantage of the fact that $\left(a - b\right)^2 = a^2 + b^2 - 2 a b$.
It will also use array broadcasting and in place operations to avoid excess memory use.
For many applications the above solution is good enough, but that isn't always the case.
\begin{lstlisting}
def dist4(A, B):
    # Use the fact that
    # (a-b)**2 = a**2 + b**2 - 2*a*b
    # Take squares of A and B.
    # Reshape so broadcasting works later
    A2 = (A**2).sum(axis=1).reshape((-1,1))
    B2 = (B**2).sum(axis=1).reshape((1,-1))
    # Use matrix multiplication to start
    # computing the remaining term.
    D = A.dot(B.T)
    # Multiply in place to avoid
    # allocating temporary arrays.
    # Notice that this operates on
    # the entire array.
    D *= -2
    # Broadcast to add in A2 and B2.
    # Perform these operations in place.
    D += A2
    D += B2
    # Take square root without
    # allocating a temporary array.
    np.sqrt(D, out=D)
    return D
\end{lstlisting}
Depending on how you are using your squared distance you could also use the fact that for $x, y \geq 0$, $x < y$ if and only if $\sqrt{x} < \sqrt{y}$.
This would be useful if all you need to know is which distances are greater than one another.
Timing these functions gives the following results:
\begin{lstlisting}[escapechar=\@]
>>> A = rand(500, 2)
>>> B = rand(1000, 2)
>>> %timeit dist1(A, B)
1 loops, best of 3: 3.18 s per loop
>>> %timeit dist2(A, B)
1 loops, best of 3: 6.86 s per loop
>>> %timeit dist3(A, B)
10 loops, best of 3: 27.6 ms per loop
>>> %timeit dist4(A, B)
100 loops, best of 3: 7.51 ms per loop
>>> A = rand(500, 20)
>>> B = rand(1000, 20)
>>> %timeit dist1(A, B)
1 loops, best of 3: 23.8 s per loop
>>> %timeit dist2(A, B)
1 loops, best of 3: 6.89 s per loop
>>> %timeit dist3(A, B)
10 loops, best of 3: 60.5 ms per loop
>>> %timeit dist4(A, B)
100 loops, best of 3: 7.82 ms per loop
>>> A = rand(50, 200)
>>> B = rand(100, 200)
>>> %timeit dist1(A, B)
1 loops, best of 3: 2.29 s per loop
>>> %timeit dist2(A, B)
10 loops, best of 3: 71.9 ms per loop
>>> %timeit dist3(A, B)
100 loops, best of 3: 5.28 ms per loop
>>> %timeit dist4(A, B)
1000 loops, best of 3: 283 @$\mu$s@ per loop
\end{lstlisting}
As you can see, the dimensions of the problem still change how long it will take.
In every case the final solution was the fastest.
Proper vectorization has, in this case, given a speedup by more than a factor of 1000 while still using a fairly reasonable amount of memory.
Plots of the times for these different functions can be seen in Figure \ref{distplot}.

\begin{figure}
\includegraphics[width=\textwidth]{distplot.pdf}
\caption{A plot of the times for the different degrees of vectorization for a function finding the distance between points represented as rows in two arrays. Arrays of points in $\mathbb{R}^{10}$ were used for this computation. As this figure suggests, vectorization speeds up the computation
by a multiplicative constant, but usually does not change the Big-Oh.}
\label{distplot}
\end{figure}

\begin{problem}
Use vectorization techniques to obtain an optimized solution to each of the following problems.
They can all be done in one line.

\begin{enumerate}[a)]

\item For an array \li{X} compute the inner product of each row with each other row.
Hint: Think matrix multiplication.

\item For an array of points \li{X} where each row represents a point in $\mathbb{R}^n$ compute $\| x\|^2$ for each row \li{x} of \li{X}.
Hint: Use array multiplication (element-wise multiplication) instead of matrix multiplication for this one.

\item Compute the matrix expression $v^T A v$ where $A$ is a square matrix and each column vector $v$ is stored as a row in an array \li{V}.
The shapes of \li{V} and \li{A} are \li{(n, d)} and \li{(n, n)} respectively.
Hint: Matrix multiplication, then array multiplicaiton, then a sum along an axis.

\item For an array \li{A} of size \li{n} count the number of elements of \li{A} less than $\frac{1}{2}$.
Hint: Sum over a boolean array.

\item Modify an array \li{A} of size \li{n} and set every value less than $\frac{1}{4}$ to $0$.
Hint: Use fancy indexing.

\item Given an array \li{A} of shape \li{(n, n)} representing a linear transform on $\mathbb{R}^n$, apply \li{A} to an array of points \li{X} where each row of \li{X} is a point in $\mathbb{R}^n$.
Hint: This is all about transposes and matrix multiplication

\item For an array \li{A} of shape \li{(m, n, n)} which represents $m$ linear transformations from $\mathbb{R}^n$ to $\mathbb{R}^n$ and an array \li{X} of shape \li{(k, n)} whose rows represent points in $\mathbb{R}^n$ compute the image of each row of \li{X} under each transformation represented in \li{A}.
Format the results so the have shape \li{(m, k, n)}.
Hint: Do a matrix multiplication, then use the \li{swapaxes()} method of an array to rearrange the axes properly.

\item Let \li{A} be an array of shape \li{(m, n)} and \li{B} be an array of shape \li{m}.
Take the sum of every element of \li{A} that lies in a row of \li{A} with a first element that is less than $\frac{1}{2}$.
Hint: Use matrix multiplication by a row vector of boolean values.

\item For an array \li{P} of shape \li{n} and a boolean array of shape \li{D} compute the array \li{A} such that \li{A[i]} is $\sum_{D[i,j] = True} \left( P[i] - P[j]\right)$.
Hint: For each \li{i} let $k_i = \sum_{D[i,j] = True} 1$ (the number of \li{True} values , then the \li{i}'th entry of \li{A} should be $k_i * P[i] - \sum_{D[i,j] = True} P[j]$.
The first part of this expression can be evaluated by array multiplication and a sum across an axis.
The second part can be evaluated with matrix multiplication.

\item Shuffle a one-dimensional array of values the way you would shuffle a deck of cards, i.e. separate it into two stacks, then take every other item from each of the two stacks.
Hint: This can be done with array slicing and multiple assignment (i.e. \li{a, b = c, d}).
To make the assignment work nicely with the slicing, be sure to use copies of both slices.
Note: In general this isn't how you would do this.
A purely random arrangement of rows can be obtained using \li{numpy.random.shuffle()}.

\end{enumerate}
\end{problem}

\section*{Image Editor}
An image is often represented as a 3D array where the first and second dimensions are the height and width respectively and the third dimension represents the intensity of each of three color channels: red, green, and blue.
The code below can be used to read and display images in Python.
\begin{lstlisting}
import matplotlib.pyplot as plt
img = plt.imread(<image-file>)
plt.imshow(img)
plt.show()
\end{lstlisting}

Matplotlib can read in PNG images on its own, otherwise it will read in images using the Python Image Library.
Depending on the format of the image, it may be read as an array of floating point values between 0 and 1 or it may be read as an array of integer values between 0 and 255.
In this lab we will assume the former. Load the image provided with the lab, entitled `musician.png':
\begin{lstlisting}
img = plt.imread('musician.png')
\end{lstlisting}
There is some extra data in this image that we do not need, so throw it out, and view the image:
\begin{lstlisting}
img = img[:,:,:-1]
plt.imshow(img)
plt.show()
\end{lstlisting}

%How is this problem different from the grayscale problem below?
%\begin{problem}
%Use vectorization techniques to obtain an optimized solution to each of the following problems.
%They can all be done in one line.
%\begin{enumerate}[a)]
%
%\item Given an array representing a color image, make an array representing a black and white image where the black and white value is the average of each of the color values.
%Hint: see the \li{mean()} method of arrays.
%
%\item For an array representing a color image calculate the absolute value of the difference between each color value and the average value for each pixel.
%Hint: pass the \li{keepdims=True} argument to the \li{mean} method so that broadcasting works properly.
%
%\end{enumerate}
%\end{problem}

\begin{problem}
Write a function that edits an array representing an image. The function should take as input
an array of shape (no. of rows, no. of columns, no. of color channels) and should then plot the edited
image. Include options that allow you to  invert the image, change it to grayscale, or add a motion blur (definitions below).
Note that the shape of a grayscale image is (no. of rows, no. of columns), and to plot a
grayscale image \li{A} you need to use the command \li{plt.imshow(A, cmap=plt.cm.Greys_r)}.
\end{problem}

\begin{itemize}
\item Invert:
Every color value for every pixel is changed to its inverse value.
For example, 0.0 becomes 1.0 and 0.4 becomes 0.6.
Remember that the minimum color value is 0.0 and the maximum is 1.0.

\begin{figure}[ht]
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{Original.jpg}
\end{minipage}
\hspace{0.1cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{Invert.jpg}
\end{minipage}
\caption{Example of an inverted image}
\end{figure}

\item Grayscale:
To convert a \li{(n,m,3)} image \li{A} to grayscale, you will create a \li{(n,m)} image \li{B} such that \li{B[i,j]} is the average of the values \li{A[i,j,0], A[i,j,1], A[i,j,2]}.
For example, if the $i,j$ pixel values are Red: 0.8, Green:, 0.2 Blue: 0.5, we set the
corresponding grayscale pixel value to $\frac{0.8+0.2+0.5}{3} = 0.5$.
Hint: use the \li{axis} argument to the mean method of NumPy arrays.

\begin{figure}[ht]
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{Original.jpg}
\end{minipage}
\hspace{0.1cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{Gray.jpg}
\end{minipage}
\caption{Example of an image that is grayscaled}
\end{figure}

\item Motion Blur:
An additional parameter $n>0$ will be used for motion blur.
The value of each color of each pixel is the average of that color value for $n$ pixels (from the current pixel to $n-1$ beyond) horizontally.
So pixel \li{[x,y,0]} would turn into the average of pixel \li{[x,y,0]} through pixel \li{[x,y+n-1,0]}.
Note: You will need to use one for-loop here.
Be sure to account for the situations where one or more of the values used in the computing the average do not exist.
For example, if an image has width $w$ and we are considering the pixel on row $r$, column $c$, if $c + n \ge w$, then we only average the pixels up to $w$.
Proper array slicing should take care of this case without any extra code.

\begin{figure}[ht]
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{Original.jpg}
\end{minipage}
\hspace{0.1cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{Motion.jpg}
\end{minipage}
\caption{Example of an image with a motion blur of 60}
\end{figure}

\end{itemize}


\documentclass[../main.tex]{subfiles}

\begin{document}

% The asterix after \subsection disables section numbering
\subsection*{Problem 2 Part A}
Determine the combination of ingredients that minimizes calories but meets all nutritional requirements.

\begin{enumerate}[i.]
	\item Formulate the problem as a linear program with an objective function and all constraints.

	We can formulate this problem as the following linear program:

	\begin{equation*}
		\begin{aligned}
			% Minimize calories
			& \text{minimize} & & 21 \cdot w_{\text{tomato}} + 16 \cdot  w_{\text{lettuce}} + 40 \cdot w_{\text{spinach}} + 41 \cdot w_{\text{carrot}} \\
			& & & + 585 \cdot w_{\text{sunflower seed}} + 120 \cdot w_{\text{smoked tofu}} + 164 \cdot w_{\text{chickpea}} + 884 \cdot w_{\text{oil}}\\ \\
			% Subject to >= 15g protein
			& \text{subject to} & & -0.85 \cdot w_{\text{tomato}} - 1.62 \cdot  w_{\text{lettuce}} - 2.86 \cdot w_{\text{spinach}} - 0.93 \cdot w_{\text{carrot}} \\
			& & & - 23.4 \cdot w_{\text{sunflower seed}} - 16 \cdot w_{\text{smoked tofu}} - 9 \cdot w_{\text{chickpea}} - 0 \cdot w_{\text{oil}} \leq -15, \\ \\
			% Subject to >= 2g fat
			& & & - 0.33 \cdot w_{\text{tomato}} - 0.2 \cdot  w_{\text{lettuce}} - 0.39 \cdot w_{\text{spinach}} - 0.24  \cdot w_{\text{carrot}} \\
			& & & - 48.7 \cdot w_{\text{sunflower seed}} - 5 \cdot w_{\text{smoked tofu}} - 2.6 \cdot w_{\text{chickpea}} - 100 \cdot w_{\text{oil}} \leq -2, \\ \\
			% Subject to <= 8g fat
			& & & 0.33 \cdot w_{\text{tomato}} + 0.2 \cdot  w_{\text{lettuce}} + 0.39 \cdot w_{\text{spinach}} + 0.24  \cdot w_{\text{carrot}} \\
			& & & + 48.7 \cdot w_{\text{sunflower seed}} + 5 \cdot w_{\text{smoked tofu}} + 2.6 \cdot w_{\text{chickpea}} + 100 \cdot w_{\text{oil}} \leq 8, \\ \\
			% Subject to >= 4g carbohydrates
			& & & - 4.64 \cdot w_{\text{tomato}} - 2.37 \cdot  w_{\text{lettuce}} - 3.63 \cdot w_{\text{spinach}} - 9.58  \cdot w_{\text{carrot}} \\
			& & & - 15 \cdot w_{\text{sunflower seed}} - 3 \cdot w_{\text{smoked tofu}} - 27 \cdot w_{\text{chickpea}} - 0 \cdot w_{\text{oil}} \leq -4, \\ \\
			% Subject to <= 200 mg sodium
			& & & 9 \cdot w_{\text{tomato}} + 28 \cdot  w_{\text{lettuce}} + 65 \cdot w_{\text{spinach}} + 69  \cdot w_{\text{carrot}} \\
			& & & + 3.8 \cdot w_{\text{sunflower seed}} + 120 \cdot w_{\text{smoked tofu}} + 78 \cdot w_{\text{chickpea}} + 0 \cdot w_{\text{oil}} \leq 200, \\ \\
			% Subject to >= 40% leafy greens by mass
			& & & 0.4 \cdot w_{\text{tomato}} - 0.6 \cdot  w_{\text{lettuce}} - 0.6 \cdot w_{\text{spinach}} + 0.4 \cdot w_{\text{carrot}} \\
			& & & + 0.4 \cdot w_{\text{sunflower seed}} + 0.4 \cdot w_{\text{smoked tofu}} + 0.4 \cdot w_{\text{chickpea}} + 0.4 \cdot w_{\text{oil}} \leq 0, \\ \\
			% Subject to weights greater than or equal to zero
			& & & - w_{\text{tomato}} \leq 0 \\
			& & & - w_{\text{lettuce}} \leq 0 \\
			& & & - w_{\text{spinach}} \leq 0 \\
			& & & - w_{\text{carrot}} \leq 0 \\
			& & & - w_{\text{sunflower seed}} \leq 0 \\
			& & & - w_{\text{smoked tofu}} \leq 0 \\
			& & & - w_{\text{chickpea}} \leq 0 \\
			& & & - w_{\text{oil}} \leq 0 \\
		\end{aligned}
	\end{equation*}

	where the $w$ parameters are the weight of each ingredient, in 100's of grams. The equations, from top to bottom, are:
	\begin{enumerate}[1)]
		\item Minimize total calories
		\item Subject to total protein $\geq$ 15 grams
		\item Subject to total fat $\geq$ 2 grams
		\item Subject to total fat $\leq$ 8 grams
		\item Subject to total carbohydrates $\geq$ 4 grams
		\item Subject to total sodium $\leq$ 200 milligrams
		\item Subject to total leafy green mass $\geq$ 40\% of total mass
		\item Subject to individual ingredient weights $\geq$ 0
	\end{enumerate}

	\item Determine the optimal solution for the linear program using any software you want. Include a copy of the code/file in the report.

	The following MATLAB code was used to generate the solution: \\

	\lstinputlisting{../problem_two/problem2_partA.m}

	where \verb|X| stores the resulting weights of ingredients (in 100's of grams), \verb|FVAL| stores the minimized number of calories, and \verb|EXITFLAG| stores the status of the \verb|linprog| optimization.

	\item What is the cost of the low calorie salad?

	The optimal low calorie salad contains the following weights of ingredients (in 100's of grams):

	\begin{equation*}
		\begin{aligned}
			w_{\text{tomato}} &\approx 0 \\
			w_{\text{lettuce}} &\approx 0.5855 \\
			w_{\text{spinach}} &\approx 0 \\
			w_{\text{carrot}} &\approx 0 \\
			w_{\text{sunflower seed}} &\approx 0 \\
			w_{\text{smoked tofu}} &\approx 0.8782 \\
			w_{\text{chickpea}} &\approx 0 \\
			w_{\text{oil}} &\approx 0 \\
		\end{aligned}
	\end{equation*}

	The optimal low calorie salad costs approximately \$2.33. It contains approximately $114.75$ kcal
\end{enumerate}

\newpage

\subsection*{Problem 2 Part B}
Veronica realizes that it is also important to minimize the cost associated with the new salad. Unfortunately some of the ingredients can be expensive. Determine the combination of ingredients that minimizes cost.

\begin{enumerate}[i.]
	\item Formulate the problem as a linear program with an objective function and all constraints.

	We can formulate this problem as the following linear program:

	\begin{equation*}
		\begin{aligned}
			% Minimize calories
			& \text{minimize} & & 1 \cdot w_{\text{tomato}} + 0.75 \cdot  w_{\text{lettuce}} + 0.5 \cdot w_{\text{spinach}} + 0.5 \cdot w_{\text{carrot}} \\
			& & & + 0.45 \cdot w_{\text{sunflower seed}} + 2.15 \cdot w_{\text{smoked tofu}} + 0.95 \cdot w_{\text{chickpea}} + 2.00 \cdot w_{\text{oil}}\\ \\
			% Subject to >= 15g protein
			& \text{subject to} & & -0.85 \cdot w_{\text{tomato}} - 1.62 \cdot  w_{\text{lettuce}} - 2.86 \cdot w_{\text{spinach}} - 0.93 \cdot w_{\text{carrot}} \\
			& & & - 23.4 \cdot w_{\text{sunflower seed}} - 16 \cdot w_{\text{smoked tofu}} - 9 \cdot w_{\text{chickpea}} - 0 \cdot w_{\text{oil}} \leq -15, \\ \\
			% Subject to >= 2g fat
			& & & - 0.33 \cdot w_{\text{tomato}} - 0.2 \cdot  w_{\text{lettuce}} - 0.39 \cdot w_{\text{spinach}} - 0.24  \cdot w_{\text{carrot}} \\
			& & & - 48.7 \cdot w_{\text{sunflower seed}} - 5 \cdot w_{\text{smoked tofu}} - 2.6 \cdot w_{\text{chickpea}} - 100 \cdot w_{\text{oil}} \leq -2, \\ \\
			% Subject to <= 8g fat
			& & & 0.33 \cdot w_{\text{tomato}} + 0.2 \cdot  w_{\text{lettuce}} + 0.39 \cdot w_{\text{spinach}} + 0.24  \cdot w_{\text{carrot}} \\
			& & & + 48.7 \cdot w_{\text{sunflower seed}} + 5 \cdot w_{\text{smoked tofu}} + 2.6 \cdot w_{\text{chickpea}} + 100 \cdot w_{\text{oil}} \leq 8, \\ \\
			% Subject to >= 4g carbohydrates
			& & & - 4.64 \cdot w_{\text{tomato}} - 2.37 \cdot  w_{\text{lettuce}} - 3.63 \cdot w_{\text{spinach}} - 9.58  \cdot w_{\text{carrot}} \\
			& & & - 15 \cdot w_{\text{sunflower seed}} - 3 \cdot w_{\text{smoked tofu}} - 27 \cdot w_{\text{chickpea}} - 0 \cdot w_{\text{oil}} \leq -4, \\ \\
			% Subject to <= 200 mg sodium
			& & & 9 \cdot w_{\text{tomato}} + 28 \cdot  w_{\text{lettuce}} + 65 \cdot w_{\text{spinach}} + 69  \cdot w_{\text{carrot}} \\
			& & & + 3.8 \cdot w_{\text{sunflower seed}} + 120 \cdot w_{\text{smoked tofu}} + 78 \cdot w_{\text{chickpea}} + 0 \cdot w_{\text{oil}} \leq 200, \\ \\
			% Subject to >= 40% leafy greens by mass
			& & & 0.4 \cdot w_{\text{tomato}} - 0.6 \cdot  w_{\text{lettuce}} - 0.6 \cdot w_{\text{spinach}} + 0.4 \cdot w_{\text{carrot}} \\
			& & & + 0.4 \cdot w_{\text{sunflower seed}} + 0.4 \cdot w_{\text{smoked tofu}} + 0.4 \cdot w_{\text{chickpea}} + 0.4 \cdot w_{\text{oil}} \leq 0, \\ \\
			% Subject to weights greater than or equal to zero
			& & & - w_{\text{tomato}} \leq 0 \\
			& & & - w_{\text{lettuce}} \leq 0 \\
			& & & - w_{\text{spinach}} \leq 0 \\
			& & & - w_{\text{carrot}} \leq 0 \\
			& & & - w_{\text{sunflower seed}} \leq 0 \\
			& & & - w_{\text{smoked tofu}} \leq 0 \\
			& & & - w_{\text{chickpea}} \leq 0 \\
			& & & - w_{\text{oil}} \leq 0 \\
		\end{aligned}
	\end{equation*}

	where the $w$ parameters are the weight of each ingredient, in 100's of grams. The equations, from top to bottom, are:
	\begin{enumerate}[1)]
		\item Minimize total cost
		\item Subject to total protein $\geq$ 15 grams
		\item Subject to total fat $\geq$ 2 grams
		\item Subject to total fat $\leq$ 8 grams
		\item Subject to total carbohydrates $\geq$ 4 grams
		\item Subject to total sodium $\leq$ 200 milligrams
		\item Subject to total leafy green mass $\geq$ 40\% of total mass
		\item Subject to individual ingredient weights $\geq$ 0
	\end{enumerate}

	\item Determine the optimal solution for the linear program using any software you want. Include a copy of the code/file in the report.

	The following MATLAB code was used to generate the solution: \\

	\lstinputlisting{../problem_two/problem2_partB.m}

	where \verb|X| stores the resulting weights of ingredients (in 100's of grams), \verb|FVAL| stores the minimized number of calories, and \verb|EXITFLAG| stores the status of the \verb|linprog| optimization.

	\item How many calories are in the low cost salad?

	The optimal low cost salad contains the following weights of ingredients (in 100's of grams):

	\begin{equation*}
		\begin{aligned}
			w_{\text{tomato}} &\approx 0 \\
			w_{\text{lettuce}} &\approx 0 \\
			w_{\text{spinach}} &\approx 0.8323 \\
			w_{\text{carrot}} &\approx 0 \\
			w_{\text{sunflower seed}} &\approx 0.0961 \\
			w_{\text{smoked tofu}} &\approx 0 \\
			w_{\text{chickpea}} &\approx 1.1524 \\
			w_{\text{oil}} &\approx 0 \\
		\end{aligned}
	\end{equation*}

	The optimal low cost salad costs approximately \$1.55. It contains approximately $278.49$ kcal

\end{enumerate}
\newpage

\subsection*{Problem 2 Part C}
Compare the results from part A and B. Veronica’s goal is to create a Very Veggie Salad that is both low calorie and low cost. She would like to sell the salad for \$5.00 and still have a profit of at least \$3.00. However if she can advertise that the salad has under 250 calories then she may be able to sell more.

\begin{enumerate}[i.]
	\item Suggest some possible ways that she select a combination of ingredients that is “near optimal” for both objectives. This is a type of multi-objective optimization.

	What Veronica should do is attempt to find a ``pareto optimal" solution that accounts for both calories and cost. In order to do this, she should introduce a new parameter $\lambda$ to her optimization formulation. In essence, she would like to:

	\begin{equation*}
		\begin{aligned}
			% Minimize price and calories
			&\text{minimize} & & (1 - \lambda) \cdot \text{CALORIES} + \lambda \cdot \text{PRICE} \\
			&\text{subject to} & & \text{CONSTRAINTS}
		\end{aligned}
	\end{equation*}

	for values of $\lambda$ between 0 and 1. When $\lambda = 0$, she will be finding the minimum calorie combination of ingredients. When $\lambda = 1$, she will be finding the minimum price combination of ingredients. For all values of $\lambda$ inbetween, she will be finding the ``pareto optimal" combination of ingredients. So, she should solve the optimization problem for values of $\lambda$ between 0 and 1. She can then examine these pareto optimal combinations of ingredients to determine which would best meet her goals of \$3.00 profit and under 250 calories. \\

	We can express this as the following linear program:

	\begin{equation*}
		\begin{aligned}
			% Minimize calories
			& \text{minimize} & & (1 - \lambda) \cdot (21 \cdot w_{\text{tomato}} + 16 \cdot  w_{\text{lettuce}} + 40 \cdot w_{\text{spinach}} + 41 \cdot w_{\text{carrot}} \\
			& & & + 585 \cdot w_{\text{sunflower seed}} + 120 \cdot w_{\text{smoked tofu}} + 164 \cdot w_{\text{chickpea}} + 884 \cdot w_{\text{oil}} )\\
			& & & + \lambda \cdot (1 \cdot w_{\text{tomato}} + 0.75 \cdot  w_{\text{lettuce}} + 0.5 \cdot w_{\text{spinach}} + 0.5 \cdot w_{\text{carrot}} \\
			& & & + 0.45 \cdot w_{\text{sunflower seed}} + 2.15 \cdot w_{\text{smoked tofu}} + 0.95 \cdot w_{\text{chickpea}} + 2.00 \cdot w_{\text{oil}} )\\ \\
			% Subject to >= 15g protein
			& \text{subject to} & & -0.85 \cdot w_{\text{tomato}} - 1.62 \cdot  w_{\text{lettuce}} - 2.86 \cdot w_{\text{spinach}} - 0.93 \cdot w_{\text{carrot}} \\
			& & & - 23.4 \cdot w_{\text{sunflower seed}} - 16 \cdot w_{\text{smoked tofu}} - 9 \cdot w_{\text{chickpea}} - 0 \cdot w_{\text{oil}} \leq -15, \\ \\
			% Subject to >= 2g fat
			& & & - 0.33 \cdot w_{\text{tomato}} - 0.2 \cdot  w_{\text{lettuce}} - 0.39 \cdot w_{\text{spinach}} - 0.24  \cdot w_{\text{carrot}} \\
			& & & - 48.7 \cdot w_{\text{sunflower seed}} - 5 \cdot w_{\text{smoked tofu}} - 2.6 \cdot w_{\text{chickpea}} - 100 \cdot w_{\text{oil}} \leq -2, \\ \\
			% Subject to <= 8g fat
			& & & 0.33 \cdot w_{\text{tomato}} + 0.2 \cdot  w_{\text{lettuce}} + 0.39 \cdot w_{\text{spinach}} + 0.24  \cdot w_{\text{carrot}} \\
			& & & + 48.7 \cdot w_{\text{sunflower seed}} + 5 \cdot w_{\text{smoked tofu}} + 2.6 \cdot w_{\text{chickpea}} + 100 \cdot w_{\text{oil}} \leq 8, \\ \\
			% Subject to >= 4g carbohydrates
			& & & - 4.64 \cdot w_{\text{tomato}} - 2.37 \cdot  w_{\text{lettuce}} - 3.63 \cdot w_{\text{spinach}} - 9.58  \cdot w_{\text{carrot}} \\
			& & & - 15 \cdot w_{\text{sunflower seed}} - 3 \cdot w_{\text{smoked tofu}} - 27 \cdot w_{\text{chickpea}} - 0 \cdot w_{\text{oil}} \leq -4, \\ \\
			% Subject to <= 200 mg sodium
			& & & 9 \cdot w_{\text{tomato}} + 28 \cdot  w_{\text{lettuce}} + 65 \cdot w_{\text{spinach}} + 69  \cdot w_{\text{carrot}} \\
			& & & + 3.8 \cdot w_{\text{sunflower seed}} + 120 \cdot w_{\text{smoked tofu}} + 78 \cdot w_{\text{chickpea}} + 0 \cdot w_{\text{oil}} \leq 200, \\ \\
			% Subject to >= 40% leafy greens by mass
			& & & 0.4 \cdot w_{\text{tomato}} - 0.6 \cdot  w_{\text{lettuce}} - 0.6 \cdot w_{\text{spinach}} + 0.4 \cdot w_{\text{carrot}} \\
			& & & + 0.4 \cdot w_{\text{sunflower seed}} + 0.4 \cdot w_{\text{smoked tofu}} + 0.4 \cdot w_{\text{chickpea}} + 0.4 \cdot w_{\text{oil}} \leq 0, \\ \\
			% Subject to weights greater than or equal to zero
			& & & - w_{\text{tomato}} \leq 0 \\
			& & & - w_{\text{lettuce}} \leq 0 \\
			& & & - w_{\text{spinach}} \leq 0 \\
			& & & - w_{\text{carrot}} \leq 0 \\
			& & & - w_{\text{sunflower seed}} \leq 0 \\
			& & & - w_{\text{smoked tofu}} \leq 0 \\
			& & & - w_{\text{chickpea}} \leq 0 \\
			& & & - w_{\text{oil}} \leq 0 \\
		\end{aligned}
	\end{equation*}

	\item What combination of ingredient would you suggest and what is the associated cost and calorie.

	Varying $\lambda$ between 0 and 1, and solving the resulting optimization problems, we obtain the following pareto optimal combinations of total calories and total price:

	\begin{center}
		\includegraphics{../problem_two/problem2_partC}
	\end{center}

	Based on this result, I would suggest the following combination of ingredients:

	\begin{equation*}
		\begin{aligned}
			w_{\text{tomato}} &\approx 0 \\
			w_{\text{lettuce}} &\approx 0 \\
			w_{\text{spinach}} &\approx 0.5346 \\
			w_{\text{carrot}} &\approx 0 \\
			w_{\text{sunflower seed}} &\approx 0.0865 \\
			w_{\text{smoked tofu}} &\approx 0.7154 \\
			w_{\text{chickpea}} &\approx 0 \\
			w_{\text{oil}} &\approx 0 \\
		\end{aligned}
	\end{equation*}

	This pareto optimal salad costs approximately \$1.84. It contains approximately $157.86$ kcal. This meets both of Veronicas goals.

	\item Note: There is not one “right” answer. Discuss how you derived your solution.

	The following MATLAB code was used to derive the answer, based on the discussion above.

	\newpage

	\lstinputlisting{../problem_two/problem2_partC.m}

\end{enumerate}

\end{document}


\subsection{Why Can't We Just Modify the Feature Vector?}
In some data science and machine learning circles, a common ad hoc way to avoid centering data prior to solving the least squares problem is to append a constant dimension to the feature vector, such that the new feature vectors are given by
\begin{equation}
  \lambda'_t =
  \begin{bmatrix}
    \lambda_t & 1
  \end{bmatrix}
\end{equation}
Intuitively, the intention of this technique is to add an additional offset
parameter $\theta_0$ to the least squares estimated parameters, which should
then capture the constant offset term $\mu_y - \mu_x \hat\Theta$ which we
calculated analytically above. In practice, though, this technique gives rise
to an entire one-dimensional subspace of possible solutions to the
least-squares problem, as can be shown using a bit of linear algebra, where the
actual solution returned by solving the normal equations is determined by the
particular numerical algorithm used to invert the covariance matrix. The
addition of a constant feature to all input data makes the sample covariance
matrix $X_T^\top X_T$ low-rank, so that the inverse $(X_T^\top X_T)^{-1}$ is
ill-posed and gives rise to a subspace of possible solutions. The best solution
of this subspace in expectation is precisely the one derived in
Section~\ref{sec:uncentered}, but the feature vector augmentation technique
gives no guarantees of recovering this solution.

\subsection{Initializing $P_T$ and $Q_T$ and Connections to Ridge Regression}
The update equations derived in Sections~\ref{sec:centered}
and~\ref{sec:uncentered} tell use how to move from the least-squares solution
at timestep $T-1$ to the least-squares solution at timestep $T$, but they don't
tell us how the relevant matrices should be initialized. For all involved
matrices except $P_T$ (or $Q_T$, in the uncentered case), it is reasonable to
initialize with matrices whose elements are all 0. If we initialize $P_T$ or
$Q_T$ to zero matrices, however, Equations~\ref{eq:centered_p_update} and~\ref{eq:uncentered_Q_T_update} show that these matrices will never be updated at all (since the update equations are multiplicative in $P_T$ and $Q_T$, respectively).

This means that we need to initialize $P_T$ and $Q_T$ to some nonzero matrix
before beginning the recursive least squares algorithm. In practice, this
initialization matrix is usually chosen as some multiple of the identity, so
that $P_0 = \alpha I$. This has a significant effect on the computed $\hat\Theta$, however, as can be seen if we examine the real normal equation in this situation:
\begin{equation}
  \hat\Theta_{REAL}(T) = (X_T^\top X_T + \frac{1}{\alpha} I)^{-1}X_T^\top Y
\end{equation}
This is precisely the solution for the ridge regression problem (also known as
$l^2$ regularized least squares or Tikhonov regularization) with regularization
coefficient $\frac{1}{\alpha}$. Thus any practical implementation of recursive
least squares which keeps around an estimate of the inverse covariance matrix
($P_T$ or $Q_T$ in our notation) is in fact computing a ridge regression
estimator. This explains the common advice to use $\alpha \approx 10^6$; a
large value for $\alpha$ corresponds to a low amount of $l^2$ regularization,
and hence closer approximation to the unregularized least squares solution.

\subsection{Forgetting Factors for Time-Varying Systems}
When the relationship between $\varphi_t$ and $y_t$ is assumed to change over
time, we need some way of prioritizing recent data over historical data in
recursive least squares. This is commonly done via a ``forgetting factor''
$\lambda \in [0, 1]$. $\lambda$ is used to give exponentially smaller weight to
older samples in the regression in a way that can be intuitively explained by
its extremal values: when $\lambda = 0$ no datum prior to the current timestep
is taken into account, and when $\lambda = 1$ we recover the recursive least
squares algorithm derived above. Common values of $\lambda$ lie between $0.95$
and $0.99$.

The way that this is practically done is by reweighting the rows of the data matrix $X$ and target matrix $Y$. Where previously these were defined as simply the vertically stacked samples, we now define them as
\begin{align}
  &X =
  \begin{bmatrix}
    \varphi_T \\
    \lambda \varphi_{T-1} \\
    \vdots \\
    \lambda^{T - 2} \varphi_2 \\
    \lambda^{T - 1} \varphi_1
  \end{bmatrix}
  &Y =
  \begin{bmatrix}
    y_T \\
    \lambda y_{T-1} \\
    \vdots \\
    \lambda^{T - 2} y_2 \\
    \lambda^{T - 1} y_1
  \end{bmatrix}
\end{align}
Similarly we redefine $\mu_x(T)$ and $\mu_y(T)$ to be the means of these new matrices:
\begin{align}
  &\mu_x(T) = \frac{1}{T}\sum_{t=1}^T \lambda^{T - t} \varphi_t
  &\mu_y(T) = \frac{1}{T}\sum_{t=1}^T \lambda^{T - t} y_t
\end{align}
By carrying this new $X$ and $Y$ through the same derivation as in the uncentered $X$, uncentered $Y$ case above, we can derive analogous matrices and update rules:
\begin{align}
  &C_T := \frac{1}{\lambda^2T^2}
  \begin{bmatrix}
    (2T - 1)^2 - 2T^2 & -(2T - 1)(T - 1) \\
    -(2T - 1)(T - 1) & (T - 1)^2
  \end{bmatrix}
  &V_T :=
  \begin{bmatrix}
    \lambda\mu_x(T - 1) \\
    \varphi_T
  \end{bmatrix} \\
  &R_T := V_T^\top C_T V_T
\end{align}
\begin{equation}
  Q_T = \frac{1}{\lambda^2} \left[Q_{T-1} - Q_{T-1}V_T^\top\left(C_T^{-1} + V_TQ_{T-1}V_T^\top\right)^{-1}V_TQ_{T-1}\right]
\end{equation}
\begin{align}
  \hat\Theta_{RAW}(T) &= \hat\Theta_{RAW}(T-1) + Q_T\left[\varphi_T^\top y_T - R_T\hat\Theta_{RAW}(T-1)\right]\\
  \hat\Theta_{LS}(T) &= \hat\Theta_{RAW}(T) - (2T - 1)Q_T\mu_x(T)^\top \mu_y(T)
\end{align}

One final note is in order about the recursive least squares algorithm with
$\lambda$-forgetting: The connection that we drew earlier between the
initialization of $Q_T$ and $l^2$ regularized least squares no longer holds, as
the initial setting of $Q_T$ is multiplied by $\lambda^2$ at each timestep, and so
the equivalent regularization coefficient gets smaller with each new
datapoint. More precisely, the recursive least squares solution with
$\lambda$-forgetting and an initialization of $Q_0 = \alpha I$ will, at time
$T$, compute the equivalent ridge regression solution
\begin{equation}
  \hat\Theta_{REAL} = (X_T^\top X_T + \frac{\lambda^{2T}}{\alpha}I)^{-1}X_T^\top Y
\end{equation}

This may be desirable if one wishes to smoothly interpolate between
the ridge regression solution when little data is available and the
unregularized least squares solution in the limit of infinite data, but we are
not aware of any existing statistical analysis of this interpolation.


Let's invent a proposition, and call it $p$. $p$ is semantically
equivalent to a ``statement''. A ``proof of $p$'' is a proof that $p$
is true. To say

\begin{zz}
  f \text{ is a proof of } p,
\end{zz}

we write

\begin{zz}
  f : p.
\end{zz}

\begin{remark}
  Outside of certain circles, the $f : p$ notation is very
  non-standard. Later in the book, it will become clear why we use it.

  I often, when writing, use the $:$ symbol to be short for ``such
  that''. Again, you'll see why all of this makes sense later.
\end{remark}

\begin{aside}
  In mathematics, it's very common to see notation you think is
  confusing, only to discover---often several years later---that the
  notation makes perfect sense and is actually brilliant.
\end{aside}

The most basic proofs begin with \term{axioms} (also called
\term{postulates}) and then prove \term{theorems} (also called
\term{lemmas}). However, most proofs incorporate previously proved
lemmas to create new lemmas. If we had to start from the most basic
axioms every time we proved anything, nothing would ever get done.

\begin{definition}
  \label{def:axiom}
  \label{def:postulate}
  An \term{axiom} (or \term{postulate}) is a proposition that we
  accept to be true without proof.
\end{definition}

\begin{remark}
  \label{r:law}
  In most contexts, the term ``law'' is ambiguous. Sometimes, it
  refers to theorems, sometimes it refers to lemmas. I won't call
  anything that I define a ``law''. However, I will call things
  ``laws'' if some vocabulary term is commonly known as ``the law of
  X'' or ``X's law''. (See, for instance, \cref{ax:modus-ponens}).
\end{remark}

\begin{definition}
  \label{def:theorem}
  A \term{theorem} (or \term{lemma}) is a proposition that can be
  proven from other axioms or theorems.
\end{definition}

\subsection{Implication}

The first thing in logic is the notion of ``implication''. We write
$a \implies b$ to say ``$a$ implies $b$''. This means ``if $a$ is
true, then $b$ is also true'', or simply ``if $a$ then $b$''.

\begin{axiom}[Law of Modus Ponens]
  \label{ax:modus-ponens}
  If $a$ implies $b$, and we know $a$ is true, then we can conclude
  $b$.

  In other words, ``$a$ implies $b$. $a$, therefore $b$''.
\end{axiom}

Usually, three dots arranged in a triangle are used to write
``therefore'': $\therefore$.  The statement in \cref{ax:modus-ponens}
can therefore be rewritten as:

\begin{alignmath}{r}
  a \implies b.\; a,\, \therefore b
\end{alignmath}

Sometimes, it's semantically convenient to write $b \impliedby a$,
which just means ``$b$ is implied by $a$''. It's the same as
$a \implies b$. This is also sometimes said ``$b$ if $a$''. This
means, ``assuming $a$, we can prove $b$''.

We'll also often write $a \iff b$ which means that $a$ implies $b$,
\xtb{and} $b$ implies $a$. Usually, you'll hear this pronounced ``$a$
if and only if $b$''. In a minute, we'll get to why people say ``if
and only if'', as opposed to something like ``implies and is implied
by''.

\begin{remark}
  Given two propositions $a$ and $b$, $a \implies b$ is actually a
  proposition. The proposition is, $a$ implies $b$.
\end{remark}

Here are some more or less ``obvious'' properties of logic

\begin{axiom}[Reflexive property]
  \label{ax:props-refl}
  For all propositions $a$, $a \implies a$.
\end{axiom}

\begin{lemma}[Transitive property]
  \label{ax:props-trans}
  For all propositions $a$, $b$, and $c$, if $a \implies b$, and $b
  \implies c$, then $a \implies c$
\end{lemma}

\begin{proof}
  Given $f : a \implies b$, and $g : b \implies c$, we are to prove $a
  \implies c$. That is, assuming $a$, prove $c$. Let $h$ be our proof
  of $a$ (i.e. $h : a$)
\end{proof}

\subsection{Negation}

Quite often, we'll need to declare that something is false. To say
that ``$a$ is false'', you write $\lnot a$, pronounced ``not $a$''.

\begin{axiom}[Double negation]
  The proposition $\lnot(\lnot a)$ is equivalent to $a$.
\end{axiom}

\begin{remark}
  In many (indeed, most) logics, double negation is not accepted as an
  axiom. Many important theorems rely on double negation.

  For instance, the Intermediate Value Theorem in Calculus states
  that, given two points on $a$ and $b$ that are values of a
  ``continuous'' function (a function whose graph does not have any
  gaps or poles), the function touches every single point between $a$
  and $b$. Every known proof of the Intermediate Value Theorem relies
  on double negation. More specifically, it relies on the
  contrapositive, which we'll define in a second.
\end{remark}

\subsection{The contrapositive}

Many logics define two values, $\top$, and $\bot$, called ``top'' and
``bottom'', respectively. $\top$ is always provable, and $\bot$ is
never provable. In those logics, the proposition $\lnot a$ is
equivalent to $a \implies \bot$. If you can prove $a$ implies
something that is never provable, you can conclude that $a$ is false.

This is the common logical technique of \term{reductio ad absurdum},
or \term{proof by contradiction}. ``Reductio ad absurdum'' is Latin
for ``reduction to the absurd''.

\begin{definition}
  Given the proposition $a \implies b$, the proposition
  $\lnot b \implies \lnot a$ is called the \term{contrapositive}.
\end{definition}

\begin{theorem}[Reductio ad absurdum]
  \label{reductio-ad-absurdum}
  Every proposition of the form $a \implies b$ is logically equivalent
  to its contrapositive.
\end{theorem}

The fact that a proposition is logically equivalent to its
contrapositive is not at all obvious. Let's go through some examples.

\begin{example}
  If someone is decapitated, then they are dead (at least within a few
  seconds). Therefore,

  \begin{zz}
    \text{Decapitated} \implies \text{Dead}
  \end{zz}

  \answergraph{images/alas-poor-yorick.png}

  Therefore, if someone is alive, you can conclude that they have not
  been decapitated. Thus,

  \begin{zz}
    \text{not Dead} \implies \text{not Decapitated}
  \end{zz}

  This is the contrapositive.
\end{example}

We will circle back to proving the equivalence between a proposition
and its contrapositive later on. To do this, we need to know what
``equivalence'' is. The next section deals with defining equivalence.

\section{Equivalence}

What exactly does it mean for two propositions to be equivalent? What
does it mean for two numbers to be equivalent? What does it mean for
two shapes to be equivalent?

You are probably tempted to say ``if $p = q$, then $p$ and $q$ are
equivalent''! Problem solved, right?

Well\dots not really. Think about this: the proposition $p \implies q$
is equivalent to $\lnot q \implies \lnot p$. They are clearly not the
same proposition, but they are logically equivalent\dots

How do we universally define equivalence? Think about it for a
second. In fact, I'm going to insert a page break so you can think
about this without cheating and reading the answer. Go for a walk, get
a cup of coffee, whatever it is you do, and think about this\dots

\newpage

Alright, you're back! The answer that mathematicians use is to define
what are called \term{equivalence relations}. What on earth is an
equivalence relation? For that matter, what is a ``relation''?

We're going to explain all of that, as well as define an equivalence
relation for propositions.

Strictly speaking, a ``relation'' is something that takes one or more
objects, and uses them to form a proposition. In this case, we're
going to be studying relations that take two objects, called ``binary
relations''. (The word ``binary'' means ``of or relating to two'').

\begin{definition}
  A \term{binary relation} is something that takes exactly two
  objects, and determines whether or not something is true about them.
\end{definition}

There are, in more advanced areas, relations that require more than
two things. However, for now, when I say ``relation'', I mean
``binary relation''.

\begin{aside}
  There are also \term{unary relations} which take only one object. I
  don't think calling them ``relations'' makes a whole lot of sense,
  because the term ``relation'' implies that you are comparing one
  thing \xti{relative} to other things. If there are no other things
  to compare, the term ``relation'' doesn't make any
  sense. Nonetheless, people use the term all the time.
\end{aside}

You already know quite a few relations.

\begin{enumerate}
\item Given two numbers $a$ and $b$, $=$ is a relation. It takes two
  numbers and decides if they are equal.
\item The symbol $\le$ is also a relation over numbers. It takes two
  numbers, and decides if the first is less than or equal to the
  second.
\item The symbol $\implies$ is also a relation. It takes two
  propositions, and decides if the first implies the second.
\end{enumerate}

What is something we can say about $=$ that we can't say about $\le$?

The answer is that $=$ is an ``equivalence relation''. Meaning that it
satisfies the properties in the next definition

\begin{definition}
  An \term{equivalence relation} is a binary relation $\sim$ satisfying
  these properties:

  \begin{enumerate}
  \item for all $a$, $a \sim a$ (reflexive property),
  \item for all $a$ and $b$, $a \sim b$ implies $b \sim a$ (symmetric
    property),
  \item for all $a$, $b$ and $c$, $a \sim b$ and $b \sim c$ imply
    $a \sim c$ (transitive property).
  \end{enumerate}
\end{definition}

The standard equivalence relation is $=$. It has all of those
properties.

If you'll notice, $\le$ and $\implies$ have all of those properties,
except symmetry.

Is there a relation for propositions that has symmetry? (That is, an
equivalence relation for propositions).

Think about it for a minute. I'll give you another page break\dots

\newpage

Turns out, there is an equivalence relation! The relation is
$\iff$. Think about it:

\begin{description}
\item[Reflexivity] For all propositions $p$, $p \iff
  p$. (\Cref{ax:props-refl})
\item[Symmetry] For all propositions $p$ and $q$, $p \iff q$ implies
  $q \iff p$.
\item[Transitivity] For all propositions $p$, $q$ and $r$, $p \iff q$
  and $q \iff r$ imply $p \iff r$. (\Cref{ax:props-trans})
\end{description}

Can you think of some other equivalence relations? Many of them are
not at all obvious.

\begin{example}
  Analog clocks and watches give the time in 12-hour format. However,
  there are 24 hours in a day. When you say ``14:00 is the same as
  2:00'', you're actually saying ``14 is congruent to 2, modulo
  12''. The way we write this is with

  \begin{zz}
    14 \equiv 2 \mod 12
  \end{zz}

  What exactly is ``modulo''? Technically speaking, $n$ modulo $m$ is
  the remainder of dividing $n$ by $m$. So, $14 / 12 = 1$ with a
  remainder of $2$. We'll get to this more in-depth when we discuss
  arithmetic.

  Here's a fun thing, you can add two times together to get a new
  time. For instance, let's say you go into work at 9:00 in the
  morning, and stay for 8 hours. $9 + 8 = 17$. However,
  $17 \equiv 5 \mod 12$, which means you stay until 5 PM. You've been
  secretly using this equivalence relation your entire life.

  We'll learn later, in the Algebra section, that integers modulo 12
  (i.e. numbers on a clock) form a \term{module} of the integers
  (hence the term ``modulo''). Meaning this:

  \begin{enumerate}
  \item Given two times, $a$ and $b$ in the module, $a + b$ is also in
    the module.
  \item Given an integer\footnote{An integer is a positive, negative
      or zero whole number.} $z$, and a time $t$, $zt$ is also in the
    subgroup.\footnote{The notation $zt$ just means ``$z$ times
      $t$''.}
  \end{enumerate}

  There are a number of other properties that have to hold, but those
  are the properties unique to modules.

  Can you think of some other modules?

  I'll give you a hint: even numbers are also a module.
\end{example}

\subsection{Contrapositive}

Circling back to the contrapositive: I claimed a proposition of the
form $a \implies b$ is equivalent to its contrapositive. Now that
we've defined equivalence, we're much more equipped to prove this.

Two propositions $p$ and $q$ are equivalent if\footnote{Actually, it's
  ``if and only if'', because two propositions $p$ and $q$ being
  ``equivalent'' means $p \iff q$. However, it's conventional to say
  ``if'' when parroting a definition.} $p \iff q$. Therefore,

\begin{lemma}
  \label{thm:contrapositive}
  $(a \implies b) \iff (\lnot b \implies \lnot a)$.
\end{lemma}

\begin{proof}
  The most common way to prove an ``if and only if'' statement is to
  separately prove the $\implies$ branch, and the $\impliedby$ branch.

  \begin{description}
  \item[Branch $\implies$] Given $a \implies b$, we are to prove
    $\lnot b \implies \lnot a$. First, we expand $\lnot b$ and $\lnot
    a$ into

    \begin{rcl}
      \lnot a & \implies & (a \implies \bot) \\
      \lnot b & \implies & (b \implies \bot) \\
    \end{rcl}

    Therefore, we are to prove
    $(b \implies \bot) \implies (a \implies \bot)$. We are trying to
    prove ``if we can show $b \implies \bot$, we can show
    $a \implies \bot$. Therefore, we can assume $b \implies \bot$, and
    then use that to prove $a \implies \bot$.

    It might be helpful to assign labels to things. Let

    \begin{rcl}
      f & : & a \implies b \\
      g & : & b \implies \bot \\
      \tabucline \\
      \text{goal} & : & a \implies \bot \\
    \end{rcl}

    We could either use the transitive property
  \end{description}
\end{proof}


\subsection{Mahalanobis distance}

We have a point. How far away is this from the mean.

For a single dimension: number of standard deviations.

What about multidimensional data?

Could do sd for all distances, but correlations between variables. If two variables are highly correlated, it's not really twice as far.

We use this:

\(D_M(\mathbf x)=\sqrt {(\mathbf x-\mathbf \mu )^TS^{-1}(\mathbf x-\mathbf \mu )}\)


\section{The M\"obius Group}
We want to study $f:\mathbb C\to\mathbb C$ in the form
$$f(x)=\frac{ax+b}{cx+d},a,b,c,d\in\mathbb C$$
This function has a pole at $x=-d/c$, so we need an element at infinity.
We can take $\mathbb C_\infty:=\mathbb C\cup\{\infty\}$ by the stereographic projection $\mathbb C_{\infty}=\mathbb C\cup\{\infty\}\cong S^2$.
Now we define the Mobius map properly
\begin{definition}
    The Mobius map $f:\mathbb C_\infty\to\mathbb C_\infty$ is defined by
    $$
    f(z)=
    \begin{cases}
        \frac{az+b}{cz+d}\text{, if $z\neq\infty$ and $z\neq -d/c$}\\
        \infty\text{, if $z=-d/c$}\\
        \frac{a}{c}\text{, if $z=\infty$}
    \end{cases}
    $$
    where $ad-bc\neq 0$.
\end{definition}
The reason why we impose the last condiciton is that we want the Mobius map to be a bijection from $\mathbb C_\infty$ to $\mathbb C_\infty$.
\begin{proposition}
    Let $\mathcal M=\{f:\mathbb C_\infty\to\mathbb C_\infty:f\text{ is a Mobius function.}\}$.
    Then $(\mathcal M,\circ,\mathrm{id})$ is a group.
\end{proposition}
\begin{proof}
    Obviously $\mathrm{id}\in\mathcal M$.
    Note also that if $g(z)=(az+b)/(cz+d), g'(z)=(a'z+b')/(c'z+d')$, then $g'(g(z))=(a''z+b'')/(c''z+d'')$ where
    $$
    \begin{pmatrix}
        a''&b''\\
        c''&d''
    \end{pmatrix}
    =
    \begin{pmatrix}
        a'&b'\\
        c'&d'
    \end{pmatrix}
    \begin{pmatrix}
        a&b\\
        c&d
    \end{pmatrix}
    $$
    Then it immediately tells us that $\mathcal M$ is closed under $\circ$ since the determinant function is multiplicative.
    Note as well that it also gives us the inverse by just finding some $a',b',c',d'$ (which exists due to our criterion on determinant) such that
    $$
    \begin{pmatrix}
        a'&b'\\
        c'&d'
    \end{pmatrix}
    =
    \begin{pmatrix}
        a&b\\
        c&d
    \end{pmatrix}^{-1}
    $$
    by noticing that the identity function corresponds to $cI,c\neq 0$.
\end{proof}
We can have $\mathcal M$ to act on $\mathbb C^\infty$ faithfully (with trivial kernel), so $\mathcal M\le\operatorname{Sym}\mathbb C_\infty$.
Now consider the Mobius transformation $f(z)=1/(z-a)$, which sends $a$ to $\infty$ and its inverse that sends $\infty$ to $a$, so there is nothing special with $\infty$ in $\mathbb C^\infty$, as one will expect as there is no special point on $S^2$.
\begin{proposition}\label{decomp_mobius}
    Every Mobius tranformation is a composition of $z\mapsto az,a\neq 0$, $z\mapsto z+b$ and $z\mapsto 1/z$.
\end{proposition}
\begin{proof}
    Let $z\mapsto (az+b)/(cz+d)$ be a mobius transformation, then if $c=0$ the proposition is trivial.
    Otherwise $c\neq 0$, then we have
    $$\frac{az+b}{cz+d}=\frac{a}{c}-\frac{ad-bc}{c(cz+d)}$$
    which can obviously be obtained from the said functions.
\end{proof}
Now, how about fixed point of a Mobius transformation?
We know that a Mobius transformation fixes at least $1$ point, but how about more?
\begin{proposition}
    A Mobius transformation fixes $3$ points is the identity.
\end{proposition}
\begin{proof}
    Suppose
    $$f:z\mapsto\frac{az+b}{cz+d}$$
    If $\infty$ is a fixed point, then $c=0$, so $f$ is a linear function.
    But then a linear function that fixes $2$ (non-infinity) points is the identity (since $f(x)-x$ is linear and a linear function has exactly $1$ root unless it is constantly $0$), $f$ is the identity.\\
    Now if $\infty$ is not a fixed point, then
    $$f(z)=z\iff \frac{az+b}{cz+d}-z=0\iff az+b-z(cz+d)=0$$
    which has at most $2$ roots (hence fixed point) since it is quadratic, unless it is the zero function, which essentially means that $c=b=0,d=a\neq 0\implies f=\mathrm{id}$.
\end{proof}
\begin{proposition}\label{mobius_3pts}
    Given distinct $z_1,z_2,z_3\in\mathbb C_\infty$ and $w_1,w_2,w_3\in\mathbb C$, then there is an unique Mobius transformation $f$ such that $f(z_i)=w_i$ for $i\in\{1,2,3\}$.
\end{proposition}
Note that since every Mobius transformation is a bijective (hence invertible), $w_i$'s are distinct as well.
\begin{proof}
    For existence, it suffices to deal with the case where $w_1,w_2,w_3$ are $0,1,\infty$, since once we've found maps $f,g$ such that $f:z_1,z_2,z_3\mapsto 0,1,\infty,g:w_1,w_2,w_3\mapsto 0,1,\infty$, then $g^{-1}\circ f$ will send $z_1,z_2,z_3$ to $w_1,w_2,w_3$.\\
    Now if none of $z_i$'s is $\infty$, we can use the interpolation
    $$f(z)=\frac{(z-z_2)(z-z_3)}{(z_1-z_2)(z_2-z_3)}+\frac{(z-z_1)(z-z_2)}{z-z_3}$$
    Otherwise, suppose $z_i=\infty$, then the map $f_i$ suffices where
    $$f_1(z)=\frac{z-z_2}{z-z_3},f_2(z)=\frac{z_1-z_3}{z-z_3}, f_3(z)=\frac{z-z_2}{z_1-z_2}$$
    For uniqueness, suppose $f,f'$ send $z_1,z_2,z_3$ to $w_1,w_2,w_3$ respectly, then $f^{-1}\circ f'$ fixes $z_1,z_2,z_3$, hence $f^{-1}\circ f'=\mathrm{id}\implies f=f'$.
\end{proof}
If $f,g\in\mathcal M$ and $f$ fixes $z_0$, then $gfg^{-1}$ fixes $g(z_0)$, which gives rise to the following observation
\begin{theorem}
    %There are three conjugacy classes of $\mathcal M$, namely the identity alone, the functions with exactly one fixed point, and the functions with exactly $2$ fixed points.
    Every member of a conjugacy class of $\mathcal M$ has the same number of fixed point(s).
\end{theorem}
\begin{proof}
    Obviously the identity itself is itself a conjugacy class.
    Now for any nonidentity $f$, $f$ has either $1$ or $2$ fixed points.\\
    If $f$ has $1$ fixed point $z_0\neq\infty$, then suppose $g(z)=1/(z-z_0)$, we know that $gfg^{-1}$ fixes $\infty$, and it cannot fix any other points because if so then applying $g^{-1}$ to that point would produce another fixed point of $f$.
    So it has to be the map $z\mapsto z+b,b\neq 0$.\\
    If $f$ has $2$ fixed point, then we consider a Mobius transformation $g$ which sends the fixed points to $0,\infty$, then $gfg^{-1}$ fixed $0$ to $\infty$ and sends $1$ to $a\in\mathbb C\neq 0,\infty$, so there is exactly one Mobius transformation $z\mapsto az, a\neq 1$.
\end{proof}
Note that $(g^{-1}fg)^n=g^{-1}f^ng$.
This allows us to compute the arbitrary (integral) power of a Mobius transformation.
\begin{definition}
    The circle in the extended complex numbers is the equation $Az\bar z+\bar Bz+B\bar z+C=0$ with $A,C\in\mathbb R,B\in\mathbb C$.
    Consider $\infty$ is a point on this circle if and only if $A=0$.
\end{definition}
Note that circles in $\mathbb C$ are also circles in $\mathbb C_\infty$, and all other $\mathbb C_\infty$ circles are lines in $\mathbb C$.
\begin{proposition}
    Circles in $\mathbb C_\infty$ are mapped to circles in $\mathbb C_\infty$ under any Mobius transformations.
\end{proposition}
\begin{proof}
    It is sufficient to verify this for $z\mapsto az,z\mapsto z+b,z\mapsto z^{-1}$ due to Proposition \ref{decomp_mobius}.
    It is then trivial.
\end{proof}
Note that every circle gets to mapped to any other circle since three points determine the circle and Proposition \ref{mobius_3pts}
\begin{definition}
    For extended complex numbers $z_1,z_2,z_3,z_4$, the cross ratio is defined by
    $$[z_1,z_2,z_3,z_4]=\frac{(z_4-z_1)(z_2-z_3)}{(z_2-z_1)(z_4-z_3)}$$
\end{definition}
We need to examine carefully when one of these numbers is infinity.
For example, if we have $z_1=\infty$, then the cross ratio is $(z_2-z_3)/(z_4-z_3)$.
\begin{corollary}
    For extended complex numbers $z_1,z_2,z_3,z_4$, the cross ratio is equal to $f(z_4)$ where $f$ is the unique Mobius transformation sending $z_1,z_2,z_3$ to $0,1,\infty$ respectively.
\end{corollary}
\begin{theorem}
    Mobius transformations preserve the cross-ratio.
\end{theorem}
\begin{proof}
    Suppose $z_1,z_2,z_3,z_4\in\mathbb C_\infty$, and $g\in\mathcal M$.
    Let $f$ be the Mobius transformation sending $z_1,z_2,z_3$ to $0,1,\infty$, so the cross ratio is $f(z_4)$, so $f\circ g^{-1}$ sends $g(z_1),g(z_2),g(z_3)$ to $0,1,\infty$, so the cross ratio of the $g(z_i)$'s is $f\circ g^{-1}(g(z_4))=f(z_4)$.
\end{proof}
The converse is also true (and proved in example sheet): If a map preserves cross-ratio, then it is a Mobius transformation.
\begin{corollary}
    Four points $z_1,z_2,z_3,z_4\in\mathbb C_\infty$ are on a circle (in the sense of $\mathbb C_\infty$) if and only if $[z_1,z_2,z_3,z_4]$ is real.
\end{corollary}
\begin{proof}
    Let $f$ be the unique Mobius transformation sending $z_1,z_2,z_3$ to $0,1,\infty$, so $[z_1,z_2,z_3,z_4]=f(z_4)$.
    Let $c$ be the unique circle passing through $z_1,z_2,z_3$, then $z_4\in c\iff f(z_4)\in f(c)$, but $f(c)=\mathbb R\cup\{\infty\}$.
\end{proof}