\section{Miscellaneous}

\subsection{FFT Frequency}

It is surprisingly hard to remember the frequency samples that match the unshifted two-sided discrete Fast Fourier Transform (FFT) for odd or even length signals. Let $f_s$ be the sample frequency and $N$ be the number of points, then the discrete frequencies of the FFT are:

$N$ even:
\begin{equation}
f[k] = \dfrac{f_s}{N}\begin{cases}
    k,& k = 0,..., \dfrac{N}{2} - 1\\
    k-N, & k = \dfrac{N}{2}, ..., N-1
\end{cases}
\end{equation}

$N$ odd:
\begin{equation}
f[k] = \dfrac{f_s}{N}\begin{cases}
    k,& k = 0,..., \dfrac{N-1}{2} \\
    k-N, & k = \dfrac{N+1}{2}, ..., N-1
\end{cases}
\end{equation}

The routine \texttt{fftfreq} takes as input the sampling rate, $f_s$, and number of points, $N$, and returns the discrete frequencies $f[k]$ that correspond to the unshifted two-sided FFT of odd or even $N$.

{\footnotesize
\VerbatimInput{\code/Misc/fftfreq.m}
}

\subsection{Phase wrapping}

Phase wrapping is easily done with $\angle \exp(i\phi)$, but there is a fun variation that uses nearest integer rounding written in terms of elementary functions:
\begin{eqnarray}
\Phi(\phi) & = & \phi - 2\pi\left[\dfrac{\phi}{2\pi} \right] \\
\ & = & -2\arctan\cot\left(\dfrac{\phi}{2} + \dfrac{\pi}{2}\right)
\end{eqnarray}

\noindent where $[\cdot ]$ means round to the nearest integer so that $-\pi \le \Phi \le \pi$. %In principle, this can be differentiated away from the wrap points.

{\footnotesize
\VerbatimInput{\code/Misc/wrap.m}
}

%http://functions.wolfram.com/IntegerFunctions/Round/introductions/FloorRelated/ShowAll.html

%
%\subsection{Complex Standard Normal Distribution}
%
%The complex standard normal distribution is defined
%
%\begin{equation}
%\gamma = \dfrac{1}{\sqrt{2}}\left( \mathcal{N}(0,1) + i \mathcal{N}(0,1)\right)
%\end{equation}
%
%\noindent where $\mathcal{N}(0,1)$ is a normal distribution with zero mean and unit variance.  This is used to add noise to complex signals or to seed power spectral densities (PSD) in the frequency domain. For the later, we require $\gamma$ to have conjugate symmetry if the inverse Fourier transform is to yield a real valued signal. Because the Fourier transform of a Gaussian random variable is also Gaussian, a lazy way to ensure conjugate symmetry is to take the DFT of an array loaded with the real valued standard normal. However, a faster and more practical way to generate a signal from a PSD is simply to seed the PSD with the complex standard normal, without conjugate symmetry, and take the real part after the IFFT.
%
%The routines \texttt{gamman} and \texttt{gammanw} return the complex standard normal and frequency domain complex standard normal, respectively, for an arbitrary number of dimensions of sizes $N_1$, $N_2$, etc.  The frequency domain version is computed with the DFT, which requires division by the dimension lengths.  It can be verified that for both the real and imaginary parts of each functions that the mean and variance are $0$ and $1/\sqrt{2}$, respectively.  In \texttt{gammanw}, we set the DC component to zero to guarantee zero mean.
%
%{\footnotesize
%\VerbatimInput{\code/Misc/gamman.m}
%}
%
%{\footnotesize
%\VerbatimInput{\code/Misc/gammanw.m}
%}

\clearpage
\newpage
\subsection{Cubic Roots}

The roots of a cubic polynomial are known analytically. Let the cubic be written
\eq{a x^3 + b x^2 + c x + d = 0}

and define
\ea{w &=& -\dfrac{b}{3a} \\
Q &=& \dfrac{3ac - b^2}{9a^2} \\
R&=& \dfrac{9abc - 27a^2d - 2b^3}{54a^3} \\
D &=& Q^3 + R^2}

When $D\ge0$, there is one real root and two complex conjugate roots. For this case, define
\ea{S &=& \sqrt[3]{R + \sqrt{D}} \\
T &=&  \sqrt[3]{R - \sqrt{D}} }

When $D < 0$, there are three real roots and the following is used
\ea{S &=& \sqrt[3]{\rho}e^{i \theta/3} \\
T &=&  \sqrt[3]{\rho}e^{-i \theta/3} \\
\rho &=& \sqrt{-Q^3} \\
\theta &=& \arccos\left(\dfrac{R}{\rho}\right)}

In all instances, the positive roots of $\sqrt[3]{}$ and $\sqrt{}$ are used. The roots of the cubic are then given by
\ea{x_1 &=& w + S + T \\
x_2 &=& w -\dfrac{1}{2}(S +T) + i\dfrac{\sqrt{3}}{2}(S-T) \\
x_3 &=& w -\dfrac{1}{2}(S +T) - i\dfrac{\sqrt{3}}{2}(S-T) }

\noindent where $x_1$ is always real.

The routine \texttt{cubicroots} is a vectorized implementation of the solution above that computes the roots of multiple cubic polynomials simultaneously. Coefficient arrays can be any size. When \texttt{nargout == 1}, the routine returns the guaranteed real root $x_1$. The key difference between this routine and Matlab's \texttt{roots} is that \texttt{roots} is not vectorized and only solves one polynomial at a time, otherwise, the two routines produce the same solutions.

{\footnotesize
\VerbatimInput{\code/Misc/cubicroots.m}
}

\subsection{Quartic Roots}

Like the cubic, the roots of a quartic polynomial are known analytically, \cite{weisstein_quartic}.  Let the quartic be written
\eq{a x^4 + b x^3 + c x^2 + d x + e = 0}

This is put in standard form by dividing the leading coefficient
\eq{x^4 + a_3 x^3 + a_2 x^2 + a_1 x + a_0 = 0}

\noindent where $a_3 = b/a$, $a_2 = c/a$, $a_1 = d/a$ and $a_0 = e/a$.  This has resolvent cubic
\eq{y^3 - a_2y^2 + (a_1a_3-4a_0) y + (4 a_2 a_0 - a_1^2-a_3^2 a_0) = 0 \label{resolvecube}}

If $y_1$ is any real root of \eqref{resolvecube} (there is always at least one), then the roots of the quartic are given by
\ea{x_{1,2} &=& w + \dfrac{1}{2}R \pm \dfrac{1}{2}D \\
x_{3,4} &=& w - \dfrac{1}{2}R \pm \dfrac{1}{2}E }

\noindent where
\ea{ D &=& \begin{cases}  \sqrt{T - R^2 + \dfrac{U}{R}}, \quad R\ne 0 \\ \sqrt{T + V},\quad \quad \quad \quad R = 0 \end{cases} \\
E &=& \begin{cases}  \sqrt{T - R^2 - \dfrac{U}{R}}, \quad R\ne 0 \\ \sqrt{T - V}, \quad \quad \quad \quad R = 0 \end{cases} }

and
\ea{w &=& -\dfrac{1}{4} a_3 \\
R &=& \sqrt{\dfrac{a_3^2}{4} - a_2 + y_1} \\
T &=& \dfrac{3 a_3^2}{4} - 2 a_2\\
U &=& \dfrac{4a_3a_2-8a_1-a_3^3}{4} \\
V &=& 2 \sqrt{y_1^2 - 4 a_0} }

The routine \texttt{quarticroots} is a vectorized implementation of the solution above that computes the roots of multiple quartic polynomials simultaneously. Coefficient arrays can be any size. The real cubic root, $y_1$, comes from \texttt{cubicroots}. Again, Matlab's \texttt{roots} only solves one polynomial at a time, otherwise the two routines produce the same solutions.

{\footnotesize
\VerbatimInput{\code/Misc/quarticroots.m}
}

\clearpage
\newpage
\subsection{Uniform Points on a Sphere}

A quick approximation of uniform points on a sphere is based on the disco ball. Points are distributed mostly evenly at discrete latitudes centered on a prime meridian. This avoids crowding at the poles that happens with uniform sampling in spherical coordinates. Specifically, the number of lines of latitude is chosen which are divided equally between $\theta = [0, \pi]$, then each circle of constant latitude is divided by the latitude spacing as many times as it will go.

The routine \texttt{discoball} takes as input the number of latitude lines \texttt{nlat} and returns the spherical coordinates of the points $(\theta,\phi)$.

\begin{figure}[H]
\centering
\includegraphics[width=3.5in]{Utilities/Figures/discoball}
\caption{Disco ball approximation for uniform points on a sphere}
\end{figure}

{\footnotesize
\VerbatimInput{\code/Misc/discoball.m}
}

\clearpage
\newpage

\subsection{Uniformly Random Points on a Sphere}

To create a set of points that are uniformly random over the unit sphere, it is not correct to draw from a uniformly random distribution of spherical angles $(\theta,\phi)$, because this leads to crowding at the poles.  Instead, the following transformation is used, \cite{randsphere},
\begin{eqnarray}
\phi &=& 2\pi U(0,1) \\
\theta &=& \arccos(2V(0,1)-1)
\end{eqnarray}

\noindent where $U(0,1)$ and $V(0,1)$ are uniform random variables from $[0,1]$.

The routine \texttt{randsphere} takes as input the total number of points, $N$, and returns the spherical $(\theta,\phi)$ coordinates of the points.

\begin{figure}[H]
\centering
\includegraphics[width=3.5in]{Utilities/Figures/uniformrand}
\caption{Uniformly random points on a sphere, $N = 1000$. }
\end{figure}

{\footnotesize
\VerbatimInput{\code/Misc/randsphere.m}
}

%
%\subsection{$\log_b(x)$}
%
%Matlab doesn't have this.
%
%\begin{equation}
%\log_b(x) = \log(x)/\log(b)
%\end{equation}
%
%{\footnotesize
%\VerbatimInput{\code/Misc/logb.m}
%}
%
%\subsection{Triangle Function}
%
%The standard triangle function is given by
%
%\begin{align}
%\textrm{tri}(t) =
%& = \max(1 - |t|, 0) \\
%&=
%\begin{cases}
%1 - |t|, & |t| < 1 \\
%0, & \mbox{otherwise}
%\end{cases}
%\end{align}
%
%This can be generalized to a triangle with arbitrary width, height, horizontal and vertical offsets as
%
%\begin{equation}
%\textrm{tri}(t) = \max\left(b\left(1-\left\vert \dfrac{x-x_o}{a} \right\vert \right) + y_o, y_o\right)
%\end{equation}

%
%{\footnotesize
%\VerbatimInput{\code/Misc/triangle.m}
%}


\section{Task B}
\label{sec:task-b}

For a constant strain triangle (CST) element, the shape functions can be defined as
\begin{equation}
  \label{eq:shape-func}
  N_{1} = 1 - \xi - \eta, \quad N_{2} = \xi, \quad N_{3} = \eta
\end{equation}

According to equations \eqref{eq:material-shape-grad}, the \textit{material}
gradient of shape function reads
\begin{equation} \tag{9.6ab} \label{eq:material-shape-grad}
  \ubar{\nabla}_{0}N_{a} = \frac{\partial N_{a}}{\partial \ubar{\bm{X}}} =
  \left( \frac{\partial \ubar{\bm{X}}}{\partial \ubar{\bm{\xi}}} \right)^{-T}
  \frac{\partial N_{a}}{\partial \ubar{\bm{\xi}}} =
  \left( \frac{\partial \ubar{\bm{X}}}{\partial \ubar{\bm{\xi}}} \right)^{-T}
  \cdot \ubar{\nabla}_{\xi} N_{a}, \quad
  \frac{\partial \ubar{\bm{X}}}{\partial \ubar{\bm{\xi}}}
  = \sum_{a=1}^{n} \ubar{\bm{X}}_{a} \otimes \ubar{\nabla}_{\xi}N_{a}
\end{equation}
%
\begin{equation}
  \ubar{\nabla}_{\xi} N_{a} = \left[
    \begin{array}{c}
      \dfrac{\partial N_{a}}{\partial \xi} \\
      \dfrac{\partial N_{a}}{\partial \eta} \\
	\end{array} \right] =
  \left[
    \begin{array}{c c c}
      \dfrac{\partial N_{1}}{\partial \xi} & \dfrac{\partial N_{2}}{\partial \xi} & \dfrac{\partial N_{3}}{\partial \xi}\\
      \dfrac{\partial N_{1}}{\partial \eta} & \dfrac{\partial N_{2}}{\partial \eta} & \dfrac{\partial N_{3}}{\partial \eta}\\
	\end{array} \right]  =
  \left[
    \begin{array}{c c c}
      -1 & 1 & 0 \\
      -1 & 0 & 1 \\
	\end{array} \right]
\end{equation}
Similarly, the \textit{spatial} shape function gradients are given as
\begin{equation} \tag{9.11ab}
  \label{eq:spatial-shape-grad}
  \ubar{\nabla} N_{a} = \frac{\partial N_{a}}{\partial \ubar{\bm{x}}} =
  \left( \frac{\partial \ubar{\bm{x}}}{\partial \ubar{\bm{\xi}}} \right)^{-T}
  \cdot \ubar{\nabla}_{\xi} N_{a}, \quad
  \frac{\partial \ubar{\bm{x}}}{\partial \ubar{\bm{\xi}}}
  = \sum_{a=1}^{n} \ubar{\bm{x}}_{a} \otimes \ubar{\nabla}_{\xi}N_{a}
\end{equation}
Lastly, the deformation gradient can be expressed as
\begin{equation} \tag{9.5}
  \label{eq:deform-grad}
  \utilde{F} = \sum_{a=1}^{n} \ubar{\bm{x}}_{a} \otimes \ubar{\nabla}_{0} N_{a}
\end{equation}

The Matlab implementation of this task can be found in \texttt{shape\_gradients.m}
(see section \ref{app:matlab-code}).

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../main"
%%% End:


\subsection{SVD as a change of basis}

Next thing to remember, are the specially nice properties of orthogonal
matrices. By definition they are square matrices ($n \times n$), and
their columns form an orthonormal basis of \R{n}; this property
implies that they are invertible, but also, that the inverse is
specially easy to compute: it is just the transpose. In addition,
orthogonal matrices are an special case of change of basis matrices:
if $Q$ is an orthogonal matrix, then it can be seen as a function
which takes vectors in the coordinates of its column basis, and that
spits as result the coordinates in the canonical basis. On the same
line, $\inv{Q}$ does represent the opposite change of basis
(from the canonical coordinates to those in terms of the columns of
$Q$). \\

Having set the proper context, let us restate the SVD factorization as
a sequence of successive transformations:

\begin{enumerate}
\item Start with a vector $\vec{x} \in \R{n}$ in canonical coordinates.
\item Perform a change of basis using matrix \trans{V}, from the
  canonical coordinates to those in terms of the columns of $V$.
\item Once expressed as coordinates of columns of $V$, apply the
  linear transformation $\Sigma$; this not only converts the vector
  from \R{n} to \R{m}, but also expresses the coordinates in terms of
  the columns of $U$.
\item Once transformed, perform another change of basis using matrix
  $U$; from the coordinates of columns of $U$ to the canonical ones
  We end then with a vector in \R{m}.
\end{enumerate}
\hfill


\lab{Least Squares and Computing Eigenvalues}{Least Squares and Computing Eigenvalues}
\label{lab:qr-applications}
\objective{Because of its numerical stability and convenient structure, the QR decomposition is the basis of many important and practical algorithms.
In this lab, we introduce linear least squares problems, tools in Python for computing least squares solutions, and two fundamental algorithms for computing eigenvalue.
The QR decomposition makes solving several of these problems quick and numerically stable.
% \\ \indent As in the previous lab, we restrict ourselves to real matrices and therefore use the transpose in place of the Hermitian conjugate.
}

\section*{Least Squares} % ====================================================

A linear system $A\x = \b$ is \emph{overdetermined} if it has more equations than unknowns.
In this situation, there is no true solution, and $\x$ can only be approximated.

The \emph{least squares solution} of $A\x = \b$, denoted $\widehat{\x}$, is the ``closest'' vector to a solution, meaning it minimizes the quantity $\|A\widehat{\x} - \b\|_2$.
% \footnote{The choice of the 2-norm is significant.}
In other words, $\widehat{\x}$ is the vector such that $A\widehat{\x}$ is the projection of $\b$ onto the range of $A$, and can be calculated by solving the \emph{normal equations},%
\footnote{See Volume 1 for a formal derivation of the normal equations.}
\[A\trp A\widehat{\x} = A\trp \b.\]

If $A$ is full rank (which it usually is in applications) its QR decomposition provides an efficient way to solve the normal equations.
Let $A = \widehat{Q}\widehat{R}$ be the reduced QR decomposition of $A$, so $\widehat{Q}$ is $m \times n$ with orthonormal columns and $\widehat{R}$ is $n \times n$, invertible, and upper triangular.
Since $\widehat{Q}\trp \widehat{Q} = I$, and since $\widehat{R}\trp$ is invertible, the normal equations can be reduced as follows (we omit the hats on $\widehat{Q}$ and $\widehat{R}$ for clarity).
\begin{align}
\nonumber
A\trp A\widehat{\x} &= A\trp \b \\ \nonumber
(Q R)\trp Q R  \widehat{\x}
&= (Q R)\trp \b \\ \nonumber
 R\trp Q\trp Q R  \widehat{\x}
&=  R\trp Q\trp \b \\ \nonumber
 R\trp R \widehat{\x}
&=  R\trp Q\trp \b \\
 R \widehat{\x}
&= Q\trp \b \label{eq:normal-equation-via-qr}
\end{align}

Thus $\widehat{\x}$ is the least squares solution to $A\x=\b$ if and only if $\widehat{R}\widehat{\x} = \widehat{Q}\trp\b.$
Since $\widehat{R}$ is upper triangular, this equation can be solved quickly with back substitution.

\begin{problem} % Solve the normal equations with QR.
Write a function that accepts an $m \times n$ matrix $A$ of rank $n$ and a vector $\b$ of length $n$.
Use the QR decomposition and (\ref{eq:normal-equation-via-qr}) to solve the normal equations corresponding to $A\x = \b$.

You may use either SciPy's QR routine or one of your own QR routines.
In addition, you may use \li{la.solve_triangular()}, SciPy's optimized routine for solving triangular systems.
\label{prob:lstsq-via-qr}
\end{problem}

\subsection*{Fitting a Line} % ------------------------------------------------

The least squares solution can be used to find the best fit curve of a chosen type to a set of points.
Consider the problem of finding the line $y = ax + b$ that best fits a set of $m$ points $\{(x_k, y_k)\}_{k=1}^m$.
Ideally, we seek $a$ and $b$ such that $y_k = ax_k + b$ for all $k$.
These equations can be simultaneously represented by the linear system
\begin{equation}
A\x =
\left[\begin{array}{cc}
x_1 & 1 \\
x_2 & 1 \\
x_3 & 1 \\
\vdots & \vdots \\
x_m & 1
\end{array}\right]
\left[\begin{array}{c} a \\ b \end{array}\right]
=
\left[\begin{array}{c} y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_m \end{array}\right]
= \b.
\label{eq:linear-least-squares}
\end{equation}
%
Note that $A$ has full column rank as long as not all of the $x_k$ values are the same.

Because this system has two unknowns, it is guaranteed to have a solution if it has two or fewer equations.
However, if there are more than two data points, the system is overdetermined if any set of three points is not collinear.
We therefore seek a least squares solution, which in this case means finding the slope $\widehat{a}$ and $y$-intercept $\widehat{b}$ such that the line $y = \widehat{a}x+\widehat{b}$ best fits the data.

Figure \ref{fig:line-fit-example} is a typical example of this idea where $\widehat{a} \approx \frac{1}{2}$ and $\widehat{b} \approx -3$.

\begin{figure}[H] % Linear regression example (without code).
    \includegraphics[width=.7\textwidth]{figures/line_fit_example.pdf}
    \caption{A linear least squares fit.}
    \label{fig:line-fit-example}
\end{figure}

\begin{problem}
The file \texttt{housing.npy} contains the purchase-only housing price index, a measure of how housing prices are changing, for the United States from 2000 to 2010.%
\footnote{See \url{http://www.fhfa.gov/DataTools/Downloads/Pages/House-Price-Index.aspx}.}
Each row in the array is a separate measurement; the columns are the year and the price index, in that order.
To avoid large numerical computations, the year measurements start at 0 instead of 2000.

Find the least squares line that relates the year to the housing price index (i.e., let year be the $x$-axis and index the $y$-axis).

\begin{enumerate}
    \item Construct the matrix $A$ and the vector $\b$ described by (\ref{eq:linear-least-squares}).\\
    (Hint: \li{np.vstack()}, \li{np.column_stack()}, and/or \li{np.ones()} may be helpful.)
    \item Use your function from Problem \ref{prob:lstsq-via-qr} to find the least squares solution.
    \item Plot the data points as a scatter plot.
    \item Plot the least squares line with the scatter plot.\\
\end{enumerate}
\end{problem}

\begin{info} % scipy.stats.linregress().
The least squares problem of fitting a line to a set of points is often called \emph{linear regression}, and the resulting line is called the \emph{linear regression line}.
SciPy's specialized tool for linear regression is \li{scipy.stats.linregress()}.
This function takes in an array of $x$-coordinates and a corresponding array of $y$-coordinates, and returns the slope and intercept of the regression line, along with a few other statistical measurements.

For example, the following code produces Figure \ref{fig:line-fit-example}.

\begin{lstlisting}
>>> import numpy as np
>>> from scipy.stats import linregress

# Generate some random data close to the line y = .5x - 3.
>>> x = np.linspace(0, 10, 20)
>>> y = .5*x - 3 + np.random.randn(20)

# Use linregress() to calculate m and b, as well as the correlation
# coefficient, p-value, and standard error. See the documentation for
# details on each of these extra return values.
>>> a, b, rvalue, pvalue, stderr = linregress(x, y)

>>> plt.plot(x, y, 'k*', label="Data Points")
>>> plt.plot(x, a*x + b, label="Least Squares Fit")
>>> plt.legend(loc="upper left")
>>> plt.show()
\end{lstlisting}
\end{info}

\subsection*{Fitting a Polynomial} % ------------------------------------------

Least squares can also be used to fit a set of data to the best fit polynomial of a specified degree.
Let $\{(x_k, y_k)\}_{k=1}^m$ be the set of $m$ data points in question.
The general form for a polynomial of degree $n$ is
\[
p_n(x) = c_n x^n + c_{n-1} x^{n-1} + \cdots + c_2 x^2 + c_1 x + c_0 = \sum_{i=0}^n c_i x^i.
\]
Note that the polynomial is uniquely determined by its $n+1$ coefficients $\{c_i\}_{i=0}^n$.
Ideally, then, we seek the set of coefficients $\{c_i\}_{i=0}^n$ such that
\[
y_k = c_n x_k^n + c_{n-1} x_k^{n-1} + \cdots + c_2 x_k^2 + c_1 x_k + c_0
\]
for all values of $k$.
These $m$ linear equations yield the linear system
\begin{equation}
A\x =
\left[\begin{array}{cccccc}
x_1^n & x_1^{n-1} & \cdots & x_1^2 & x_1 & 1 \\
x_2^n & x_2^{n-1} & \cdots & x_2^2 & x_2 & 1 \\
x_3^n & x_3^{n-1} & \cdots & x_3^2 & x_3 & 1 \\
\vdots & \vdots & & \vdots & \vdots & \vdots \\
x_m^n & x_m^{n-1} & \cdots & x_m^2 & x_m & 1 \\
\end{array}\right]
\left[\begin{array}{c}
c_n \\ c_{n-1} \\ \vdots \\ c_2 \\ c_1 \\ c_0
\end{array}\right]
=
\left[\begin{array}{c} y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_m \end{array}\right]
= \b.
\label{eq:polynomial-least-squares}
\end{equation}
%
If $m > n+1$ this system is overdetermined, requiring a least squares solution.

\subsubsection*{Working with Polynomials in NumPy} % - - - - - - - - - - - - -

The $m \times (n+1)$ matrix $A$ of (\ref{eq:polynomial-least-squares}) is called a \emph{Vandermonde matrix}.%
\footnote{Vandermonde matrices have many special properties and are useful for many applications, including polynomial interpolation and discrete Fourier analysis.}
% a matrix with entries $a_{ij} = x_i^{n-j+1}$.
NumPy's \li{np.vander()} is a convenient tool for quickly constructing a Vandermonde matrix, given the values $\{x_k\}_{k=1}^m$ and the number of desired columns.

\begin{lstlisting}
>>> print(np.vander([2, 3, 5], 2))
[[2 1]                              # [[2**1, 2**0]
 [3 1]                              #  [3**1, 3**0]
 [5 1]]                             #  [5**1, 5**0]]

>>> print(np.vander([2, 3, 5, 4], 3))
[[ 4  2  1]                         # [[2**2, 2**1, 2**0]
 [ 9  3  1]                         #  [3**2, 3**1, 3**0]
 [25  5  1]                         #  [5**2, 5**1, 5**0]
 [16  4  1]]                        #  [4**2, 4**1, 4**0]
\end{lstlisting}

NumPy also has powerful tools for working efficiently with polynomials.
The class \li{np.poly1d} represents a 1-dimensional polynomial.
Instances of this class are callable like a function.%
\footnote{Class instances can be made callable by implementing the \lif{__call__()} magic method.}
The constructor accepts the polynomial's coefficients, from largest degree to smallest.

Table \ref{table:numpy-poly1d} lists some attributes and methods of the \li{np.poly1d} class.
% For more on NumPy's polynomial routines, see \url{http://docs.scipy.org/doc/numpy/reference/routines.polynomials.html}.

\begin{table}[H]
\begin{tabular}{r|l}
    Attribute & Description \\
    \hline
    \li{coeffs} & The $n+1$ coefficients, from greatest degree to least. \\
    \li{order} & The polynomial degree ($n$). \\
    \li{roots} & The $n-1$ roots. \\
    \\
    Method & Returns \\
    \hline
    \li{deriv()} & The coefficients of the polynomial after being differentiated. \\
    \li{integ()} & The coefficients of the polynomial after being integrated (with $c_0 = 0$).
\end{tabular}
\caption{Attributes and methods of the \li{np.poly1d} class.}
\label{table:numpy-poly1d}
\end{table}
%
\begin{lstlisting}
# Create a callable object for the polynomial f(x) = (x-1)(x-2) = x^2 - 3x + 2.
>>> f = np.poly1d([1, -3, 2])
>>> print(f)
   2
1 x - 3 x + 2

# Evaluate f(x) for several values of x in a single function call.
>>> f([1, 2, 3, 4])
array([0, 0, 2, 6])
\end{lstlisting}
% # Evaluate f(x) at 1, 2, 3, and 4 without creating f(x) explicitly.
% >>> np.polyval([1, -3, 2], [1, 2, 3, 4])
% array([0, 0, 2, 6])

\begin{problem} % Polynomial fitting.
The data in \texttt{housing.npy} is nonlinear, and might be better fit by a polynomial than a line.

Write a function that uses (\ref{eq:polynomial-least-squares}) to calculate the polynomials of degree $3$, $6$, $9$, and $12$ that best fit the data.
Plot the original data points and each least squares polynomial together in individual subplots.
\\(Hint: define a separate, refined domain with \li{np.linspace()} and use this domain to smoothly plot the polynomials.)

Instead of using Problem \ref{prob:lstsq-via-qr} to solve the normal equations, you may use SciPy's least squares routine, \li{scipy.linalg.lstsq()}.

\begin{lstlisting}
>>> from scipy import linalg as la

# Define A and b appropriately.

# Solve the normal equations using SciPy's least squares routine.
# The least squares solution is the first of four return values.
>>> x = la.lstsq(A, b)[0]
\end{lstlisting}

Compare your results to \li{np.polyfit()}.
This function receives an array of $x$ values, an array of $y$ values, and an integer for the polynomial degree, and returns the coefficients of the best fit polynomial of that degree.

\begin{comment}
\begin{lstlisting}
# Generate some random data close to the line y = x^2 - 3x + 2.
>>> x = np.linspace(0, 10, 20)
>>> y = x**2 - 3*x + 2 + np.random.randn(20)

# Use np.polyfit() to calculate the best fit 2nd degree polynomial.
>>> coeffs = np.polyfit(x, y, 2)

>>> domain = np.linspace(0, 10, 200)
>>> plt.plot(x, y, 'k*')
>>> plt.plot(domain, np.polyval(coeffs, domain))
>>> plt.show()
\end{lstlisting}
\end{comment}

\label{prob:polynomial-least-squares}
\end{problem}

\begin{warn} % Overfitting
Having more parameters in a least squares model is not always better.
For a set of $m$ points, the best fit polynomial of degree $m-1$ \emph{interpolates} the data set, meaning that $p(x_k) = y_k$ exactly for each $k$.
In this case there are enough unknowns that the system is no longer overdetermined.
However, such polynomials are highly subject to numerical errors and are unlikely to accurately represent true patterns in the data.

Choosing to have too many unknowns in a fitting problem is (fittingly) called \emph{overfitting}, and is an important issue to avoid in any statistical model.
\end{warn}

\subsection*{Fitting a Circle} % ----------------------------------------------

Suppose the set of $m$ points $\{(x_k, y_k)\}_{k=1}^m$ are arranged in a nearly circular pattern.
The general equation of a circle with radius $r$ and center $(c_1, c_2)$ is
\begin{equation}
(x-c_1)^2 + (y-c_2)^2 = r^2.
\label{eq:standard-circle}
\end{equation}

The circle is uniquely determined  by $r$, $c_1$, and $c_2$, so these are the parameters that should be solved for in a least squares formulation of the problem.
However, (\ref{eq:standard-circle}) is not linear in any of these variables.
%
\begin{align}
\nonumber (x - c_1)^2 + (y - c_2)^2 &= r^2 \\
\nonumber x^2 - 2c_1 x + c_1^2 + y^2 - 2c_2y + c_2^2 &= r^2 \\
x^2 + y^2 & = 2c_1 x + 2c_2 y + r^2 - c_1^2 - c_2^2
\label{eq:circle-expanded}
\end{align}

The quadratic terms $x^2$ and $y^2$ are acceptable because the points $\{(x_k, y_k)\}_{k=1}^m$ are given.
To eliminate the nonlinear terms in the unknown parameters $r$, $c_1$, and $c_2$, define a new variable $c_3 = r^2 - c_1^2 - c_2^2$.
Then for each point $(x_k, y_k)$, (\ref{eq:circle-expanded}) becomes
\[2c_1x_k + 2c_2y_k + c_3 = x_k^2 + y_k^2.\]
These $m$ equations are linear in $c_1$, $c_2$, and $c_3$, and can be written as the linear system
\begin{align}
\left[\begin{array}{ccc}
2 x_1 & 2 y_1 & 1 \\
2 x_2 & 2 y_2 & 1 \\
\vdots & \vdots & \vdots \\
2 x_m & 2 y_m & 1
\end{array}\right]
\left[\begin{array}{c} c_1 \\ c_2 \\ c_3 \end{array}\right]
=
\left[\begin{array}{c}
x_1^2 + y_1^2 \\
x_2^2 + y_2^2 \\
\vdots \\
x_m^2 + y_m^2
\end{array}\right].
\label{eq:circle-least-squares}
\end{align}

After solving for the least squares solution, $r$ can be recovered with the relation $r = \sqrt{c_1^2 + c_2^2 + c_3}$.
Finally, plotting a circle is best done with polar coordinates.
Using the same variables as before, the circle can be represented in polar coordinates by setting
\begin{equation}
x = r\cos(\theta) + c_1, \qquad
y = r\sin(\theta) + c_2, \qquad
\theta\in [0,2\pi].
\label{eq:circle-polar-rep}
\end{equation}
To plot the circle, solve the least squares system for $c_1$, $c_2$, and $r$, define an array for $\theta$, then use (\ref{eq:circle-polar-rep}) to calculate the coordinates of the points the circle.

\begin{lstlisting}
# Load some data and construct the matrix A and the vector b.
>>> xk, yk = np.load("circle.npy").T
>>> A = np.column_stack((2*xk, 2*yk, np.ones_like(xk)))
>>> b = xk**2 + yk**2

# Calculate the least squares solution and solve for the radius.
>>> c1, c2, c3 = la.lstsq(A, b)[0]
>>> r = np.sqrt(c1**2 + c2**2 + c3)

# Plot the circle using polar coordinates.
>>> theta = np.linspace(0, 2*np.pi, 200)
>>> x = r*np.cos(theta) + c1
>>> y = r*np.sin(theta) + c2
>>> plt.plot(x, y)                  # Plot the circle.
>>> plt.plot(xk, yk, 'k*')          # Plot the data points.
>>> plt.axis("equal")
\end{lstlisting}

\begin{figure}[H]
    \includegraphics[width=.7\textwidth]{figures/circle_fit_example.pdf}
\end{figure}

\begin{problem}
The general equation for an ellipse is \[ax^2 + bx + cxy + dy + ey^2 = 1.\]
Write a function that calculates the parameters for the ellipse that best fits the data in the file \texttt{ellipse.npy}.
Plot the original data points and the ellipse together, using the following function to plot the ellipse.

\begin{lstlisting}
def plot_ellipse(a, b, c, d, e):
    """Plot an ellipse of the form ax^2 + bx + cxy + dy + ey^2 = 1."""
    theta = np.linspace(0, 2*np.pi, 200)
    cos_t, sin_t = np.cos(theta), np.sin(theta)
    A = a*(cos_t**2) + c*cos_t*sin_t + e*(sin_t**2)
    B = b*cos_t + d*sin_t
    r = (-B + np.sqrt(B**2 + 4*A)) / (2*A)
    plt.plot(r*cos_t, r*sin_t, lw=2)
    plt.gca().set_aspect("equal", "datalim")
\end{lstlisting}
\end{problem}

\section*{Computing Eigenvalues} % ============================================

The eigenvalues of an $n \times n$ matrix $A$ are the roots of its characteristic polynomial $\det(A - \lambda I)$.
Thus, finding the eigenvalues of $A$ amounts to computing the roots of a polynomial of degree $n$.
However, for $n \ge 5$, it is provably impossible to find an algebraic closed-form solution to this problem.%
\footnote{This result, called \emph{Abel's impossibility theorem}, was first proven by Niels Heinrik Abel in 1824.}
In addition, numerically computing the roots of a polynomial is a famously ill-conditioned problem, meaning that small changes in the coefficients of the polynomial (brought about by small changes in the entries of $A$) may yield wildly different results.
Instead, eigenvalues must be computed with iterative methods.

\subsection*{The Power Method} % ----------------------------------------------

The \emph{dominant eigenvalue} of the $n \times n$ matrix $A$ is the unique eigenvalue of greatest magnitude, if such an eigenvalue exists.
The \emph{power method} iteratively computes the dominant eigenvalue of $A$ and its corresponding eigenvector.

Begin by choosing a vector $\x_0$ such that $\|\x_0\|=1$, and define
\[\x_{k+1}=\frac{A\x_k}{\|A\x_k\|}.\]
If $A$ has a dominant eigenvalue $\lambda$, and if the projection of $\x_0$ onto the subspace spanned by the eigenvectors corresponding to $\lambda$ is nonzero, then the sequence of vectors $(x_k)_{k=0}^\infty$ converges to an eigenvector $\x$ of $A$ corresponding to $\lambda$.

Since $\x$ is an eigenvector of $A$, $A\x = \lambda \x$.
Left multiplying by $\x\trp$ on each side gives $\x\trp A\x = \lambda \x\trp\x$, and hence $\lambda = \frac{\x\trp A\x}{\x\trp\x}$.
This ratio is called the \emph{Rayleigh quotient}.
However, since each $\x_k$ is normalized, $\x\trp\x = \|\x\|^2 = 1$, so $\lambda = \x\trp A\x$.

The entire algorithm is summarized below.

\begin{algorithm}[H] % The Power Method
\begin{algorithmic}[1]
\Procedure{Power Method}{$A$}
    \State $m, n \gets \shape{A}$
        \Comment{$A$ is square so $m = n$.}
    \State $\x_0 \gets \text{random}(n)$
        \Comment{A random vector of length $n$}
    \State $\x_0 \gets \x_0/\|\x_0\|$
        \Comment{Normalize $\x_0$}
    \For{$k = 1,\ 2,\ \ldots,\ N-1$}
        \label{step:power-method-stopping-criterion}
        \State $\x_{k+1} \gets A\x_k$
        \State $\x_{k+1} \gets \x_{k+1}/\|\x_{k+1}\|$
    \EndFor
    \State \pseudoli{return} $\x_N\trp A \x_N,\ \x_N$
\EndProcedure
\end{algorithmic}
\caption{}
\label{Alg:power-method}
\end{algorithm}

The power method is limited by a few assumptions.
First, not all square matrices $A$ have a dominant eigenvalue.
However, the Perron-Frobenius theorem guarantees that if all entries of $A$ are positive, then $A$ has a dominant eigenvalue.
Second, there is no way to choose an $\x_0$ that is guaranteed to have a nonzero projection onto the span of the eigenvectors corresponding to $\lambda$, though a random $\x_0$ will almost surely satisfy this condition.
Even with these assumptions, a rigorous proof that the power method converges is most convenient with tools from spectral calculus.
% See the Volume 1 text for details.

\begin{problem} % Implement the power method.
Write a function that accepts an $n \times n$ matrix $A$, a maximum number of iterations $N$, and a stopping tolerance \li{tol}.
Use Algorithm \ref{Alg:power-method} to compute the dominant eigenvalue of $A$ and a corresponding eigenvector.
Continue the loop in step \ref{step:power-method-stopping-criterion} until either $\|\x_{k+1} - \x_k\|$ is less than the tolerance \li{tol}, or until iterating the maximum number of times $N$.

Test your function on square matrices with all positive entries, verifying that $A\x = \lambda\x$.
Use SciPy's eigenvalue solver, \li{scipy.linalg.eig()}, to compute all of the eigenvalues and corresponding eigenvectors of $A$ and check that $\lambda$ is the dominant eigenvalue of $A$.

\begin{lstlisting}
# Construct a random matrix with positive entries.
>>> A = np.random.random((10,10))

# Compute the eigenvalues and eigenvectors of A via SciPy.
>>> eigs, vecs = la.eig(A)

# Get the dominant eigenvalue and eigenvector of A.
# The eigenvector of the kth eigenvalue is the kth column of 'vecs'.
>>> loc = np.argmax(eigs)
>>> lamb, x = eigs[loc], vecs[:,loc]

# Verify that Ax = lambda x.
>>> np.allclose(A @ x, lamb * x)
<<True>>
\end{lstlisting}
\end{problem}

\subsection*{The QR Algorithm} % ----------------------------------------------

An obvious shortcoming of the power method is that it only computes one eigenvalue and eigenvector.
The QR algorithm, on the other hand, attempts to find all eigenvalues of $A$.

Let $A_0 = A$, and for arbitrary $k$ let $Q_kR_k = A_k$ be the QR decomposition of $A_k$.
Since $A$ is square, so are $Q_k$ and $R_k$, so they can be recombined in reverse order:
\[A_{k+1}=R_kQ_k.\]
This recursive definition establishes an important relation between the $A_k$:
\[Q_k^{-1}A_kQ_k = Q_k^{-1}(Q_kR_k)Q_k = (Q_k^{-1}Q_k)(R_kQ_k) = A_{k+1}.\]
Thus, $A_k$ is orthonormally similar to $A_{k+1}$, and similar matrices have the same eigenvalues.
% Furthermore, since each of the $Q_k$ are orthonormal, the computation is numerically stable.
The series of matrices $(A_k)_{k=0}^\infty$ converges to the block matrix

\[
S =
\left[\begin{array}{cccc}
S_1    & *      & \cdots & *      \\
\0     & S_2    & \ddots & \vdots \\
\vdots & \ddots & \ddots & *      \\
\0     & \cdots &     \0 & S_m
\end{array}\right].
\qquad
\text{For example,}\ S =
\left[\begin{array}{ccccc}
\arrayrulecolor{lightgray}
s_1 & *       & *       & \cdots & *      \\
0   & s_{2,1} & s_{2,2} & \cdots & *      \\
    & s_{2,3} & s_{2,4} & \cdots & *      \\
    &         &         & \ddots & \vdots \\
    &         &         &        & s_m    \\
\end{array}\right].
\]
Each $S_i$ is either a $1\times1$ or $2\times2$ matrix.%
\footnote{If all of the $S_i$ are $1\times1$ matrices, then the upper triangular $S$ is called the \emph{Schur form} of $A$.
If some of the $S_i$ are $2\times2$ matrices, then $S$ is called the \emph{real Schur form} of $A$.}
In the example above on the right, since the first subdiagonal entry is zero, $S_1$ is the $1\times 1$ matrix with a single entry, $s_1$.
But as $s_{2,3}$ is not zero, $S_2$ is $2\times 2$.

Since $S$ is block upper triangular, its eigenvalues are the eigenvalues of its diagonal $S_i$ blocks.
Then because $A$ is similar to each $A_k$, those eigenvalues of $S$ are the eigenvalues of $A$.

% Second, each iteration of the algorithm transfers some of the ``mass'' from the lower to the upper triangle.
% This is what makes $A_0, A_1, A_2, \ldots$ converge to a matrix $S$ which has the described form.

When $A$ has real entries but complex eigenvalues, $2 \times 2$ $S_i$ blocks appear in $S$.
Finding eigenvalues of a $2 \times 2$ matrix is equivalent to finding the roots of a 2nd degree polynomial,
\begin{align}
% \nonumber S_i=\left[\begin{array}{cc}a&b\\c&d\end{array}\right] \\
\det(S_i - \lambda I) =
\left|\begin{array}{cc}
a - \lambda & b           \\
c           & d - \lambda
\end{array}\right|
= (a - \lambda)(d - \lambda) - bc
= \lambda^2 - (a+d)\lambda + (ad - bc),
\label{eq:qr-algorithm-roots}
\end{align}
which has a closed form solution via the quadratic equation.
This implies that complex eigenvalues come in conjugate pairs.

\subsubsection*{Hessenberg Preconditioning} % - - - - - - - - - - - - - - - - -

The QR algorithm works more accurately and efficiently on matrices that are in upper Hessenberg form, as upper Hessenberg matrices are already close to triangular.
Furthermore, if $H = QR$ is the QR decomposition of upper Hessenberg $H$ then $RQ$ is also upper Hessenberg, so the almost-triangular form is preserved at each iteration.
Putting a matrix in upper Hessenberg form before applying the QR algorithm is called \emph{Hessenberg preconditioning}.

% Second, an iteration of the QR algorithm can be computed in $\mathcal{O}(n^2)$ time on an upper Hessenberg matrix, as opposed to $\mathcal{O}(n^3)$ time on a regular matrix.
% This is because so many entries of an upper Hessenberg matrix are 0.
% If we apply the QR algorithm to an upper Hessenberg matrix $H$, then this speed-up happens in each iteration of the algorithm, since if $H = QR$ is the QR decomposition of $H$ then $RQ$ is also upper Hessenberg.

With preconditioning in mind, the entire QR algorithm is as follows.

\begin{algorithm}[H] % The Power Method
\begin{algorithmic}[1]
\Procedure{QR Algorithm}{$A$, $N$}
    \State $m, n \gets \shape{A}$
    \State $S \gets \text{hessenberg}(A)$ \label{step:qr-alg-hessenberg}
        \Comment{Put $A$ in upper Hessenberg form.}
    \For{$k = 0,\ 1,\ \ldots,\ N-1$} \label{step:qr-alg-niters}
        \State $Q, R \gets \text{qr}(S)$ \label{step:qr-alg-qr-S}
            \Comment{Get the QR decomposition of $A_k$.}
        \State $S \gets RQ$
            \Comment{Recombine $R_k$ and $Q_k$ into $A_{k+1}$.}
    \EndFor
    \State \texttt{eigs} $\gets$ \texttt{[]}
        \Comment{Initialize an empty list of eigenvalues.}
    \State $i \gets 0$
    \While{$i < n$}
        \If{$S_i$ is $1 \times 1$} \label{step:qr-alg-S_i-1x1-or-2x2}
            \State Append the only entry $s_i$ of $S_i$ to \texttt{eigs}
        \ElsIf{$S_i$ is $2 \times 2$}
            \State Calculate the eigenvalues of $S_i$
                \label{step:qr-alg-S_i-eigs}
            \State Append the eigenvalues of $S_i$ to \texttt{eigs}
            \State $i \gets i + 1$
        \EndIf
        \State $i \gets i + 1$
            \Comment{Move to the next $S_i$.}
    \EndWhile
    \State \pseudoli{return} \texttt{eigs}
\EndProcedure
\end{algorithmic}
\caption{}
\label{Alg:qr-algorithm}
\end{algorithm}

\begin{problem}
Write a function that accepts an $n \times n$ matrix $A$, a number of iterations $N$, and a tolerance \li{tol}.
Use Algorithm \ref{Alg:qr-algorithm} to implement the QR algorithm with Hessenberg preconditioning, returning the eigenvalues of $A$.

Consider the following implementation details.
\begin{itemize}
    \item Use \li{scipy.linalg.hessenberg()} or your own Hessenburg algorithm to reduce $A$ to upper Hessenberg form in step \ref{step:qr-alg-hessenberg}.
    \item The loop in step \ref{step:qr-alg-niters} should run for $N$ total iterations.
    \item Use \li{scipy.linalg.qr()} or one of your own QR factorization routines to compute the QR decomposition of $S$ in step \ref{step:qr-alg-qr-S}.
    Note that since $S$ is in upper Hessenberg form, Givens rotations are the most efficient way to produce $Q$ and $R$.
    \item Assume that $S_i$ is $1 \times 1$ in step \ref{step:qr-alg-S_i-1x1-or-2x2} if one of two following criteria hold:
    \begin{itemize}
        \item $S_i$ is the last diagonal entry of $S$.
        \item The absolute value of element below the $i$th main diagonal entry of $S$ (the lower left element of the $2\times 2$ block) is less than \li{tol}.
    \end{itemize}
    \item If $S_i$ is $2 \times 2$, use the quadratic formula and \eqref{eq:qr-algorithm-roots} to compute its eigenvalues.
    Use the function \li{cmath.sqrt()} to correctly compute the square root of a negative number.
\end{itemize}

Test your function on small random symmetric matrices, comparing your results to SciPy's \li{scipy.linalg.eig()}.
To construct a random symmetric matrix, note that $A + A\trp$ is always symmetric.
% How many iterations are necessary?
% How small does \li{tol} have to be?
% How large can $A$ be?
\end{problem}

\begin{info}
Algorithm \ref{Alg:qr-algorithm} is theoretically sound, but can still be greatly improved.
Most modern computer packages instead use the \emph{implicit QR algorithm}, an improved version of the QR algorithm, to compute eigenvalues.

For large matrices, there are other iterative methods besides the power method and the QR algorithm for efficiently computing eigenvalues.
They include the Arnoldi iteration, the Jacobi method, the Rayleigh quotient method, and others.
% We will return to this subject after studying spectral calculus and Krylov subspaces.
\end{info}

\begin{comment} % TODO: WLS, other norms?, QR alg improvements
\newpage

\section*{Additional Material} % ==============================================

\subsection*{Weighted Least Squares} % ----------------------------------------

A \emph{weighted 2-norm} $\|\cdot\|_W$ is defined by the nonsingular matrix $W$ as follows.
\[\|\x\|_W = \|W\x\|_2 = \sqrt{\x\trp W\trp W\x}\]
Typically $W$ is chosen to be a diagonal matrix, in which case the product $W\trp W$ is also diagonal.
Given the overdetermined system $A\x = \b$, the problem of choosing $\widehat\x$ to minimize $\|A\widehat\x - \b\|_W$ is called a \emph{weighted least squares} problem.
This problem has a slightly different set of normal equations.
\[A\trp W\trp W A \widehat{\x} = A\trp W\trp W \b\]

Letting $C = W A$ and $\z = W \b$, this equation reduces to the usual normal equations.
\[C\trp C \widehat{\x} = C\trp \z\]

Left multiplying $A$ by the diagonal matrix $W$ multiplies the rows of $A$ by the diagonal entries of $W$.
Thus the weighted least squares problem can be solved efficiently by representing $W$ as a 1-D array of its diagonal entries and using array broadcasting.

\begin{lstlisting}
>>> C = np.vstack(w) * A
>>> z = w * b
>>> x = la.lstsq(C, z)[0]
\end{lstlisting}

Weighted least squares is useful when some points in a data set are considered more important than others.
Consider the problem of fitting data to an exponential curve $y = ae^{kx}$ where points $(x_k,y_k)$ are known.
Taking the natural log of both sides gives $\log(y) = \log(a) + kx$, so the least squares problem is as follows.
\[
A\x =
\left[\begin{array}{cc}
x_1 & 1 \\
x_2 & 1 \\
\vdots & \vdots \\
x_m & 1
\end{array}\right]
\left[\begin{array}{c} k \\ \log(a) \end{array}\right]
=
\left[\begin{array}{c}
\log(y_1) \\ \log(y_2) \\ \vdots \\ \log(y_m)
\end{array}\right]
= \b
\]

% TODO: Finish this section with some exponential data and showing how WLS works better than OLS in this case.

\subsection*{Using other Norms} % ---------------------------------------------

In these Least Squares problems, we have found best fit lines and ellipses relative to the 2-norm.
It is possible to generalize the idea of best fit curves relative to other norms.
See Figure \ref{fig:lstsq-many-norms} for an illustration of this.

\begin{figure}[h]
    \centering
    \includegraphics[width=.7\textwidth]{figures/ellipsefit.pdf}
    % TODO: where did this picture come from? CVXOPT?
    \label{fig:lstsq-many-norms}
\end{figure}

\subsection*{Improvements to the QR Algorithm} % ------------------------------

\end{comment}


\subsection{\(L_p\) regularisation}

\subsubsection{Introduction}

We can generalise this to:

\(w_{l_p} = \arg \min ||y-Xw||^2_2+\lambda ||w||^p_p\)

For ridge regression there is always a solution.

For least squares there is a solution if \(X^TX\) is invertible

For Lasso we must use numerical optimisation.

lasso and \(L_1\) induces sparcity

Goal is \(min ||y - f(x)|| + \lambda g(w)\)

Ridge regression: \(g(w)=||w||\)

If \(\lambda =0\), OLS, if infinite, \(w\) goes to \(0\).

Normal equation changes to: \(\lambda I + X^TX)^{-1}X^Ty\)

We can preprocess to avoid processing of 1s. shift mean of \(y\) to \(0\). normalise \(x\) mean \(0\) var \(1\).


\section{Clustering}

\subsection{Clustering}
\subsection{A clustering Objective}
$G_{j} \subset \{ i| c_{i}=j \}$ where $G_{j}$ is set of all indices i for which $c_{i}=j$\\
\textbullet Group representatives: n-vectors $z_1,..,z_k$

\textbullet Clustering objective is\\
$J^{clust} = \frac{1}{N} \sum_{i=1}^N\Vert x_i - Z_{c_i}\Vert^2$\\
\textbullet mean square distance from vectors to associated representative\\
\textbullet goal: choose clustering $c_i$ and representatives $z_j$
to minimize $J_clust$
% Group representatives: We want each rep to be close to the vectors, i.e $||x_{i}-z_{c_{i}}||$ where $x_{i}$ is in group $j=c_{i}$ so $z_{c_{i}}$ is the representative vector associated with data vector $x_{i}$ \\
% $J^{clust}=(||x_{1}-z_{c_{1}}||^2 + ...+||x_{N}-z_{c_{N}}||^2)/N$\\
% Partitioning vectors with representatives fixed: $||x_{i}-z_{c_{i}}||=\min\limits_{j=1,....,k} ||x_{i}-z_{j}||$\\
% $J^{clust}=\Big(\min\limits_{j=1,....,k} ||x_{1}-z_{j}||^{2}+....+\min\limits_{j=1,....,k} ||x_{N}-z_{j}||^{2}\Big)/N$\\
% Optimizing group representatives with assignment fixed: $J^{clust}=J_{1}+...+J_{k}$ where $J_{j}=\big(1/N\big)\sum\limits_{i\in G_{j}}||x_{i}-z_{j}||^{2}$\\
% To minimize mean square distance $z_{j}=\big(1/|G_{j}|\big))\sum\limits_{i\in G_{j}}x_{i}$ where $|G_{j}|$ is standard notation for number of elements in set $G_{j}$
\subsection{The k-means algorithm}
\rule{\linewidth}{0.4pt}
\textbf{given} $x_1,..,x_N \in \textbf{R}^n$ and $z_1,...,z_k \in \textbf{R}^n$\\
\textbf{repeat}\\
-- \textit{Update partition}: assign i to $G_j$, $j = argmin_{j'} \Vert x_i - z_{j'} \Vert_2$\\
-- \textit{Update centroids}: $Z_j = \frac{1}{|G_j|}\sum_{i \in G_j}x_i$\\
\textbf{until} $z1,...,zk$ stop changing\\
\rule{\linewidth}{0.4pt}


\chapter{Parabolic PDEs}

Heat equation:

\begin{equation*}
\frac{\partial u}{\partial t} (x, t) = c \Delta u(x,t) + f(x, t), \quad (x, t) \in \Omega \times [0, T]
\end{equation*}

where $\Omega$ is an open subject of $\mathbb{R}^d \; (d = 1, 2, \text{ or } 3)$ (interprets a body), $[0, T]$, in a time interval.
$u$ represents temperature; $c$ depends on conductivity, specific heat and density of the body $\Omega$; $f$ represents sources or sinks of heat.

We have to add:
\begin{itemize}
	\item boundary conditions: $u(x, t)=0, \quad (x,t) \in \Gamma \times [0, T]$ where $T$ is the border of $\Omega$;
	\item initial condition: $u(x, 0) = u_0(x), \quad x \in \Omega$
\end{itemize}

The problem: given $c, f, u_0$, find $u$!

\section{Numerical solution of the heat equation}

Consider $\Omega = (0, 1)^2$, $h=\frac{1}{N}$, with $N$ positive integer.
Then we have $\Omega_h, \Gamma_h, \Delta_h$.

The discrete problem is:

\begin{equation*}
\frac{\partial}{\partial t} u_h((x, y), t) = \Delta_h u_h(x, y) + f(x, y),
\quad (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
u_h((x, y), t) = 0, \quad (x, y) \in \Gamma_h
\end{equation*}

\begin{equation*}
u_h((x,y), 0) = u_0(x, y), \quad (x, y) \in \Omega_h
\end{equation*}

This is a linear system of $(N-1)^2$ scalar linear differential equations in the $(N-1)^2$ unknown functions:

\begin{equation*}
u_h ((x, y), \cdot): [0, T] \rightarrow \mathbb{R}, \quad (x, y) \in \Omega_h
\end{equation*}

This is only a \textit{semi-discretization} because the time is not discretized (it's called \textit{method of lines} since the lines are the unknown functions).
For a full discretization we need a numerical method for the time-integration.

The simplest method to solve ODEs of type
\begin{equation*}
y'(t) = f(t, y(t)), \quad t \in [0, T]
\end{equation*}
is the Explicit Euler (also called Forward Euler) method, which scheme is $y_{m+1} = y_m + k f(t_m, y_m), \quad m=0, \dots, M-1$.

There's also the Implicit Euler (also called Backward Euler) method: $y_{m+1} = y_m + f(t_{m+1}, y_{m-})$.

And also the Trapezoidal Rule: $y_{m+1} = y_m + \frac{1}{2} (f(t_m, y_m) + f(t_{m+1}, y_{m+1})$.

We consider the \textit{1D} case:

\begin{equation*}
\frac{\partial}{\partial t} u(x, t) = \frac{\partial^2}{\partial x^2} u(x, t) + f(x, t), \quad (x, t) \in \overbrace{(0, 1)}^{= \Omega} \times [0, T]
\end{equation*}

\begin{equation*}
u(0, t) = u(1, t) = 0
\end{equation*}
\begin{equation*}
u(x, 0) = u_0(x), \quad x \in (0, 1)
\end{equation*}

with: $\Omega = (0, 1)$, $\Omega_h = \{nh: n \in \{1, \dots, N-1\} \}$ and $\Gamma_h = \{0, 1\}$.

We discretize in space by the centered difference scheme for the second derivative in space and in time by the forward difference scheme for the first derivative in time. We use $h = \frac{1}{N}$ as space stepsize and $k = \frac{T}{M}$ as time stepsize.
discrete point in space: $nh$ with $n \in \{1, \dots, N-1\}$
discrete point in time: $mk$ with $k \in \{1, \dots, u\}$

$U_n^m$ denotes two discrete approximation of $u(nh, mk)$.
At the point $(nh, mk)$, $n \in \{1, \dots, N-1\}$ and $m \in \{0, 1, \dots, M-1\}$ we have the discrete equation:

\begin{equation*}
\underbrace{\frac{U_n^{m+1} - U_n^{m}}{k}}_{\approx \frac{\partial u}{\partial t}(x, t) \text{ with } x=nh,\,t=mk } = c \underbrace{\frac{U_{n-1}^{m} - 2U_n^{m} + U_{n+1}^{m}}{h^2}}_{\approx \frac{\partial u}{\partial x^2}(x, t)} + \underbrace{f_n^m}_{= f(nk, mk)}
\end{equation*}

\begin{equation*}
U_n^{m+1} = \frac{c k}{h^2} U_{n-1}^m + (1 - \frac{2 c k}{h^2}) U_n^m + \frac{c k}{h^2} U_{n+1}^m + k f_n^m =
\end{equation*}

\begin{equation*}
= U_n^m + \frac{ck}{h^2}U_{n-1}^m - \frac{2 c k}{h^2} U_n^m + \frac{c k}{h^2}U_{n+1}^m
\end{equation*}

Consider:
\begin{equation*}
U^m = (U_1^m, \dots, U_{N-1}^m), f^m = (f_1^m, \dots, f_{N-1}^m)
\end{equation*}

\begin{equation*}
U^{m+1} = U^m + k(\Delta_h U^m + f^m)
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}

\begin{equation*}
\end{equation*}


\subsubsection{Derivatives}
The derivative of a function $y = f(x)$, notated $f'(x)$, gives the slope of the tangent line to $f$ at $x$.
\begin{definition}
	\begin{equation*}
		f'(x) = \lim\limits_{h \to 0}{\frac{f(x+h) - f(x)}{h}}
	\end{equation*}
\end{definition}

\noindent
Below are some properties of the derivative. Let $f$ and $g$ be functions of $x$ and $p$ a scalar.
\begin{enumerate}[label=]
	\item Linearity
	\begin{equation*}
		\left( pf \pm g \right)' = pf' \pm g'
	\end{equation*}
	\item Product Rule
	\begin{equation*}
		\left( fg \right)' = f'g + fg'
	\end{equation*}
	\item Quotient Rule
	\begin{equation*}
		\left( \frac{f}{g} \right)' = \frac{f'g - fg'}{g^2}
	\end{equation*}
	\item Chain Rule
	\begin{equation*}
		\left( f \circ g \right)' = \left( f' \circ g \right) \cdot g'
	\end{equation*}
	\item Power Rule
	\begin{equation*}
		\dd{x} p^x = px^{p-1} \text{, } p \neq 0
	\end{equation*}
	\item Exponent Rule
	\begin{equation*}
		\dd{x} p^x = p^x \ln{p} \text{, } p > 0
	\end{equation*}
	\item The Power Rule and Exponent Rule are two cases of the same rule
	\begin{equation*}
		\dd{x} f^g = gf^{g-1}f' + f^g\ln{f}g'
	\end{equation*}
\end{enumerate}
Using the definition of the derivative and these rules, we can find the derivatives to some common functions\\
\begin{center}
	\begin{tabular}{c | c}
		\large{$\dd{}{x} p = 0$} 				& \large{$\dd{}{x} e^x = e^x$} 			\\ [0.75em]
		\large{$\dd{}{x} \ln{x} = \frac{1}{x}$} & \large{$\dd{}{x} \sin{x} = \cos{x}$} 	\\ [0.75em]
		\large{$\dd{}{x} \cos{x} = -\sin{x}$} 	& \large{$\dd{}{x} \tan{x} = \sec^2{x}$}
	\end{tabular}
\end{center}


\section{Definition of the Derivative}
\begin{definition}
	The derivative of a function $f(x)$, notated $f^\prime(x)$, is the slope of $f$ at any point along $f$.
	This is also the slope of the tangent line to $f$ at this point, which is also the instantaneous rate of change of $f$ at this point.
	\begin{equation}
		f^\prime(x) = \lim_{h \to 0}{\frac{f(x+h)-f(x)}{h}} \text{ (assuming the limit exists).}
	\end{equation}
\end{definition}

\begin{example}
	Find the derivative of $f(x)=x^2+4$.
\end{example}
\begin{answer}
	Applying the definition,
	\begin{align*}
		f^\prime(x) &= \lim_{h\to 0}{\frac{f(x+h)-f(x)}{h}} \\
		&= \lim_{h \to 0}{\frac{(x+h)^2+4 - x^2 - 4}{h}} \\
		&= \lim_{h \to 0}{\frac{x^2 + 2xh + h^2 + 4 - x^2 - 4}{h}} \\
		&= \lim_{h \to 0}{\frac{2xh + h^2}{h}} \\
		&= \lim_{h\to 0}{2x + h} \\
		&= 2x.\footnotemark
	\end{align*}
\end{answer}
\footnotetext{Note that the constant term 4 didn't contribute anything to the outcome. It was canceled immediately when subtracting $f(x)$.}

There are a couple different notations that all mean the derivative of $y = f(x)$.
You should be familiar with all of them.
\begin{table}[H]
\begin{center}
\begin{tabular}{ l l l }
	$\begin{aligned}y^\prime\end{aligned}$ & & $\begin{aligned}\dd{y}{x}\end{aligned}$ \\
	& & \\
	$\begin{aligned}\dd{f}{x}\end{aligned}$ & & $\begin{aligned}\dd{}{x}f(x)\end{aligned}$
\end{tabular}
\end{center}
\end{table}

\subsection{Derivative at a Point}
The formula given in the definition is useful because it gives a function that can give the derivative at any point, but it may not be useful or feasible to use this formula.
Instead, we can use a formula to just give us the derivative at one point.
\begin{equation}
	f^\prime(a) = \lim_{x \to a}{\frac{f(x)-f(a)}{x-a}}.
\end{equation}

\begin{example}
	Find the derivative of $f(x) = \frac{1}{x}$ at $x=2$.
\end{example}
\begin{answer}
	Applying the formula for the derivative at a point,
	\begin{align*}
		f^\prime(2) &= \lim_{x \to 2}{\frac{f(x)-f(2)}{x-2}} \\
		&= \lim_{x \to 2}{\frac{1/x - 1/2}{x-2}} \\
		& = \lim_{x \to 2}{\frac{-(x-2)}{2x(x-2)}} \\
		&= \lim_{x \to 2}{\frac{-1}{2x}} \\
		&= \frac{-1}{4}.
	\end{align*}
\end{answer}

\subsection{Left \& Right Hand Derivatives}
The normal derivative is defined in terms of a two-sided limit, meaning that the left and right hand limits are equal.
However, we can also calculate left and right hand derivatives at every point along the function's domain, which may be useful if the left and right hand derivatives are not equal or the point in question is on the boundary of the domain.

\begin{example}
	Find the left and high hand derivatives of the following function at $x=1$.
	Say if the derivative at this point exists and why.
	\begin{equation*}
		f(x) = \begin{cases}
			x^2 + x & x \leq 1 \\
			x+1 & x > 1
		\end{cases}.
	\end{equation*}
\end{example}
\begin{answer}
	Evaluating the left hand limit,
	\begin{align*}
		f^\prime(1^-) &= \lim_{x \to 1^-}{\frac{f(x)-f(1)}{x-1}} \\
		&= \lim_{x \to 1^-}{\frac{x^2 + x - 2}{x-1}} \\
		&= \lim_{x \to 1^-}{\frac{(x-1)(x+2)}{x-1}} \\
		&= \lim_{x \to 1^-}{x+2} \\
		&= 3.
	\end{align*}

	Evaluating the right hand limit,
	\begin{align*}
		f^\prime(1^+) &= \lim_{x \to 1^+}{\frac{f(x)-f(1)}{x-1}} \\
		&= \lim_{x \to 1^+}{\frac{x + 1 - 2}{x-1}} \\
		&= \lim_{x \to 1^+}{\frac{x-1}{x-1}} \\
		&= 1.
	\end{align*}

	Since $f^\prime(1^-) \neq f^\prime(1^+)$, the derivative does not exist at $x=1$.
\end{answer}


\chapter{Math Symbols, Notations, and Identities}

\section{Symbols and Notations}
Here are the symbols and notations that you must absolutely know!

\subsection{Sets of Numbers}
\begin{symbollist}
    \item[$\mathbb{N}$] = Set of all Natural Numbers.
    \item[$\mathbb{Z}$] = Set of all Integers.
    \item[$\mathbb{Q}$] = Set of all Rational Numbers.
    \item[$\mathbb{R}$] = Set of all Real Numbers.
    \item[$\mathbb{C}$] = Set of all Complex Numbers.
\end{symbollist}

\subsection{Propositional Logic}
\begin{symbollist}
    \item[$\neg P$] = Negation of the proposition, \emph{P}.
    \item[$P \wedge Q$] = Conjunction of the propositions \emph{P and Q}.
    \item[$P \vee Q$] = Disjunction of the propositions \emph{P and Q}.
    \item[$P \oplus Q$] = P XOR Q (Exclusive Or).
    \item[$P \rightarrow Q$] = "If P, then Q" (Conditional).
    \item[$P \leftrightarrow Q$] = "P if and only if Q" (Biconditional).
    \item[$P \Leftrightarrow Q$] = P is equivalent to Q.
\end{symbollist}

% This would prevent LaTeX from "spreading out" the text to fill up the whole
% page, which I personally prefer, but isn't necessary

% \vfill

\subsection{Set Theory}
\begin{symbollist}
    \item[$\emptyset$] = Empty Set.
    \item[$ x \in \mathbb{Q}$] = x belongs to a rational number.
    \item[$A \subset B$] = Set A is a subset of set B.
    \item[$S = \{1,2,3,4\}$] =  An example of a set in roster form.
    \item[$S = \{x \mid 1 \leq x \leq 4, x \in \mathbb{Z}\}$] = An example of a
        set in set-builder form.
    \item[$A \cup B$] = A union B.
    \item[$A \cap B$] = A intersection B.
    \item[$A - B$] = A minus B (Set difference).
    \item[$\overline{A}$] = Complement of A.
    \item[$\mid A \mid$] = Number of elements in A (Cardinality).
\end{symbollist}

\subsection{Functions}
\begin{symbollist}
    \item[$\lceil x \rceil $] = Ceiling of x.
    \item[$\lfloor x \rfloor$] = Floor of x.
    \item[$O(f(x))$] = Big-O of the function \emph{f}.
    \item[$\Omega(f(x))$] = Big-Omega of the function \emph{f}.
    \item[$\Theta(f(x))$] = Big-Theta of the function \emph{f}.
\end{symbollist}

\subsection{Number Theory}
\begin{symbollist}
    \item[$a / b$] = a divides b.
    \item[$a \equiv b \lbrack m \rbrack$] = a is congruent to b modulo m.
\end{symbollist}

\subsection{Miscellaneous}
\begin{symbollist}
    \item[$\sum_{i = 1} ^ {n} i$] = Sum of first n terms.
    \item[$\prod_{i = 1} ^ {n} i$] = Product of first n terms.  $n!$ = \emph{n}
        factorial.
    \item[$\forall x \in \mathbb{Z}, P(x)$] = For all x in the set of integers,
        P(x) is true.
    \item[$\exists x \in \mathbb{Z}, P(x)$] = There exists an x such that P(x)
        is true.
\end{symbollist}

\pagebreak[4]

\section{Important Identities}
\indent \indent I have compiled a bunch of identities that are going to prove
to be very useful for proofs or problem solving. Again, I have broken them down
into chapters for better reference \footnote{I haven't included basic
conjunctions and disjunctions of propositions because you should be knowing
them by now!}.

\subsection{Simple Mathematical Identities}
\begin{symbollist}
    \item $(a + b)^{2} = a^{2} + 2ab + b^{2}$
    \item $(a - b)^{2} = a^{2} - 2ab + b^{2}$
    \item $a^{2} - b^{2} = (a + b)\times(a - b)$
    \item $a^{m} \times a^{n} = a^{m + n}$
    \item $(a^{m})^{n} = a^{mn}$
    \item $\log(a \times b) = \log(a) + \log(b)$
    \item $\log(a^{b}) = b \times \log(a)$
\end{symbollist}

\subsection{Propositional Logic}
\begin{symbollist}
    \item $\neg (\neg P) = P$
    \item $P \oplus Q = (P \vee Q) \wedge (\neg(P \wedge Q))$
    \item $\neg(P \wedge Q) = \neg P \vee \neg Q$
    \item $\neg(P \vee Q) = \neg P \wedge \neg Q$
    \item $P \vee P = P$
    \item $P \wedge P = P$
    \item $P \wedge Q = Q \wedge P$
    \item $P \vee Q = Q \vee P$
    \item $(P \wedge Q) \wedge R = P \wedge (Q \wedge R)$
    \item $(P \vee Q) \vee R = P \vee (Q \vee R)$
    \item $P \wedge (Q \vee R) = (P \vee Q) \wedge (P \vee R)$
    \item $P \vee (Q \wedge R) = (P \wedge R) \vee (P \wedge R)$
    \item $P \leftrightarrow Q \Leftrightarrow (P \rightarrow Q) \wedge (Q
    \rightarrow P)$
    \item $P \rightarrow Q = \neg P \vee Q$
\end{symbollist}

\subsection{Functions}
\begin{symbollist}
    \item $ x - 1 < \lfloor x \rfloor \leq x \leq \lceil x \rceil < x + 1$
    \item $ \lfloor - x \rfloor = - \lceil x \rceil$
    \item $ \lceil -x \rceil = - \lfloor x \rfloor$
    \item $ \lfloor x + n \rfloor = \lfloor x \rfloor + n$, for some \emph{n}
        $\in \mathbb{Z}$
    \item $ \lceil x + n \rceil = \lceil x \rceil + n$, for some \emph{n} $\in
        \mathbb{Z}$
    \item $(\exists n \in \mathbb{Z})(\exists \epsilon \in \mathbb{R}) \lfloor
        x \rfloor = n + \epsilon$, where $ 0 \leq \epsilon \leq 1$
    \item $f(x) = O(g(x)) \rightarrow (\exists k \in \mathbb{Z})(\exists c \in
        \mathbb{R^{+}})(\forall x > k)[f(x) \leq c \times g(x)]$
    \item $f(x) = \Omega(g(x)) \rightarrow (\exists k \in \mathbb{Z})(\exists c \in
        \mathbb{R^{+}})(\forall x > k)[f(x) \geq c \times g(x)]$
    \item $f(x) = \Theta(g(x)) \rightarrow [f(x) = O(g(x))] \wedge [f(x) =
        \Omega(g(x))]$
\end{symbollist}

\subsection{Number Theory}
\begin{symbollist}
    \item $a / b \rightarrow (\exists c \in \mathbb{Z})[a = b \times c]$
    \item $a / b \wedge a / c \rightarrow a / (b + c)$
    \item $a/ b \wedge b / c \rightarrow a / c $
    \item $\gcd(a, b) = am + bn$, for some $m, n \in \mathbb{Z}$
        (\textbf{Bezout's Identity})
    \item $\gcd(a, b) \times \mathrm{lcm}(a, b) = a \times b$
    \item $a \equiv b[m] \wedge c \equiv d[m] \rightarrow (a + b) \equiv (c +
        d)[m]$
    \item $a \equiv b[m] \wedge c \equiv d[m] \rightarrow (ab) \equiv
        (cd)[m]$
    \item If \emph{p} is a prime number, then $a^{p} \equiv a[p]$
        (\textbf{Fermat's Little Theorem})
    \item If \emph{p} is a prime number and $p / ab$, then $p / a \vee p / b$
        (\textbf{Euclid's Identity})
\end{symbollist}

\subsection{Counting}
\begin{symbollist}
    \item $\mid A \cup B \mid$ =  $\mid A \mid + \mid B \mid - \mid A \cap B
        \mid$
    \item $P(n,r) \frac{n!}{(n - r)!}$ (\textbf{Permutations})
    \item $C(n,r)$ or $\binom{n}{r}$ = $\frac{n!}{r!(n - r)!}$
        (\textbf{Combinations})
\end{symbollist}


\subsection{Jacobian matrix}

If we have \(n\) inputs and \(m\) functions such that:

\(f_i(\mathbf x)\)

The Jacobian is a matrix where:

\(J_{ij}= \dfrac{\delta f_i}{\delta x_j}\)


\chapter{Sequences and Series}
\section{Sequences and Series}
An infinite sequence is the sequence whose domain lines is defined on all
$x\in(0,\infty)$.  It is defined as follows:
\begin{equation}
a_1,a_2,a_3,...,a_n,...
\end{equation}

If the sequence were to only contain the first $n$ terms, it would be a
\textit{finite sequence}.

\section{Summation Notation}
To define the series that we expressed above, we can use a notation known as
"summation notation", which involves the use of the Greek letter
\textit{sigma}.  Let's define the sequence that we did earlier, this time using
summation notation:

\begin{equation}
\sum_{i=1}^na_i=a_1+a_2+a_3+a_4+...+a_n+...
\end{equation}

\subsection{Properties of Sums}
\begin{equation}
\sum_{i=1}^nc=cn
\end{equation}

\begin{equation}
\sum_{i=1}^nca_i=c\sum_{i=1}^{n}a_i
\end{equation}

\begin{equation}
\sum_{i=1}^{n}(a_i \pm b_i)=\sum_{i=1}^{n}a_i\pm\sum_{i=1}^{n}b_i
\end{equation}

\subsection{A Distinction}
There is an important distinction to be made: The first $n$ terms of any
sequence is referred to as the finite series, or a partial sum of teh entier
sequence, and is denoted by the following expression:
\begin{equation}
\sum_{i=1}^{n}a_i=a_1+a_2+a_3+a_4+...+a_n
\end{equation}

However, the infinite sries is all of the terms in an infinite sequence, and is
expressed by the following:
\begin{equation}
\sum_{i=1}^{\infty}a_i=a_1+a_2+a_3+a_4+...+a_n+...
\end{equation}

Consider the following two examples.  The first is a partial sum, and the second
is the sum of a series:
\begin{align*}
\sum_{i=1}^{3}\frac{3}{10^i}=\frac{3}{10^1}+\frac{3}{10^2}+\frac{3}{10^3}\\
=\frac{3}{10}+\frac{3}{100}+\frac{3}{1000}\\
=0.3+0.03+0.003\\
=0.333
\end{align*}

\begin{align*}
\sum_{i=1}^{\infty}\frac{3}{10^i}=\frac{3}{10^1}+\frac{3}{10^2}+\frac{3}{10^3}+...
\\
=\frac{3}{10}+\frac{3}{10}+\frac{3}{100}+...\\
=0.3+0.03+0.003+...\\
=0.333...
=\frac{1}{3}.
\end{align*}

\section{Arithmetic Sequences}
An arithmetic sequence is one where the differences between consecutive terms
are the same.  The sequence takes the following progression:
\begin{equation}
  a_1,a_1+d,a_1+2d,a_1+3d...
\end{equation}

\subsection{$n$-th term}
Therefore, the formula for the $n$th term is:
\begin{equation}
  a_1+(n-1)d
\end{equation}

\subsection{$n$-th partial sum}
To calculuate the $n$-th partial sum of the series, use the following formula:
\begin{equation}
  S_n=\frac{n}{2}(a_1+a_n)
\end{equation}

\section{Geometric Sequences}
A series is geometric when each consecutive is related by a ratio to the
previous term.  Consider the following example:
\begin{equation}
  a_1,a_1r,a_1r^2,...,a_1r^{n-1}
\end{equation}

\subsection{$n$-th term}
Therefore, the $n$-th term is defined as follows:
\begin{equation}
  a_n=a_1r^{n-1}
\end{equation}

\subsection{Summations of Finite Series}
The sum of a finite geometric series is given as follows:
\begin{equation}
  S_n=\sum_{i=1}^{n}a_1r^{i-1}=a_1\Big(\frac{1-r^n}{1-r}\Big)
\end{equation}

\subsection{Summations of Infinite Series}
The sum of an infinite geometric series is given as follows:
\begin{equation}
  S=\sum_{i=1}^{\infty}a_1r^i=\frac{a_1}{1-r}
\end{equation}


\section{Application: Dynamical systems}

The migration matrices discussed above give an example of a discrete
dynamical system. We call them discrete because they involve discrete
values taken at a sequence of points rather than on a continuous
interval of time.

An example of a situation which can be studied in this way is a
predator prey model. Consider the following model where $x$ is the
number of prey and $y$ the number of predators in a certain area at a
certain time. These are functions of $n\in \N$ where $n=1,2,\ldots$
are the ends of intervals of time which may be of interest in the
problem. In other words, $x (n)$ is the number of prey at the end of
the $n\th$ interval of time.  An example of this situation may be
modelled by the following equation
\begin{equation*}
  \begin{mymatrix}{c}
    x(n+1) \\
    y(n+1)
  \end{mymatrix} =\begin{mymatrix}{rr}
    2 & -3 \\
    1 & 4
  \end{mymatrix} \begin{mymatrix}{c}
    x(n) \\
    y(n)
  \end{mymatrix}.
\end{equation*}
This says that from time period $n$ to $n+1$, $x$ increases if there
are more $x$ and decreases as there are more $y$. In the context of
this example, this means that as the number of predators increases,
the number of prey decreases. As for $y$, it increases if there are
more $y$ and also if there are more $x$.

This is an example of a matrix recurrence which we define now.

\begin{definition}{Matrix recurrence}{matrix-recurrence}
  Suppose a dynamical system is given by
  \begin{eqnarray*}
    x_{n+1} &=& a x_n + b y_n \\
    y_{n+1} &=& c x_n + d y_n
  \end{eqnarray*}
  This system can be expressed as $V_{n+1} = A V_{n}$ where
  $V_{n} = \begin{mymatrix}{r}
    x_n \\
    y_n
  \end{mymatrix}$ and $A = \begin{mymatrix}{rr}
    a & b \\
    c & d
  \end{mymatrix}$.
\end{definition}

In this section, we will examine how to find solutions to a dynamical
system given certain initial conditions. This process involves several
concepts previously studied, including matrix diagonalization and
Markov matrices. The procedure is given as follows. Recall that when
diagonalized, we can write $A^{n} = PD^{n}P^{-1}$.

\begin{procedure}{Solving a dynamical system}{solving-dynamical}
  Suppose a dynamical system is given by
  \begin{eqnarray*}
    x_{n+1} &=& a x_n + b y_n \\
    y_{n+1} &=& c x_n + d y_n
  \end{eqnarray*}
  Given initial conditions $x_0$ and $y_0$, the solutions to the
  system are found as follows:
  \begin{enumerate}
  \item Express the dynamical system in the form $V_{n+1} = AV_n$.
  \item Diagonalize $A$ to be written as $A = PDP^{-1}$.
  \item Then $V_{n} = PD^{n} P^{-1} V_{0}$ where $V_{0}$ is the vector
    containing the initial conditions.
  \item If given specific values for $n$, substitute into this
    equation. Otherwise, find a general solution for $n$.
  \end{enumerate}
\end{procedure}

We will now consider an example in detail.

\begin{example}{Solutions of a discrete dynamical system}{solutions-dynamical-system}
  Suppose a dynamical system is given by
  \begin{eqnarray*}
    x_{n+1} &=& 1.5 x_n - 0.5y_n\\
    y_{n+1} &=& 1.0 x_n
  \end{eqnarray*}

  Express this system as a matrix recurrence and find solutions to the
  dynamical system for initial conditions $x_0=20, y_0=10$.
\end{example}

\begin{solution}
  First, we express the system as a matrix recurrence.
  \begin{eqnarray*}
    V_{n+1} &=& AV_{n}\\
    \begin{mymatrix}{c}
      x(n+1) \\
      y(n+1)
    \end{mymatrix} &=&\begin{mymatrix}{rr}
      1.5 & -0.5 \\
      1.0 & 0
    \end{mymatrix} \begin{mymatrix}{c}
      x(n) \\
      y(n)
    \end{mymatrix}.
  \end{eqnarray*}
  Then
  \begin{equation*}
    A
    =
    \begin{mymatrix}{rr}
      1.5 & -0.5 \\
      1.0 & 0
    \end{mymatrix}.
  \end{equation*}
  You can verify that the eigenvalues of $A$ are $1$ and $0.5$. By
  diagonalizing, we can write $A$ in the form
  \begin{equation*}
    P^{-1} D P =
    \begin{mymatrix}{rr}
      1 & 1 \\
      1 & 2
    \end{mymatrix} \begin{mymatrix}{rr}
      1 & 0 \\
      0 & 0.5
    \end{mymatrix} \begin{mymatrix}{rr}
      2 & -1 \\
      -1 & 1
    \end{mymatrix}.
  \end{equation*}
  Now given an initial condition
  \begin{equation*}
    V_0 = \begin{mymatrix}{c}
      x_{0} \\
      y_{0}
    \end{mymatrix},
  \end{equation*}
  the solution to the dynamical system is given by
  \begin{eqnarray*}
    V_n &=& P D^n P^{-1} V_0\\
    \begin{mymatrix}{c}
      x(n) \\
      y(n)
    \end{mymatrix} &=&\begin{mymatrix}{rr}
      1 & 1 \\
      1 & 2
    \end{mymatrix} \begin{mymatrix}{rr}
      1 & 0 \\
      0 & 0.5
    \end{mymatrix} ^{n}\begin{mymatrix}{rr}
      2 & -1 \\
      -1 & 1
    \end{mymatrix} \begin{mymatrix}{c}
      x_{0} \\
      y_{0}
    \end{mymatrix} \\
        &=& \begin{mymatrix}{rr}
          1 & 1 \\
          1 & 2
        \end{mymatrix} \begin{mymatrix}{rr}
          1 & 0 \\
          0 & (0.5) ^{n}
        \end{mymatrix} \begin{mymatrix}{rr}
          2 & -1 \\
          -1 & 1
        \end{mymatrix} \begin{mymatrix}{c}
          x_{0} \\
          y_{0}
        \end{mymatrix} \\
        &=&\begin{mymatrix}{c}
          y_{0}((0.5) ^{n}-1) -x_{0}((0.5)
          ^{n}-2) \\
          y_{0}(2(0.5) ^{n}-1) -x_{0}(2(0.5)
          ^{n}-2)
        \end{mymatrix}.
  \end{eqnarray*}
  If we let $n$ become arbitrarily large, this vector approaches
  \begin{equation*}
    \begin{mymatrix}{c}
      2x_{0}-y_{0} \\
      2x_{0}-y_{0}
    \end{mymatrix}.
  \end{equation*}
  Thus for large $n$,
  \begin{equation*}
    \begin{mymatrix}{c}
      x(n) \\
      y(n)
    \end{mymatrix} \approx \begin{mymatrix}{c}
      2x_{0}-y_{0} \\
      2x_{0}-y_{0}
    \end{mymatrix}.
  \end{equation*}
  Now suppose the initial condition is given by
  \begin{equation*}
    \begin{mymatrix}{c}
      x_{0} \\
      y_{0}
    \end{mymatrix}
    =
    \begin{mymatrix}{r}
      20 \\
      10
    \end{mymatrix}.
  \end{equation*}
  Then, we can find solutions for various values of $n$. Here are the
  solutions for values of $n$ between $1$ and $5$:
  \begin{equation*}
    n=1:
    \begin{mymatrix}{r}
      25.0 \\
      20.0
    \end{mymatrix},
    n=2: \begin{mymatrix}{r}
      27.5 \\
      25.0
    \end{mymatrix},
    n=3: \begin{mymatrix}{r}
      28.75 \\
      27.5
    \end{mymatrix}
  \end{equation*}
  \begin{equation*}
    n=4: \begin{mymatrix}{r}
      29.375 \\
      28.75
    \end{mymatrix},
    n=5: \begin{mymatrix}{r}
      29.688 \\
      29.375
    \end{mymatrix}.
  \end{equation*}
  Notice that as $n$ increases, we approach the vector given by
  \begin{equation*}
    \begin{mymatrix}{c}
      2x_{0}-y_{0} \\
      2x_{0}-y_{0}
    \end{mymatrix}
    =
    \begin{mymatrix}{r}
      2(20)- 10\\
      2(20)-10
    \end{mymatrix}
    =
    \begin{mymatrix}{r}
      30\\
      30
    \end{mymatrix}.
  \end{equation*}
  These solutions are graphed in the following figure.
  \begin{center}
    \begin{tikzpicture}[font=\small]
      \draw[<->](0,2) --(0,0)--(2,0);
      \draw (0.5, -0.05)--(0.5,0.05);
      \draw (1, -0.05)--(1,0.05);
      \draw (1.5, -0.05)--(1.5,0.05);
      \draw (-0.05, 0.5)--(0.05,0.5);
      \draw (-0.05, 1)--(0.05,1);
      \draw (-0.05, 1.5)--(0.05,1.5);
      \draw[fill] (0.8,0.75) circle [radius=0.025];
      \draw[fill] (1.2,1.3) circle [radius=0.025];
      \draw[fill] (1.45,1.6) circle [radius=0.025];
      \node[right] at (2,0){x};
      \node[right] at (0,2){y};
      \node[below] at (0.5,0){$28$};
      \node[below] at (1,0){$29$};
      \node[below] at (1.5,0){$30$};
      \node[left] at (0,0.5){$27$};
      \node[left] at (0,1){$28$};
      \node[left] at (0,1.5){$29$};
    \end{tikzpicture}
  \end{center}
\end{solution}

The following example demonstrates another system which exhibits some
interesting behavior. When we graph the solutions, it is possible for
the ordered pairs to spiral around the origin.

\begin{example}{Finding solutions to a dynamical system}{find-solutions-dynamical-system}
  Suppose a dynamical system is of the form
  \begin{equation*}
    \begin{mymatrix}{c}
      x(n+1) \\
      y(n+1)
    \end{mymatrix} =\begin{mymatrix}{rr}
      0.7 & 0.7 \\
      -0.7 & 0.7
    \end{mymatrix} \begin{mymatrix}{c}
      x(n) \\
      y(n)
    \end{mymatrix}.
  \end{equation*}
  Find solutions to the dynamical system for given initial conditions.
\end{example}

\begin{solution}
  Let
  \begin{equation*}
    A
    =
    \begin{mymatrix}{rr}
      0.7 & 0.7 \\
      -0.7 & 0.7
    \end{mymatrix}.
  \end{equation*}
  To find solutions, we must diagonalize $A$. You can verify that the
  eigenvalues of $A$ are complex and are given by
  $\eigenvar_1 = 0.7+0.7i$ and $\eigenvar_2 = 0.7-0.7i$. The eigenvector
  for $\eigenvar_1 = 0.7+0.7i$ is
  \begin{equation*}
    \begin{mymatrix}{r}
      1 \\
      i
    \end{mymatrix}
  \end{equation*}
  and that the eigenvector for $\eigenvar_2 = 0.7-0.7i$ is
  \begin{equation*}
    \begin{mymatrix}{r}
      1 \\
      -i
    \end{mymatrix}.
  \end{equation*}
  Thus the matrix $A$ can be written in the form
  \begin{equation*}
    \begin{mymatrix}{rr}
      1 & 1 \\
      i & -i
    \end{mymatrix} \begin{mymatrix}{cc}
      0.7+0.7i & 0 \\
      0 & 0.7-0.7i
    \end{mymatrix} \begin{mymatrix}{rr}
      \frac{1}{2} & -\frac{1}{2}i \\
      \frac{1}{2} & \frac{1}{2}i
    \end{mymatrix},
  \end{equation*}
  and so
  \begin{eqnarray*}
    V_n &=& PD^nP^{-1}V_0 \\
    \begin{mymatrix}{c}
      x(n) \\
      y(n)
    \end{mymatrix} &=&\begin{mymatrix}{rr}
      1 & 1 \\
      i & -i
    \end{mymatrix} \begin{mymatrix}{cc}
      (0.7+0.7i) ^{n} & 0 \\
      0 & (0.7-0.7i) ^{n}
    \end{mymatrix} \begin{mymatrix}{rr}
      \frac{1}{2} & -\frac{1}{2}i \\
      \frac{1}{2} & \frac{1}{2}i
    \end{mymatrix} \begin{mymatrix}{c}
      x_{0} \\
      y_{0}
    \end{mymatrix}.
  \end{eqnarray*}
  The explicit solution is given by
  \begin{equation*}
    \begin{mymatrix}{c}
      x_{0}(\frac{1}{2}(0.7-0.7i) ^{n}+\frac{1}{2}
      (0.7+0.7i) ^{n}) + \allowbreak y_{0}(\frac{1}{2}
      i(0.7-0.7i) ^{n}-\frac{1}{2}i (
      0.7+0.7i)  ^{n}) \\
      y_{0}(\frac{1}{2} (0.7-0.7i) ^{n}+\frac{1}{2}
      (0.7+0.7i) ^{n}) -  x_{0}(\frac{1}{2}
      i(0.7-0.7i) ^{n}-\frac{1}{2}i(
      0.7+0.7i) ^{n})
    \end{mymatrix}.
  \end{equation*}
  Suppose the initial condition is
  \begin{equation*}
    \begin{mymatrix}{c}
      x_{0} \\
      y_{0}
    \end{mymatrix} =\begin{mymatrix}{r}
      10 \\
      10
    \end{mymatrix}.
  \end{equation*}
  Then one obtains the following sequence of values which are graphed
  below by letting $n=1,2,\ldots,20$

  \begin{center}
    \includegraphics[bb=0 0 800 800,scale=.2]{figures/4dec.eps}
  \end{center}

  In this picture, the dots are the values and the dashed line is to
  help to picture what is happening.

  These points are getting gradually closer to the origin, but they
  are circling the origin in the clockwise direction as they do so. As
  $n$ increases, the vector $\begin{mymatrix}{c}
    x(n) \\
    y(n)
  \end{mymatrix}$ approaches $ \begin{mymatrix}{r}
    0 \\
    0
  \end{mymatrix}$.
\end{solution}


\lab{Python}{Symbolic Computation in Python}{SymPy}
\label{lab:SymPy}

\objective{Become familiar with some of the basic tools available in SymPy}

Python is good for more than just analysis of numerical data.
There are several packages available which allow symbolic computation in Python.
One such package is SymPy.
SymPy is designed to be a fully featured computer algebra system in Python.
It is actually used for much of the symbolic computation in Sage.
An example of what we mean by "symbolic computation" is the following:
\begin{lstlisting}
import sympy as sy
x = sy.symbols(`x')
sy.expand((x+1)**10)
\end{lstlisting}
which will return the following:
\begin{lstlisting}
x**10 + 10*x**9 + 45*x**8 + 120*x**7 + 210*x**6 + 252*x**5 + 210*x**4 + 120*x**3 + 45*x**2 + 10*x + 1
\end{lstlisting}
When used properly, this package can simplify large amounts of algebra for you.
This can be incredibly useful in a wide variety of situations.
As you may have guessed, such packages generally have a wide variety of features.
If you want to know how to do something, consider checking the documentation  at the SymPy website (\url{http://sympy.org/en/index.html}).
This lab should teach you to do some basic symbolic manipulations in SymPy, but keep in mind that this introduction is not comprehensive.

\section*{Basic Number Types}
SymPy has some good built in datatypes which can be used to represent rational numbers and arbitrary precision floating point numbers.
Arbitrary precision floating point operations are supported through the package \li{mpmath}.
These can be useful if you need to do computation to a very high precision, or to avoid possible overflow error in computation.
They are, however, much more costly to compute.

You can declare a rational number $\frac{a}{b}$ using \li{sy.Rational(a, b)}.
A real number $r$ of precision $n$ can be declared using $sy.Float(r,n)$.

A nice example of the use of these datatypes is the following function which computes $\pi$ to the $n$th digit.
\begin{lstlisting}
def mypi(n):
    #calculates pi to n decimal points.
    tot = sy.Rational(0, 1)
    term = 1
    bound = sy.Rational(1, 10)**(n+1)
    i = 0
    while bound <= term:
        term = 6 * sy.Rational(sy.factorial(2*i), 4**i*(sy.factorial(i))**2*(2*i+1)*2**(2*i+1))
        tot += term
        i += 1
    return sy.Float(tot, n)
\end{lstlisting}
This function works by evaluating the Taylor Series for $6\arcsin\left(\frac{1}{2}\right)$.
We used a rather crude error estimate to ensure that we were close enough to break the loop.

\begin{problem}
Write a function which evaluates $e$ to the $n$th digit.
Use the series
\begin{equation*}
e=\sum_{n=0}^{\infty} \frac{1}{n!}
\end{equation*}
Use the same condition as above to determine when to break your while loop.
\end{problem}

\section*{Symbolic Manipulation}
In SymPy, you need to declare symbolic objects before you use them.
To define a symbol $x$, we write \li{x = sy.symbols(`x')}.
This can also be used to define multiple variables at once, as in \li{x, y, z = sy.symbols(`x,y,z')}.
The string form of each variable on the right is used for showing expressions that involve the variable.
You will need to be careful about capitalization and that last `s' in the name of the function.
Calling \li{sy.Symbol()} will allow you to create a single symbolic variable, but \li{sy.symbol} is a submodule and cannot be called at all.

SymPy can be used to solve difficult expressions for given variables.
Not everything can be solved, but SymPy's algorithms are pretty good and can often save a great deal of time.
We consider the following equation:
\begin{equation*}
\frac{w}{w-x}+\frac{x}{x-y}+\frac{y}{y-z}+\frac{z}{z-w}=0
\end{equation*}
Say we need an explicit solution for a given variable.
Since the expression is symmetrical, it doesn't really matter which one, but we can solve this in the following way:
\begin{lstlisting}
import sympy as sy
w, x, y, z = sy.symbols(`w, x, y, z')
expr = w/(w-x) + x/(x-y) + y/(y-z) + z/(z-w)
sy.solve(expr, w)
\end{lstlisting}
In any variable, this expression is quadratic, but each coefficient will depend on the other 3 variables.
This would be a terrible pain to do by hand, but SymPy can take care of that for us.
It is worth noting that SymPy cannot do everything, but if we keep in mind what we are doing and are familiar with the tools it has, it can make complex algebraic operations a great deal faster.
We should also note that \li{solve()} returns a list of expressions.
If we want an expression to work with, we must take an item from the list.

In this particular example, you will notice that we only used a symbolic expression and not a full equation.
What SymPy did was set the expression equal to zero, solve for the variable we wanted, then return the result.
SymPy also supports equation objects (and inequalities), but this approach is often easier.
If you need to declare equations, they can be declared using something like \li{equation = Eq(x,  y)} which represents the equation $x=y$

\begin{problem}
Use SymPy to solve the equation $y=e^x+x$ for $x$.
The answer will be in terms of the Lambert W function, which is a special function available in most major symbolic math libraries.
It is included in both SciPy and SymPy and is defined as the inverse of $y=x e^x$.
\end{problem}

SymPy can also be used to expand and simplify different symbolic expressions.
As an example we will simplify the expression
\begin{equation*}
\frac{w x^2 y^2 - w x^2 - w y^2 + w - x^2 y^2 z + 2 x^2 y^2 + x^2 z - 2 x^2 + y^2 z - 2 y^2 - z + 2}{w x y - w x - w y + w - x y z + 2 x y + x z - 2 x + y z - 2 y - z + 2}
\end{equation*}
\begin{lstlisting}
w, x, y, z=sy.symbols(`w, x, y, z')
expr = (w*x**2*y**2 - w*x**2 - w*y**2 + w - x**2*y**2*z + 2*x**2*y**2 + x**2*z - 2*x**2 + y**2*z - 2*y**2 - z + 2)/(w*x*y - w*x - w*y + w - x*y*z + 2*x*y + x*z - 2*x + y*z - 2*y - z + 2)
expr.simplify()
\end{lstlisting}
When we evaluate this cell, we can see that the expression simplifies to $x y + x + y +1$.
\li{simplify()} is the general simplification method for a SymPy expression.
It can be called as a function from the module, i.e. as \li{sy.simplify()} or it can be used as a method for an object as in the example above.
You can also tell SymPy to do more specific types of simplification, for example if you want to factor an expression, you can use \li{factor()}.
If you want to tell it to expand an expression you can use \li{expand()}, if you want it to focus purely on simplifying the trigonometric aspects of the expression you can use \li{trigsimp()}.
If you want SymPy to cancel variable expressions in the numerator and denominator of all rational sub-expressions, use \li{cancel()}.
There are several other kinds of algebraic manipulations you can do in SymPy, see the documentation for a more comprehensive list.
Many of these more important functions in SymPy are also available as methods to expressions.
This is the case with all of the above examples.

Be sure to be careful when writing out symbolic expressions.
Python floating point numbers are still floating point numbers and Python integers are still integers.
A simple example is the expression \li{(3/4)*sy.sin(x)} which evaluates to 0, and should probably have been written as \li{3*sy.sin(x)/4} or \li{sy.Rational(3,4)*sy.sin(x)}.
Be careful about using floating points as well.
The expression \li{(3./4.)*sy.sin(x)} will evaluate to \li{.75*sy.sin(x)}, and the floating point calculations will be carry through to the rest of your symbolic work with that expression.

Another useful feature is substitution.
Substitution can be done using the \li{subs()} method of an expression.
%As of this writing, there is no subs function, though one has been discussed.
%I deliberately left the ambiguity here for compatibility with future releases.
You can substitute numbers and variables in for variables and even expressions.
For example, if you want to see what an expression looks like if $x$ is set to zero, you can use:
\begin{lstlisting}
expr.subs(x, 0)
\end{lstlisting}
where \li{expr} is the expression you have already defined.
Note that none of these operations modify the expression in place.
They return a modified version of the expression, but do not actually change the original.

Substitution also can be used (to some extent) to substitute one expression for another .
For example, if you want to apply the double angle identity to replace products of sines and cosines, you could use the following:
\begin{lstlisting}
expr.subs(sy.sin(x) * sy.cos(x), sy.sin(2*x)/2)
\end{lstlisting}
If you want to eliminate higher powers of a variable in an expression you can use something like:
\begin{lstlisting}
expr.subs(x**3, 0)
\end{lstlisting}
which will eliminate all terms of the expression involving $x^3$.
At present time this will not eliminate terms involving $x^4$ or higher powers of $x$ that are not divisible by 3.

\begin{problem}
Problems of the following form are useful in proving that finite difference schemes remain bounded as the number of iterations increases.
This property is called stability and will be discussed later in Volume 4.
Here we will just walk you through the symbolic manipulations necessary to show that a scheme is stable.
Consider the Crank-Nicolson finite difference scheme for the partial differential equation $u_t+a u_x=0$.
\begin{equation*}
\frac{v_{m}^{n+1}-v_{m}^{n}}{k}+a \frac{v_{m+1}^{n+1}-v_{m-1}^{n+1}+v_{m+1}^{n}-v_{m-1}^{n}}{4h}=0
\end{equation*}
Due to an important result by Von Neumann, we know that this scheme will be stable if and only if when we make the following substitutions, the absolute value of the amplification factor $g$ is always less than or equal to 1.
Prove that this scheme is stable in the following way:
\begin{itemize}
\item Substitute $g^a e^{i b \theta}$ for $v_a^b$.
Note that when you have solved the expression for $g$, $g$ will be a function of $\theta$.
You are free to assume that $\theta$, $h$, and $k$ are real.
You can tell SymPy that these variables are real by including the argument \li{real=True} in the \li{symbols} function when you declare them.
$g$ may or may not be real, depending on the problem.
\li{sy.I} is the imaginary unit $i$.
\item Use SymPy to cancel redundant terms, simplify the expression, and solve for $g$.
Since the whole expression is set to zero, factor it, look at the factors and cancel all you can.
You will want to apply Euler's formula before you solve for $g$.
In this case we only want to make this particular substitution, so do this using the \li{subs()} method.
\item Once you have an expression for $g$, we may consider $\abs{g}^2$, which can be found by multiplying $g$ by the complex conjugate of $g$.
You can take the conjugate of an expression using the \li{.conjugate()} method.
In this case, $\abs{g}$ should always be $1$, which shows that this scheme is stable for any $h$ and $k$.
\end{itemize}
\end{problem}

\section*{Calculus in SymPy}
SymPy can also be used to take limits, integrals, and derivatives.
Again, this can be very helpful when doing things that would be difficult to do by hand.
For example, the following equation takes the 20'th partial derivative with respect to $x$ of
\begin{equation*}
\prod_{i=1}^{23} \left(x+i y\right)
\end{equation*}
\begin{lstlisting}
x, y, i = sy.symbols(`x, y, i')
expr = sy.product((x+i*y), (i, 1, 23))
expr = expr.expand()
expr.diff(x, 20)
\end{lstlisting}

We can also integrate difficult things, for example, if we want to integrate $e^x\sin(x)\sinh(x)$, this can be done with one line in SymPy.
\begin{lstlisting}
sy.Integral(sy.sin(x) * sy.exp(x) * sy.sinh(x), x).doit()
\end{lstlisting}
Notice the \li{.doit()} method.
This tells SymPy to evaluate all derivatives, integrals, limits, etc. inside the expression.

Derivatives can be taken using the \li{sy.Derivative()} function, or the \li{.diff()} method of a SymPy expression.
As an example, we will take the 20'th derivative of the Lambert W function we saw earlier.
This can be done like this:
\begin{lstlisting}
from sympy.functions.elementary.exponential import LambertW
sy.Derivative(LambertW(x), x, 20).doit()
\end{lstlisting}
or, equivalently, like this:
\begin{lstlisting}
from sympy.functions.elementary.exponential import LambertW
LambertW(x).diff(x, 20).doit()
\end{lstlisting}
the second argument is the number of derivatives to take.
If it is omitted, one derivative will be taken.
You can also pass additional variables as arguments, for example, if we want to take $\frac{\partial}{\partial x} \left( \frac{\partial}{\partial y}\sin\left(x y\right)\right)$, we can do it in this way:
\begin{lstlisting}
expr = sy.sin(x*y)
expr.diff(x,y)
\end{lstlisting}
These tricks also work with integrals, with the exception of taking multiple integrals in the same variable.
You can integrate two different variables along two different bounds, as in:
\begin{lstlisting}
sy.integrate(y**2*x**2, (x, -1, 1), (y, -1, 1))
\end{lstlisting}
and you can integrate with respect to multiple variables, as in:
\begin{lstlisting}
sy.integrate(y**2 * x**2, x, y)
\end{lstlisting}
But, as of this writing, you can \emph{not} do something of the sort:
\begin{lstlisting}
sy.integrate(x**3, x, 20)
\end{lstlisting}
You would have to do this with some sort of for loop, or, simply by passing 20 $x$'s as arguments to the function.

SymPy also supports numerical integration and differentiation.
If you want to evaluate a definite integral, something like this will work.
\begin{lstlisting}
sy.Integral(sy.sin(x)*sy.exp(x)*sy.sinh(x), (x, -1, 1)).doit()
\end{lstlisting}
This sort of thing will still return a purely symbolic expression, but you can get an actual value for it by either converting it to a SymPy \li{Float} object, or by using the \li{sy.N()} function, which numerically evaluates expressions as best as it can.
SymPy can also evaluate integrals numerically without finding a symbolic solution. That can be done like this:
\begin{lstlisting}
sy.N(sy.Integral(sy.sin(x)*sy.exp(x)*sy.sinh(x), (x, -1, 1)))
\end{lstlisting}
If for some reason you wanted to evaluate the expression to more digits of precision, you can add the number of significant digits as an argument to \li{sy.N()}.

\begin{problem}
Use SymPy to symbolically evaluate
\begin{equation*}
\int_0^\infty \sin^2 \left(x\right) dx
\end{equation*}
In SymPy, positive infinity is represented by the object \li{sy.oo}
\end{problem}

\begin{problem}
Use SymPy to numerically estimate $\frac{\partial}{\partial x}e^{\sin\left(\cos\left(x\right)\right)}$ at $x=1$.
\end{problem}

You can also use SymPy to solve some sorts of basic ordinary differential equations.
This will solve the equation $y_{xx}-2*y_x+y=\sin\left(x\right)$
\begin{lstlisting}
x = sy.symbols(`x')
f = sy.Function(`f')
eq = f(x).diff(x, 2) - 2*f(x).diff(x) + f(x) - sy.sin(x)
sy.dsolve(eq)
\end{lstlisting}
or, equivalently,
\begin{lstlisting}
x = sy.symbols(`x')
f = sy.Function(`f')
eq = f(x).diff(x, 2) - 2*f(x).diff(x) + f(x) - sy.sin(x)
sy.dsolve(eq)
\end{lstlisting}
\begin{problem}
Use SymPy to solve the following differential equation:
\begin{equation*}
\begin{split}
 y_{xxxxxx} & + 3y_{xxxx} + 3y_{xx} + y = \\
& x^{10}e^x + x^{11}\sin\left(x\right) + x^{12}e^x\sin\left(x\right) -x^{13}\cos\left(2x\right) + x^{14}e^x\cos\left(3x\right)
\end{split}
\end{equation*}
You may recall from your last class on differential equations that this sort of problem is solved by the method of undetermined coefficients.
Imagine how terrible this would be to do by hand!
\end{problem}

In addition, SymPy includes several integral transforms, such as the Laplace, Fourier, Sine, and Cosine Transforms.
SymPy also allows you to do simple separation of variables on PDEs, Taylor Series, Laurent Series, Fourier Series, and many, \textit{many} other things.

\section*{Interfacing With Numerical Software}
SymPy also has a variety of built in ways to take an expression and turn it into a python function that can be evaluated quickly.
You can numerically evaluate expressions using \li{subs} to substitute in values like you would symbols, but this can be slow for large arrays.
A simple way to evaluate an expression more quickly is the \li{lambdify} function in \li{sympy.utilities.lambdify}.
It takes an expression and makes it into a callable python function.
It allows you to specify which library to use to compute the function, so if you would like to operate on arrays, you can use NumPy functions for $\sin$, $\cos$, etc.
It also allows you to tell it to use functions from the math library, SymPy itself, and mpmath.
We can take an expression in $x$ and $y$ and make a corresponding python function \li{fc} which uses NumPy as its backend in the following way:
\begin{lstlisting}
import sympy as sy
from sympy.utilities.lambdify import lambdify
import numpy as np
x, y = sy.symbols(`x, y')
expr = sy.sin(x) * sy.exp(x) - sy.cosh(x)
fc = lambdify([x, y], expr, `numpy')
\end{lstlisting}
This interoperability allows us to move quickly between symbolic calculations in SymPy and faster numerical evaluation on arrays in NumPy.

SymPy also has built in ways that allow you to make functions that iterate over arrays using F2PY, Cython, or Theano.
Using these other libraries can result in very fast numerical evaluation of symbolic expressions with relatively little effort.
These automatic wrapping features depend on other libraries and may require some setup.

\begin{problem}
Use the \li{lambdify} function to make a NumPy-dependent function for your solution to the previous problem. Set all the constants that would come from the initial conditions equal to 1.
The solution to the differential equation is an equality, but you can get the right hand side of the equality using \li{expr.rhs}.
Use the function you have just defined to plot the solution from -3 to 3.
It should be very large, and should look kind of like a sine curve.
\end{problem}

Sympy also includes a simplified plotting wrapper around Matplotlib.
A simple example is:
\begin{lstlisting}
x = sy.symbols(`x')
expr = sy.sin(x)*sy.exp(x)
sy.plot(expr, (x, -3, 3))
\end{lstlisting}
which will plots $\sin\left(x\right) e^x$ for values of $x$ from -3 to 3.

It is also worth noting that SymPy has several nice options for printing equations.
If you want to get a rough idea of what the equation looks like, you can use \li{sy.pprint()}.
It can interface with IPython to display the formula more clearly as well.
If you are using IPython, you can enable pretty printing by loading the extension that comes with SymPy.
In SymPy 7.2 this is done like this:
\begin{lstlisting}
%load_ext sympy.interactive.ipythonprinting
\end{lstlisting}
In SymPy 7.3 it is:
\begin{lstlisting}
import sympy as sy
sy.init_printing()
\end{lstlisting}
If you need to write the formula in \LaTeX, the function \li{sy.latex()} will convert it to \LaTeX{} for you.
SymPy has many more useful features.
When in doubt, take a look at the SymPy documentation.