\lab{Introduction to Matplotlib}{Matplotlib}
\label{lab:Matplotlib}
\objective{
Matplotlib is the most commonly-used data visualization library in Python. Being able to visualize data helps to determine patterns, to communicate results, and is a key component of applied and computational mathematics.
In this lab we introduce techniques for visualizing data in 1, 2, and 3 dimensions.
The plotting techniques presented here will be used in the remainder of the labs in the manual.
}

\section*{Line Plots} % =======================================================

The quickest way to visualize a simple 1-dimensional array is via a \emph{line plot}.
The following code creates an array of outputs of the function $f(x) = x^2$, then visualizes the array using the \li{matplotlib} module.\footnote{Like NumPy, Matplotlib is \emph{not} part of the Python standard library, but it is included in most Python distributions.}

\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt

>>> y = np.arange(-5,6)**2
>>> y
array([25, 16,  9,  4,  1,  0,  1,  4,  9, 16, 25])

# Visualize the plot.
>>> plt.plot(y)                     # Draw the line plot.
<<[<matplotlib.lines.Line2D object at 0x1084762d0>]>>
>>> plt.show()                      # Reveal the resulting plot.
\end{lstlisting}

The result is shown in Figure \ref{fig:basic1}.
Just as \li{np} is a standard alias for NumPy, \li{plt} is a standard alias for \li{matplotlib.pyplot} in the Python community.

The call \li{plt.plot(y)} creates a figure and draws straight lines connecting the entries of \li{y} relative to the $y$-axis.
The $x$-axis is by default the index of the array, namely the integers from $0$ to $10$.
Calling \li{plt.show()} then displays the figure.

\begin{figure}[H] % plt.plot(y) compared to plt.plot(x,y).
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.5\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/basic1.pdf}
    \caption{\li{plt.plot(y)} uses the indices of\\the array for the $x$-axis.}
    \label{fig:basic1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/basic2.pdf}
    \caption{\li{plt.plot(x,y)} specifies both the\\domain and the range.}
    \label{fig:basic2}
\end{subfigure}
\caption{Simple plots of $f(x) = x^2$ over the interval $x\in[-5,5]$.}
\end{figure}

\begin{problem} % Law of Large Numbers / NumPy review.
Write a function that accepts an integer $n$ as input.
\begin{enumerate}
\item Use \li{np.random.randn()} or \li{np.random.normal()} to create an $n\times n$ array of values randomly sampled from the standard normal distribution.
\item Compute the mean of each row of the array.
\\(Hint: use \li{np.mean()} and specify the \li{axis} keyword argument.)
\item Return the variance of these means.
\\(Hint: use \li{np.var()} to calcualte the variance).
\end{enumerate}
Define a new function that creates an array of the results of the first function with inputs $n = 100,\ 200,\ \ldots,\ 1000$.
Plot (and show) the resulting array.

This result illustrates one version of the \emph{Law of Large Numbers}.
\end{problem}

\subsection*{Specifying a Domain} % -----------------------------------------

An obvious problem with Figure \ref{fig:basic1} is that the $x$-axis does not correspond correctly to the $y$-axis for the function $f(x) = x^2$ that is being drawn.
To correct this, we need an array for the domain as well as one for the range.
First define an array \li{x} for the domain, then use that array to calculate the range \li{y} of $f$.
The command \li{plt.plot(x,y)} then plots \li{x} against \li{y}.
That is, each point \li{(x[i], y[i])} is drawn and consecutive points are connected.
% Thus both arrays must have the same number of elements to be compatible.

Another problem with Figure \ref{fig:basic1} is its poor resolution; the curve is visibly bumpy, especially near the bottom of the curve.
NumPy's \li{np.linspace()} function makes it easy to get a higher-resolution domain.
Recall that \li{range()} and \li{np.arange()} return a list or array of evenly-spaced values in a given interval, where the \emph{spacing} between the entries is specified.
In contrast, \li{np.linspace()} creates an array of evenly-spaced values in a given interval where the \emph{number of elements} is specified.

\begin{lstlisting}
# 4 evenly-spaced values between 0 and 32 (including endpoints).
>>> np.linspace(0, 32, 4)
array([  0.        ,  10.66666667,  21.33333333,  32.        ])

# Get 50 evenly-spaced values from -5 to 5 (including endpoints).
>>> x = np.linspace(-5, 5, 50)
>>> y = x**2                        # Calculate the range of f(x) = x**2.
>>> plt.plot(x, y)
>>> plt.show()
\end{lstlisting}

The resulting plot is shown in Figure \ref{fig:basic2}.
Note that this time, the $x$-axis is correctly aligned with the $y$-axis.
The resolution is also much better because \li{x} and \li{y} have $50$ entries each instead of only $10$.

All calls to \li{plt} functions modify the same figure until \li{plt.show()} is executed, which displays the current figure and resets the system.%
\footnote{Use \li{plt.figure()} to manually create several figures at once.}
The next time a \li{plt} function is called a new figure is created.
This makes it possible to plot several lines in a single figure.

\begin{problem} % Plot two lines (sin() and cos()).
Write a function that plots the functions $\sin(x)$, $\cos(x)$, and $\arctan(x)$ on the domain $[-2\pi, 2\pi]$ (use \li{np.pi} for $\pi$).
% Call \li{plt.xlim(-2*np.pi, 2*np.pi)} before \li{plt.show()} to stretch the $x$-axis appropriately.
Make sure the domain is refined enough to produce a figure with good resolution.
\end{problem}

\begin{info} % Interactive Mode.
Plotting can seem a little mystical because the actual plot doesn't appear until \li{plt.show()} is executed.
Matplotlib's \emph{interactive mode} allows the user to see the plot be constructed one piece at a time.
Use \li{plt.ion()} to turn interactive mode on and \li{plt.ioff()} to turn it off.
This is very useful for quick experimentation.

Try executing the following commands in IPython:

\begin{lstlisting}
In [1]: import numpy as np
In [2]: from matplotlib import pyplot as plt

# Turn interactive mode on and make some plots.
In [3]: plt.ion()
In [4]: x = np.linspace(1, 4, 100)
In [5]: plt.plot(x, np.log(x))
In [6]: plt.plot(x, np.exp(x))

# Clear the figure, then turn interactive mode off.
In [7]: plt.clf()
In [8]: plt.ioff()
\end{lstlisting}

Use interactive mode \textbf{only} with IPython.
Using interactive mode in a non-interactive setting may freeze the window or cause other problems.
\end{info}

\section*{Plot Customization} % ===============================================

\li{plt.plot()} receives several keyword arguments for customizing the drawing.
For example, the color and style of the line are specified by the following string arguments.
%
\begin{table}[H] % Color and style.
\begin{tabular}{r|l}
    Key & Color \\
    \hline
    \li{'b'} & blue\\
    \li{'g'} & green\\
    \li{'r'} & red\\
    \li{'c'} & cyan\\
    \li{'m'} & magenta\\
    % \li{'y'} & yellow\\
    \li{'k'} & black\\
    % \li{'w'} & white
\end{tabular}
\qquad
\begin{tabular}{r|l}
    Key & Style \\
    \hline
    \li{'-'} & solid line\\
    \li{'--'} & dashed line\\
    \li{'-.'} & dash-dot line\\
    \li{':'} & dotted line\\
    % \li{'.'} & point marker\\
    \li{'o'} & circle marker\\
    % \li{'*'} & star marker\\
    \li{'+'} & plus marker
\end{tabular}
\end{table}

Specify one or both of these string codes as an argument to \li{plt.plot()} to change from the default color and style.
Other \li{plt} functions further customize a figure.
%
\begin{table}[H]
\centering
\begin{tabular}{r|l}
    Function & \multicolumn{1}{|c}{Description}\\
    \hline
    % \li{grid()} & Add grid lines\\
    \li{legend()} & Place a legend in the plot\\
    % \li{text()} & Add text at a given position on the plot\\
    \li{title()} & Add a title to the plot\\
    \li{xlim()} & Set the limits of the $x$-axis\\
    \li{ylim()} & Set the limits of the $y$-axis\\
    % \li{xticks()} & set the location of the tick marks on the x axis, returns current locations if no arguments are given\\
    % \li{yticks()} & set the location of the tick marks on the y axis, returns current locations if no arguments are given\\
    \li{xlabel()} & Add a label to the $x$-axis\\
    \li{ylabel()} & Add a label to the $y$-axis
\end{tabular}
\end{table}

\begin{lstlisting}
>>> x1 = np.linspace(-2, 4, 100)
>>> plt.plot(x1, np.exp(x1), 'g:', linewidth=6, label="Exponential")
>>> plt.title("This is the title.", fontsize=18)
>>> plt.legend(loc="upper left")    # plt.legend() uses the 'label' argument of
>>> plt.show()                      # plt.plot() to create a legend.

>>> x2 = np.linspace(1, 4, 100)
>>> plt.plot(x2, np.log(x2), 'r+', markersize=4)
>>> plt.xlim(0, 5)                  # Set the visible limits of the x axis.
>>> plt.xlabel("The x axis")        # Give the x axis a label.
>>> plt.show()
\end{lstlisting}

\begin{figure}[H] % Figure customizations.
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/custom1.pdf}
    \label{fig:custom1}
\end{subfigure}
%
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/custom2.pdf}
    \label{fig:custom2}
\end{subfigure}
\label{fig:custom}
\end{figure}

See Appendix \ref{mpltables} for more comprehensive lists of colors, line styles, and figure customization routines.

\begin{problem} % Line plots with different domains but uniform style.
Write a function to plot the curve $f(x) = \frac{1}{x-1}$ on the domain $[-2,6]$.
Although $f(x)$ has a discontinuity at $x=1$, a single call to \li{plt.plot()} will attempt to make the curve look continuous.
\begin{enumerate}
\item Split up the domain and plot the two sides of the curve separately so that the graph looks discontinuous at $x=1$.
\item Plot both curves with a thick, dashed magenta line.\\
The keyword arguments \li{linewidth} or \li{lw} specify the line thickness.
\item Change the range of the $y$-axis to be $[-6, 6]$.
\end{enumerate}
The plot should resemble the figure below.

\begin{figure}[H] % Solution.
\includegraphics[width=.5\textwidth]{figures/discontinuousProblem.pdf}
\end{figure}
\end{problem}

\subsection*{Subplots} % ------------------------------------------------------

\emph{Subplots} are non-overlapping plots arranged in a grid within a single figure.
To create a figure with a grid of subplots, use \li{plt.subplot(numrows, numcols, fignum)}.
Here, \li{numrows} is the number of rows of subplots in the figure, \li{numcols} is the number of columns, and \li{fignum} specifies which subplot to modify.
% This index starts at 1 and increments across rows first.
See Figure \ref{fig:subplots-layout}.

\begin{figure}[H] % The layout created by subplots(23i).
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.15\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/layout_1.pdf}
\end{subfigure}
%
\begin{subfigure}{.15\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/layout_2.pdf}
\end{subfigure}
%
\begin{subfigure}{.15\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/layout_3.pdf}
\end{subfigure}
\\
\begin{subfigure}{.15\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/layout_4.pdf}
\end{subfigure}
%
\begin{subfigure}{.15\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/layout_5.pdf}
\end{subfigure}
%
\begin{subfigure}{.15\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/layout_6.pdf}
\end{subfigure}
\caption{The layout of subplots with \li{plt.subplot(2,3,i)} (2 rows, 3 columns), where \li{i} is the index pictured above.}
\label{fig:subplots-layout}
\end{figure}

If the inputs for \li{plt.subplot()} are all integers, the commas between the entries can be omitted.
For example, \li{plt.subplot(3,2,2)} and \li{plt.subplot(322)} are equivalent.

\begin{lstlisting}
>>> x = np.linspace(.1, 2, 200)

>>> plt.subplot(121)                # Start drawing the first subplot.
>>> plt.plot(x, np.exp(x), 'k', lw=2)
>>> plt.plot(x, np.exp(2*x), 'b', lw=2)
>>> plt.title("Exponential", fontsize=18)

>>> plt.subplot(122)                # Start drawing the second subplot.
>>> plt.plot(x, np.log(x), 'k', lw=2)
>>> plt.plot(x, np.log(2*x), 'b', lw=2)
>>> plt.title("Logarithmic", fontsize=18)

>>> plt.show()
\end{lstlisting}

\begin{figure}[H] % The layout created by plt.subplot(23i).
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/subplots_1.pdf}
\end{subfigure}
%
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/subplots_2.pdf}
\end{subfigure}
\end{figure}

\begin{problem} % Subplots of sine functions.
Write a function that plots the functions $\sin(x)$, $\sin(2x)$, $2\sin(x)$, and $2\sin(2x)$ on the domain $[0, 2\pi]$, each in a separate subplot.
\begin{enumerate}
    \item Arrange the plots in a square grid of 4 subplots.
    \item Set the limits of each subplot to $[0, 2\pi]\times[-2,2]$.
    \\(Hint: \li{plt.axis()} can do this in one line, instead of using both
    \\\li{plt.xlim()} and \li{plt.ylim()}.)
    \item Use \li{plt.title()} to give each subplot an appropriate title.
    \item Use \li{plt.suptitle()} to give the overall figure a title.
    \item Use the following colors and line styles.
    \begin{align*}\begin{array}{rcl}
    \sin(x)\text{: green solid line.} && \sin(2x)\text{: red dashed line.}\\ \\
    2\sin(x)\text{: blue dashed line.} && 2\sin(2x)\text{: magenta dotted line.}
    \end{array}\end{align*}
\end{enumerate}
\end{problem}

\section*{Other Kinds of Plots} % =============================================

Line plots are not always the most illuminating choice graph to describe a set of data.
Matplotlib provides several other easy ways to visualize data.

\begin{itemize}
\item A \emph{scatter plot} plots two 1-dimensional arrays against each other without drawing lines between the points.
Scatter plots are particularly useful for data that is not inherently correlated or ordered.

To create a scatter plot, use \li{plt.plot()} and specify a point marker (such as \li{'o'} or \li{'+'}) for the line style.%
\footnote{\li{plt.scatter()} can also be used to create scatter plots, but it accepts slightly different arguments than \li{plt.plot()}. We will explore the appropriate usage of this function in a later lab.}

\item A \emph{histogram} groups entries of a 1-dimensional data set into a given number of intervals, called \emph{bins}.
Each bin has a bar whose height indicates the number of values that fall in the range of the bin.
% The more bins, the greater the detail, but having too many bins can also destroy the picture by creating interval gaps between values.
Histograms are best for displaying distributions, relating data values to frequency.

To create a histogram, use \li{plt.hist()} instead of \li{plt.plot()}.
Use the argument \li{bins} to specify the edges of the bins, or to choose a number of bins.
The \li{<<range>>} argument specifies the outer limits of the first and last bins.
\end{itemize}

\begin{lstlisting}
# Get 500 random samples from two normal distributions.
>>> x = np.random.normal(scale=1.5, size=500)
>>> y = np.random.normal(scale=0.5, size=500)

# Draw a scatter plot of x against y, using a circle marker.
>>> plt.subplot(121)
>>> plt.plot(x, y, 'o', markersize=10)

# Draw a histogram to display the distribution of the data in x.
>>> plt.subplot(122)
>>> plt.hist(x, bins=np.arange(-4.5, 5.5))      # Or, equivalently,
#   plt.hist(x, bins=9, range=[-4.5, 4.5])

>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/scatterplot.pdf}
\end{subfigure}
%
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/histogram.pdf}
\end{subfigure}
\end{figure}

% On the histogram, specifying 9 bins in the range $[-4.5, 4.5]$ creates a bin centered over each integer from $-4$ to $4$.

\begin{problem} % FARS data visualization.
The Fatality Analysis Reporting System (FARS) is a nationwide census providing yearly data regarding fatal injuries suffered in motor vehicle traffic crashes.%
\footnote{See \url{http://www.nhtsa.gov/FARS}.}
The array contained in \texttt{FARS.npy} is a small subset of the FARS database from 2010--2014.
Each of the 148,206 rows in the array represents a different car crash; the columns represent the hour (in military time, as an integer), the longitude, and the latitude, in that order.

Write a function to visualize the data in \texttt{FARS.npy}.
Use \li{np.load()} to load the data, then create a single figure with two subplots:
%
\begin{enumerate}
\item A scatter plot of longitudes against latitudes.
Because of the large number of data points, use black pixel markers (use \li{"k,"} as the third argument to \li{plt.plot()}).
Label both axes.
\\
(Hint: Use \li{plt.axis("equal")} to fix the axis ratio on the scatter plot).

\item A histogram of the hours of the day, with one bin per hour.
Set the limits of the $x$-axis appropriately.
Label the $x$-axis.
\end{enumerate}
\end{problem}

% Other kinds of plots for 1-dimensional data includes bar plots, box plots, and others.
% See Appendix \ref{mpltables} for examples and syntax.

\subsection*{Visualizing 3-D Surfaces} % --------------------------------------

To plot a function $f: \mathbb{R}^2 \rightarrow \mathbb{R}$, we must choose and construct a 2-dimensional domain, then calculate the function at each point of that domain.
The standard tool for creating a 2-dimensional domain in the Cartesian plane is \li{np.meshgrid()}.
Given two 1-dimensional coordinate arrays, \li{np.meshgrid()} creates two corresponding coordinate matrices.
%
\begin{figure}[H] % np.meshgrid() visual demonstration.
\begin{tikzpicture}[>=stealth', shorten <= .1cm,shorten >=.1cm, dot/.style=
    {circle,fill=black,minimum size=3pt,inner sep=0pt, outer sep=-1pt} ]

\foreach \x/\y in {0/0, 0/2, 0/4, 2/0, 2/2, 2/4, 4/0, 4/2, 4/4}
    \node[draw, dot]at(\x,\y){};
\foreach \x/\y in {0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2}
    \node[draw=none]at(\x*2-.5, \y*2+.3){(\x,\y)};

\foreach \x/\y in {0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2}
    \node[draw=none]at(\x*.75+7, \y*.75+.1){\y};
\foreach \x/\y in {0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2}
    \node[draw=none]at(\x*.75+7, \y*-.75+3.9){\x};

\draw[-, thick](6.7,-.25)--(6.7,1.95);
\draw[-, thick](8.8,-.25)--(8.8,1.95);
\draw[-, thick](6.7,2.05)--(6.7,4.25);
\draw[-, thick](8.8,2.05)--(8.8,4.25);
\draw[-, thick](8.8,4.14)--(8.7,4.14);
\draw[-, thick](8.8,2.16)--(8.7,2.16);
\draw[-, thick](6.7,4.14)--(6.8,4.14);
\draw[-, thick](6.7,2.16)--(6.8,2.16);
\draw[-, thick](8.8,1.84)--(8.7,1.84);
\draw[-, thick](8.8,-.135)--(8.7,-.135);
\draw[-, thick](6.8,1.84)--(6.7,1.84);
\draw[-, thick](6.8,-.135)--(6.7,-.135);

\node[draw=none](X)at(6.3,.9){\texttt{Y}=};
\node[draw=none](y)at(6.3,3.15){\texttt{X}=};

\node[draw=none](point1)at(-.3, -.6){\texttt{x}=\big[0,};
\node[draw=none, node distance=2.35cm](point2)
    [right of=point1]{1,};
\node[draw=none, node distance=2cm](point3)
    [right of=point2]{2\big]};
\node[draw=none, rotate=270](point4)at(4.6,4.25)
    {\texttt{y}=\big[2,};
\node[draw=none, rotate=270, node distance=2.35cm](point5)
    [right of=point4]{1,};
\node[draw=none, rotate=270, node distance=2cm](point6)
    [right of=point5]{0\big]};
\end{tikzpicture}
\caption{\li{np.meshgrid(x, y)}, returns the arrays \li{X} and \li{Y}.
The returned arrays give the $x$- and $y$-coordinates of the points in the grid formed by \li{x} and \li{y}.
Specifically, the arrays \li{X} and \li{Y} satisfy \li{(X[i,j], Y[i,j]) = (x[i],y[j])}.}
\label{fig:meshgrid}
\end{figure}

With a 2-dimensional domain, we usually visualize $f$ with two kinds of plots.

\begin{itemize}
\item A \emph{heat map} assigns a color to each entry in the matrix, producing a 2-dimensional picture describing a 3-dimensional shape.
Darker colors typically correspond to lower values while lighter colors typically correspond to higher values.

Use \li{plt.pcolormesh()} to create a heat map.

\item A \emph{contour map} draws several \emph{level curves} of $f$.
A level curve corresponding to the constant $c$ is the collection of points $\left\{(x,y)\mid c = f(x,y)\right\}$.
Coloring the space between the level curves produces a discretized version of a heat map.
Including more and more level curves makes a filled contour plot look more and more like the complete, blended heat map.

Use \li{plt.contour()} to create a contour plot and \li{plt.contourf()} to create a filled contour plot.
Specify either the number of level curves to draw, or a list of constants corresponding to specific level curves.
\end{itemize}

These three functions all receive the keyword argument \li{cmap} to specify a color scheme (some of the better schemes are \li{"viridis"}, \li{"magma"}, and \li{"Spectral"}).
See \url{http://matplotlib.org/examples/color/colormaps_reference.html} for the list of all Matplotlib color schemes.

Finally, to see how the colors in these plots relate to the values of the function, use \li{plt.colorbar()} to draw the color scale beside the plot.

\begin{lstlisting}
# Create a 2-D domain with np.meshgrid().
>>> x = np.linspace(-np.pi, np.pi, 100)
>>> y = x.copy()
>>> X, Y = np.meshgrid(x, y)

# Calculate z = f(x,y) = sin(x)sin(y) using the meshgrid coordinates.
>>> Z = np.sin(X) * np.sin(Y)

# Plot the heat map of f over the 2-D domain.
>>> plt.subplot(131)
>>> plt.pcolormesh(X, Y, Z, cmap="viridis")
>>> plt.colorbar()
>>> plt.xlim(-np.pi, np.pi)
>>> plt.ylim(-np.pi, np.pi)

# Plot a contour map of f with 10 level curves.
>>> plt.subplot(132)
>>> plt.contour(X, Y, Z, 10, cmap="Spectral")
>>> plt.colorbar()

# Plot a filled contour map, specifying the level curves.
>>> plt.subplot(133)
>>> plt.contourf(X, Y, Z, [-1, -.8, -.5, 0, .5, .8, 1], cmap="magma")
>>> plt.colorbar()

>>> plt.show()
\end{lstlisting}

\begin{figure}[H] % heat map and contour plots.
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.33\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/heatmap.png}
\end{subfigure}%
\begin{subfigure}{.33\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/contour.pdf}
\end{subfigure}%
\begin{subfigure}{.33\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/contourf.pdf}
\end{subfigure}
\end{figure}

\begin{problem} % Heat map / contour plot of a function f:R2->R.
\label{prob:heatmap}
Write a function to plot $f(x,y) = \frac{\sin(x)\sin(y)}{xy}$ on the domain $[-2\pi,2\pi] \times [-2\pi,2\pi]$.

\begin{enumerate}
\item Create 2 subplots: one with a heat map of $f$, and one with a contour map of $f$.
Choose an appropriate number of level curves, or specify the curves yourself.
\item Set the limits of each subplot to $[-2\pi,2\pi] \times [-2\pi,2\pi]$.
\item Choose a non-default color scheme.
\item Include the color scale bar for each subplot.
\end{enumerate}
\end{problem}

\begin{info} % Note about plt.imshow().
Images are usually stored as either a 2-dimensional array (for black-and-white pictures) or a 3-dimensional array (a stack of 2-dimensional arrays, one for each RGB value).
This kind of data does not require a domain, and is easily visualized with \li{plt.imshow()}.
\end{info}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{3-D Plotting} % --------------------------------------------------

Matplotlib can also be used to plot 3-dimensional surfaces.
The following code produces the surface corresponding to $f(x,y) = \sin(x)\sin(y)$.

\begin{lstlisting}
# Create the domain and calculate the range like usual.
>>> x = np.linspace(-np.pi, np.pi, 200)
>>> y = np.copy(x)
>>> X, Y = np.meshgrid(x, y)
>>> Z = np.sin(X)*np.sin(Y)

# Draw the corresponding 3-D plot using some extra tools.
>>> from mpl_toolkits.mplot3d import Axes3D
>>> fig = plt.figure()
>>> ax = fig.gca(projection='3d')
>>> ax.plot_surface(X, Y, Z)
>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
    \includegraphics[width=.7\textwidth]{figures/surface_plot.pdf}
\end{figure}

\subsection*{Animations} % ----------------------------------------------------

Lines and other graphs can be altered dynamically to produce animations.
Follow these steps to create a Matplotlib animation:
%
\begin{enumerate}
    \item Calculate all data that is needed for the animation.
    \item Define a figure explicitly with \li{plt.figure()} and set its window boundaries.
    \item Draw empty objects that can be altered dynamically.
    \item Define a function to update the drawing objects.
    \item Use \li{matplotlib.animation.FuncAnimation()}.
\end{enumerate}

The submodule \li{matplotlib.animation} contains the tools putting together and managing animations.
The function \li{matplotlib.animation.FuncAnimation()} accepts the figure to animate, the function that updates the figure, the number of frames to show before repeating, and how fast to run the animation (lower numbers mean faster animations).

\begin{lstlisting}
from matplotlib.animation import FuncAnimation

def sine_animation():
    # Calculate the data to be animated.
    x = np.linspace(0, 2*np.pi, 200)[:-1]
    y = np.sin(x)

    # Create a figure and set its window boundaries.
    fig = plt.figure()
    plt.xlim(0, 2*np.pi)
    plt.ylim(-1.2, 1.2)

    # Draw an empty line. The comma after 'drawing' is crucial.
    drawing, = plt.plot([],[])

    # Define a function that updates the line data.
    def update(index):
        drawing.set_data(x[:index], y[:index])
        return drawing,                     # Note the comma!

    a = FuncAnimation(fig, update, frames=len(x), interval=10)
    plt.show()
\end{lstlisting}

Try using the following function in place of \li{update()}.
Can you explain why this animation is different from the original?

\begin{lstlisting}
def wave(index):
    drawing.set_data(x, np.roll(y, index))
    return drawing,
\end{lstlisting}

To animate multiple objects at once, define the objects separately and make sure the update function returns both objects.

\begin{lstlisting}
def sine_cosine_animation():
    x = np.linspace(0, 2*np.pi, 200)[:-1]
    y1, y2 = np.sin(x), np.cos(x)

    fig = plt.figure()
    plt.xlim(0, 2*np.pi)
    plt.ylim(-1.2, 1.2)

    sin_drawing, = plt.plot([],[])
    cos_drawing, = plt.plot([],[])

    def update(index):
        sin_drawing.set_data(x[:index], y1[:index])
        cos_drawing.set_data(x[:index], y2[:index])
        return sin_drawing, cos_drawing,

    a = FuncAnimation(fig, update, frames=len(x), interval=10)
    plt.show()
\end{lstlisting}

Animations are very useful for describing parametrized curves, as the ``speed'' of the curve is displayed.
The code below animates the rose curve, parametrized by the angle $\theta \in [0, 2\pi]$, given by the following equations:
%
\[\begin{array}{ccc}
x(\theta) = \cos(\theta)\cos(6\theta), && y(\theta) = \sin(\theta)\cos(6\theta)
\end{array}\]

\begin{lstlisting}
def rose_animation():
    # Calculate the parametrized data.
    theta = np.linspace(0, 2*np.pi, 200)
    x = np.cos(theta)*np.cos(6*theta)
    y = np.sin(theta)*np.cos(6*theta)

    fig = plt.figure()
    plt.xlim(-1.2, 1.2)
    plt.ylim(-1.2, 1.2)
    plt.gca().set_aspect("equal")           # Make the figure exactly square.

    drawing, = plt.plot([],[])

    # Define a function that updates the line data.
    def update(index):
        drawing.set_data(x[:index], y[:index])
        return drawing,

    a = FuncAnimation(fig, update, frames=len(x), interval=10, repeat=False)
    plt.show()              # repeat=False freezes the animation at the end.
\end{lstlisting}

Animations can also be 3-dimensional.
The only major difference is an extra operation to set the 3-dimensional component of the drawn object.
The code below animates the space curve parametrized by the following equations:
%
\[\begin{array}{ccccc}
x(\theta) = \cos(\theta)\cos(6\theta), &&
y(\theta) = \sin(\theta)\cos(6\theta), &&
z(\theta) = \frac{\theta}{10}
\end{array}\]

\begin{lstlisting}
def rose_animation_3D():
    theta = np.linspace(0, 2*np.pi, 200)
    x = np.cos(theta) * np.cos(6*theta)
    y = np.sin(theta) * np.cos(6*theta)
    z = theta / 10

    fig = plt.figure()
    ax = fig.gca(projection='3d')           # Make the figure 3-D.
    ax.set_xlim3d(-1.2, 1.2)                # Use ax instead of plt.
    ax.set_ylim3d(-1.2, 1.2)
    ax.set_aspect("equal")

    drawing, = ax.plot([],[],[])            # Provide 3 empty lists.

    # Update the first 2 dimensions like usual, then update the 3-D component.
    def update(index):
        drawing.set_data(x[:index], y[:index])
        drawing.set_3d_properties(z[:index])
        return drawing,

    a = FuncAnimation(fig, update, frames=len(x), interval=10, repeat=False)
    plt.show()
\end{lstlisting}

\begin{comment} % TODO: An ORIGINAL example of using widgets
\subsection*{Interactive Plots} % ---------------------------------------------

Matplotlib plots can be made interactive by adding \emph{widgets}.
Consider the following example, TAKEN FROM THE MATPLOTLIB DOCS BASICALLY AHHH

\begin{lstlisting}
>>> from matplotlib import widgets as wg

>>> ax = plt.subplot(111)
>>> plt.subplots_adjust(bottom=.25)         # Make some space for a slider bar.
>>> t = np.arange(0., 1., .001)
>>> a0 = 5.
>>> f0 = 3.
>>> s = a0 * np.sin(2 * np.pi * f0 * t)
>>> l = plt.plot(t, s)[0]
>>> plt.axis([0, 1, -10, 10])
>>> axfreq = plt.axes([.25, .05, .65, .03])
>>> axamp = plt.axes([.25, .1, .65, .03])

# Make some slider bars.
>>> sfreq = wg.Slider(axfreq, 'Freq', .1, 30., valinit=f0)
>>> samp = wg.Slider(axamp, 'Amp', .1, 10., valinit=a0)
>>> def update(val):                        # Function for updating the plot.
...     amp = samp.val                          # Read from one slider.
...     freq = sfreq.val                        # Read from the other slider.
...     l.set_ydata(amp * np.sin(2 * np.pi * freq * t))
...     plt.draw()                              # Refresh the plot.
>>> sfreq.on_changed(update)                # Connect the sliders to update().
>>> samp.on_changed(update)
>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
\includegraphics[width=.7\textwidth]{figures/interactive_plot.pdf}
\end{figure}
\end{comment}

% =============================================================================
% =============================================================================
% Stuff to move ===============================================================
% =============================================================================
% =============================================================================

\begin{comment}
\begin{info} % IPython Notebook inline plotting (move to notebook intro)
If you are executing these Matplotlib commands in an IPython shell, executing the \li{plt.show()} method will open a new window with the plot.
If you are using IPython Notebook, you have the option to display the plots within your notebook.
You may opt into this feature by running \li{\%matplotlib inline} or \li{\%matplotlib notebook} in your IPython Notebook.
The \li{inline} option shows the plot, whereas the \li{notebook} option shows the plot and provides controls to interact with the plot.
Additionally, when using this option, the plot is displayed after running the \li{plt.plot()} command; the \li{plt.show()} command is not necessary.
\end{info}
\end{comment}


\subsection{Green's Theorem for Circulation}
\begin{theorem}[Green's Theorem for Circulation]
	Let $C$ be a closed, counter-clockwise oriented curve in $\mathbb{R}^2$. For any differentiable vector field,
	\begin{equation*}
		\vec{F}(x,y) = \langle Q(x,y), R(x,y) \rangle \text{, } \oint\limits_{C}{R\mathrm{d}x + Q\mathrm{d}y} = \iint\limits_{D}{\left(\frac{\partial Q}{\partial x} - \frac{\partial R}{\partial y}\right)\mathrm{d}x\mathrm{d}y}
	\end{equation*}
	where $D$ is the interior of $C$.
\end{theorem}
\noindent
Or, in more modern notation,
\begin{equation*}
	\oint\limits_{C}{\vec{F} \cdot \mathrm{d}\vec{r}} = \iint\limits_{D}{\nabla\times\vec{F}\mathrm{d}A}
\end{equation*}

[INSERT IMAGE]

\noindent
This is saying that summing up the interior of the derivative is the to summing up the boundary of the function, very similar to the FTC.\\
One can think of this in a physical sense as saying that the work done by the vector field in moving a particle counter-clockwise on $C$ is equal to the rotation (curl) inside of  $C$ ($D$).\\
This theorem also relates the idea of path independence and curl of a conservative vector field that we proved the 2D case for. The left side shows path independence and will be 0 for conservative vector fields, and the right side shows curl, which will also be 0 for conservative vector fields.

\begin{proof}[Partial Proof]
	We will prove the 2D case, but the underlying argument is easily generalized.
	\begin{equation*}
		\oint\limits_{C}{\vec{F} \cdot \mathrm{d}\vec{r}} = \oint\limits_{c}{\langle P(x,y), 0 \rangle \cdot \mathrm{d}\vec{r}} + \oint\limits_{C}{\langle 0, Q(x,y) \rangle \cdot \mathrm{d}\vec{r}}
	\end{equation*}
	If $D$ is convex, we an break $C$ into 2 curves on the same interval, $C_1$ and $C_2$ such that $C_1 : \vec{r} \langle x, h_1(x) \rangle$ and $C_2 : \vec{r} \langle x, h_2(x) \rangle$, where $x \in [a,b]$.

	[INSERT IMAGE]

	\begin{equation*}
		= \int_{a}^{b}{P(x,h_1(x))\mathrm{d}x} - \int_{a}^{b}{P(x,h_2(x))\mathrm{d}x}
	\end{equation*} \begin{equation*}
		= -\int_{a}^{b}{P(x,h_2(x)) - P(x,h_1(x))\mathrm{d}x}
	\end{equation*} \begin{equation*}
		= -\int_{a}^{b}{P(x,y)\rvert_{y=h_1(x)}^{y=h_2(x)}\mathrm{d}x}
	\end{equation*} \begin{equation*}
		= -\int_{a}^{b}{\int_{h_1(x)}^{h_2(x)}{\frac{\partial P}{\partial y}\mathrm{d}y}\mathrm{d}x}
	\end{equation*} \begin{equation*}
		= -\iint\limits_{D}{\frac{\partial P}{\partial y}\mathrm{d}A} + \iint\limits_{D}{\frac{\partial Q}{\partial x}\mathrm{d}A}
	\end{equation*} \begin{equation*}
		= \iint\limits_{D}{\nabla \times \vec{F}\mathrm{d}A}
	\end{equation*}
\end{proof}

\noindent
For example, let's use Green's Theorem for Circulation to compute $\oint\limits_{C}{\vec{F} \cdot \mathrm{d}\vec{r}}$ where $\vec{F} = \langle x^2, xy + x^2 \rangle$ and $C$ is the unit circle.
\begin{equation*}
	\oint\limits_{C}{\vec{F} \cdot \mathrm{d}\vec{r}} = \iint\limits_{D}{\nabla \times \vec{F}\mathrm{d}A}
\end{equation*}
\begin{equation*}
	\nabla \times \vec{F} = y+2x
\end{equation*}
Since $C$ is the unit circle, we'll evaluate the integral using polar coordinates. So
\begin{equation*}
	\mathrm{d}A = r\mathrm{d}r\mathrm{d}\theta
\end{equation*}
\begin{align*}
	\iint\limits_{D}{\nabla \times \vec{F}\mathrm{d}A} &= \int_{0}^{2\pi}{\int_{0}^{1}{(r\sin{\theta} + 2r\cos{\theta})r\mathrm{d}r}\mathrm{d}\theta} \\
	&= \int_{0}^{2\pi}{\left(\frac{\sin{\theta}}{3} + \frac{2\cos{\theta}}{3}\right)\mathrm{d}\theta} = 0
\end{align*}
Note that although this particular circulation is 0, we know that the vector field is not conservative because the curl is not 0.

\input{./vectorAnalysis/areaClosedRegion}


\subsection{Eigenvalues \& Eigenvectors}
\begin{definition}
	Let $A$ be an $n \times n$ matrix. A scalar $\lambda$ and a vector $\vec{v}$ are an eigenvalue and eigenvector of $A$ if
	\begin{equation*}
		A\vec{v} = \lambda\vec{v}.
	\end{equation*}
\end{definition}
\noindent
We call $p(\lambda) = \det{(A - \lambda I)}$ the characteristic polynomial of $A$. The eigenvalues for $A$ are the solutions to the equation
\begin{equation*}
	p(\lambda) = \det{(A - \lambda I)} = 0.
\end{equation*}
Once we have an eigenvalue, we can find the basis vectors for the corresponding eigenspace by solving the equation
\begin{equation*}
	\left( A - \lambda I \right)\vec{v} = \vec{0}.
\end{equation*}
The basis vectors of the eigenspace for $A$ are the union of the basis vectors of each eigenspace corresponding to each eigenvalue.

\begin{example}
	Find the eigenvalues and eigenvectors of
	\begin{equation*}
		A = \begin{bmatrix}
			2 & 1 & 3 \\
			1 & 2 & 3 \\
			3 & 3 & 20
		\end{bmatrix}.
	\end{equation*}
\end{example}
\begin{equation*}
	p(\lambda) = \begin{array}{|ccc|}
		2-\lambda & 1 & 3 \\
		1 & 2-\lambda & 3 \\
		3 & 3 & 20-\lambda
	\end{array} =
	-(\lambda - 21)(\lambda - 2)(\lambda - 1) = 0 \implies \lambda = 1 \text{, } 2 \text{, and } 21.
\end{equation*}
When $\lambda = 1$,
\begin{equation*}
	A-\lambda I = \left[
	\begin{array}{ccc|c}
		2-1 & 1 & 3 & 0 \\
		1 & 2-1 & 3 & 0 \\
		3 & 3 & 20-1 & 0
	\end{array}
	\right] \to \left[
	\begin{array}{ccc|c}
		1 & 1 & 0 & 0 \\
		0 & 0 & 1 & 0 \\
		0 & 0 & 0 & 0
	\end{array}
	\right] \implies \vec{v_{1}} = t \begin{bmatrix}
		-1 \\
		1 \\
		0
	\end{bmatrix}.
\end{equation*}
When $\lambda = 2$,
\begin{equation*}
	A-\lambda I = \left[
	\begin{array}{ccc|c}
		2-2 & 1 & 3 & 0 \\
		1 & 2-2 & 3 & 0 \\
		3 & 3 & 20-2 & 0
	\end{array}
	\right] \to \left[
	\begin{array}{ccc|c}
		1 & 0 & 3 & 0 \\
		0 & 1 & 3 & 0 \\
		0 & 0 & 0 & 0
	\end{array}
	\right] \implies \vec{v_{2}} = \begin{bmatrix}
		-3 \\
		-3 \\
		1
	\end{bmatrix}.
\end{equation*}
When $\lambda = 21$,
\begin{equation*}
	A-\lambda I = \left[
	\begin{array}{ccc|c}
		2-21 & 1 & 3 & 0 \\
		1 & 2-21 & 3 & 0 \\
		3 & 3 & 20-21 & 0
	\end{array}
	\right] \to \left[
	\begin{array}{ccc|c}
		1 & 0 & -1/6 & 0 \\
		0 & 1 & -1/6 & 0 \\
		0 & 0 & 0 & 0
	\end{array}
	\right] \implies \vec{v_{21}} = \begin{bmatrix}
		1 \\
		1 \\
		6
	\end{bmatrix}.
\end{equation*}
Bonus: $A$'s diagonalization is
\begin{equation*}
	A = PDP^{-1} \implies \begin{bmatrix}
		2 & 1 & 3 \\
		1 & 2 & 3 \\
		3 & 3 & 20
	\end{bmatrix} = \begin{bmatrix}
		-1 & -3 & 1 \\
		1 & -3 & 1 \\
		0 & 1 & 6
	\end{bmatrix} \begin{bmatrix}
		1 & 0 & 0 \\
		0 & 2 & 0 \\
		0 & 0 & 21
	\end{bmatrix} \begin{bmatrix}
		-1/2 & 1/2 & 0 \\
		-3/19 & -3/19 & 1/19 \\
		1/38 & 1/38 & 3/19
	\end{bmatrix}.
\end{equation*}


\subsection{Expectation of OLS estimators}

\subsubsection{Expectation in terms of observables}

We have: \(\hat{\theta }=(X^TX)^{-1}X^Ty\)

Let’s take the expectation.

\(E[\hat{\theta }]=E[(X^TX)^{-1}X^Ty]\)

\subsubsection{Expectation in terms of errors}

Let’s model \(y\) as a function of \(X\). As we place no restrictions on the error terms, this is not as assumption.

\(y=X\theta +\epsilon\).

\(E[\hat{\theta }]=E[(X^TX)^{-1}X^T(X\theta +\epsilon)]\)

\(E[\hat{\theta }]=E[(X^TX)^{-1}X^TX\theta ]+E[(X^TX)^{-1}X^T \epsilon)]\)

\(E[\hat{\theta }]=\theta +E[(X^TX)^{-1}X^T \epsilon)]\)

\(E[\hat{\theta }]=\theta +E[(X^TX)^{-1}X^T]E[ \epsilon]+cov [(X^TX)^{-1}X^T ,\epsilon]\)

\subsubsection{The Gauss-Markov: Expected error is \(0\)}

\(E[\epsilon  =0]\)

This means that:

\(E[\hat{\theta }]=\theta + cov [(X^TX)^{-1}X^T ,\epsilon]\)

\subsubsection{The Gauss-Markov: Errors and indepedent variables are uncorrelated}

If the error terms and \(X\) are uncorrelated then \(E[\epsilon|X]=0\) and therefore:

\(E[\hat{\theta }]=\theta\)

So this is an unbiased estimator, so long as the condition holds.


\subsection{Concave and convex functions}

\subsubsection{Convex functions}

A convex function is one where:

\(\forall x_1, x_2\in \mathbb{R} \forall t \in [0,1] [f(tx_1+(1-t)x_2 \le tf(x_1)+(1-t)f(x_2)]\)

That is, for any two points of a function, a line between the two points is above the curve.

A function is strictly convex if the line between two points is strictly above the curve:

\(\forall x_1, x_2\in \mathbb{R} \forall t \in (0,1) [f(tx_1+(1-t)x_2 < tf(x_1)+(1-t)f(x_2)]\)

An example is \(y=x^2\).

\subsubsection{Concave functions}

A concave function is an upside down convex function. The line between two points is below the curve.

\(\forall x_1, x_2\in \mathbb{R} \forall t \in [0,1] [f(tx_1+(1-t)x_2 \ge tf(x_1)+(1-t)f(x_2)]\)

A function is strictly concave if the line between two points is strictly below the curve:

\(\forall x_1, x_2\in \mathbb{R} \forall t \in (0,1) [f(tx_1+(1-t)x_2 > tf(x_1)+(1-t)f(x_2)]\)

An example is \(y=-x^2\).

\subsubsection{Affine functions}

If a function is both concave and convex, then the line between two points must be the function itself. This means the function is an affine function.

\(y=cx\)


\subsection{Quotient Rule}
\begin{lemma}
	Let $f$ and $g$ be differentiable functions where $g \neq 0$. Then
	\begin{equation}
		\left(\frac{f}{g}\right)^\prime = \frac{gf^\prime - fg^\prime}{g^2}.
	\end{equation}
\end{lemma}
\begin{proof}
	Using the definition of the derivative and limit properties,
	\begin{align*}
		\left(\frac{f}{g}\right)^\prime &= \lim_{h\to 0}{\frac{\frac{f(x+h)}{g(x+h)} - \frac{f(x)}{g(x)}}{h}} \\
		&= \lim_{h\to 0}{\frac{1}{h}\frac{f(x+h)g(x) - f(x)g(x+h)}{g(x+h)g(x)}} \\
		&= \lim_{h\to 0}{\frac{1}{h}\hspace{3pt}\frac{f(x+h)g(x) - f(x)g(x) + f(x)g(x) - f(x)g(x+h)}{g(x+h)g(x)}} \\
		&= \lim_{h \to 0}{\frac{1}{g(x+h)g(x)}\hspace{3pt}\frac{f(x+h)g(x)-f(x)g(x)+f(x)g(x)-f(x)g(x+h)}{h}} \\
		&= \lim_{h \to 0}{\frac{1}{g(x+h)g(x)}\left(\frac{f(x+h)g(x)-f(x)g(x)}{h}+\frac{f(x)g(x)-f(x)g(x+h)}{h}\right)} \\
		&= \lim_{h \to 0}{\frac{1}{g(x+h)g(x)}} \left(g(x)\lim_{h \to 0}{\frac{f(x+h)-f(x)}{h}} - f(x)\lim_{h\to 0}{\frac{g(x+h)-g(x)}{h}}\right) \\
		&= \frac{1}{g^2(x)}\left(g(x)f^\prime(x) - f(x)g^\prime(x)\right) \\
		&= \frac{gf^\prime - fg^\prime}{g^2}.
	\end{align*}
\end{proof}

\begin{example}
	Given that the derivative of $\sin{(x)}$ is $\cos{(x)}$, find the derivative of $\frac{\sin{(x)}}{x^2}$.
\end{example}
\begin{answer}
	Applying the quotient and power rules,
	\begin{equation*}
		f^\prime(x) = \frac{x^2\cos{(x) - 2x\sin{(x)}}}{x^4} = \frac{x\cos{(x)}-2\sin{(x)}}{x^3}.
	\end{equation*}
\end{answer}


\subsection{Complex numbers}
When Eigenmath starts up, it defines the symbol $i$ as $i=\sqrt{-1}$.
Other than that, there is nothing special about $i$.
It is just a regular symbol that can be redefined and used for some other purpose if need be.

Complex quantities can be entered in either rectangular or polar form.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
a+i*b
\end{Verbatim}

$\displaystyle a+ib$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
exp(i*pi/3)
\end{Verbatim}

$\displaystyle \exp(\frac{1}{3}i\pi)$

Converting to rectangular or polar coordinates causes
simplification of mixed forms.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = 1+i
B = sqrt(2)*exp(i*pi/4)
A-B
\end{Verbatim}

$1+i-2^{1/2}\exp(\frac{1}{4}i\pi)$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
rect(last)
\end{Verbatim}

$\displaystyle 0$

Rectangular complex quantities, when raised to a power, are multiplied out.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
(a+i*b)^2
\end{Verbatim}

$\displaystyle a^2-b^2+2iab$

When $a$ and $b$ are numerical and the power is negative, the evaluation is done as follows.
$$i
(a+ib)^{-n}
=\left[\frac{a-ib}{(a+ib)(a-ib)}\right]^n=
\left[\frac{a-ib}{a^2+b^2}\right]^n$$
Of course, this causes $i$ to be removed from the denominator.
%For $n=1$ we have
%$${1\over a+ib}={a-ib\over a^2+b^2}$$
Here are a few examples.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
1/(2-i)
\end{Verbatim}

$\displaystyle \frac{2}{5}+\frac{1}{5}i$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
(-1+3i)/(2-i)
\end{Verbatim}

$\displaystyle -1+i$

The absolute value of a complex number returns its magnitude.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
abs(3+4*i)
\end{Verbatim}

$\displaystyle 5$

Since symbols can have complex values, the absolute value
of a symbolic expression is not computed.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
abs(a+b*i)
\end{Verbatim}

$\displaystyle {\rm abs}(a+ib)$

The result is not $\sqrt{a^2+b^2}$ because that would assume that
$a$ and $b$ are real.
For example, suppose that $a=0$ and $b=i$.
Then
$$|a+ib|=|-1|=1$$
and
$$\sqrt{a^2+b^2}=\sqrt{-1}=i$$
Hence
$$|a+ib|\ne\sqrt{a^2+b^2}\quad\hbox{for some $a,b\in\mathbb C$}$$

The $mag$ function can be used instead of $abs$.
It treats symbols like $a$ and $b$ as real.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
mag(a+b*i)
\end{Verbatim}

$\displaystyle (a^2+b^2)^{1/2}$

The imaginary unit can be changed from $i$ to $j$
by defining $j=\sqrt{-1}$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
j = sqrt(-1)
sqrt(-4)
\end{Verbatim}

$\displaystyle 2j$


\section{Regression: OLS}

\formdesc{Form}

\begin{equation}
	y = \beta_0 + \beta_1 x
\end{equation}

describes the true, unobserved model, while

\begin{equation}
	\hat{y} = b_0 + b_1x
\end{equation}

describes the estimated model. The estimate $\hat{y}$ describes the average value around which subjects where $x = x_i$ will cluster.
\hformbar

\formdesc{Residuals}

\begin{equation}
	e_i = y_i - \hat{y}_i
\end{equation}

is the residual of the $i$th observations $(x_i, y_i)$, the differences between the observed response $(y_i)$ and the prediction $\hat{y}$

\hformbar

\formdesc{Correlation}

\begin{equation}
	r = \frac{1}{n - 1} \sum_{i=1}^n \frac{x_i - \bar{x}}{s_x} ~ \frac{y_i - \bar{y}}{s_y}
\end{equation}

describes the strength of the linear relationship between two variables $x$ and $y$, where $0 \leq r \leq 1$, and $s$ is sample standard deviation
\hformbar

\formdesc{Least Squares Criterion}

\begin{equation}
	\mathrm{arg\,min} \sum_{i=1}^n e_i^2  \equiv e_1^2 + e_2^2 + \ldots + e_n^2
\end{equation}

describes the best fitting line, the \textit{least squares line}, i.e., minimizes the sum of squared residuals. To calculate:

\begin{equation}
	b_1 = \frac{s_y}{s_x} r
\end{equation}

then use the fact that the point $(\bar{x}, \bar{y})$ is on the least squares line to set $x_0 = \bar{x}$ and $y_0 = \bar{y}$ along with the slope $b_1$, solve for $x$ in:

\begin{equation}
	y - \hat{y} = b_1 (x - \hat{x})
\end{equation}

\subsection*{Assumptions}

\begin{enumerate}
	\item \textit{Linearity}. Data must show a linear trend
	\item \textit{Near normal residuals}
	\item \textit{Constant variability}. The variability of points around the least-squares line must be constant, e.g., the scale of $e$ cannot increase as $x$ increases producing a fanning pattern
	\item \textit{Independent observations}.
\end{enumerate}

\hformbar

\formdesc{$r^2$}

\begin{equation}
	r^2 = 1 - \frac{ \sum_{i=1}^n (y_i - \hat{y}_i)^2 }{ \sum_{i=1}^n (y_i - \bar{y})^2 } = 1 - \frac{SSE}{SST}
\end{equation}

\begin{itemize}
	\item $SSE$ is summed squares of residuals:
	\item $SSTO$ is total sum of squares
\end{itemize}

\hformbar

\formdesc{Regression: T-Test}

Tests to determine if null hypothesis $b_1 = 0$ is to be rejected
\hformbar

\newpage


%!TEX root = TTK4215-Summary.tex
\section{Preliminaries}
\subsection{Norms}
We say $x \in \mathcal{L}_p$ when $||x||_p$ exists.
\paragraph{General $p$-norm}
\begin{equation}
	||x||_p
	=
	\left(
		\int_{0}^{\infty} |x(t)|^p \dif t
	\right)
	^{1/p}
\end{equation}

\paragraph{$\mathcal{L}_\infty$-norm}
\begin{equation}
	||x||_\infty
	=
	\sup_{t \geq 0} |x(t)|
\end{equation}

%!TEX root = TTK4215-Summary.tex
\subsection{Models for dynamic systems}

\begin{equation}
	\V{\dot{x}}
	=
	\M{A} \V{x}
	+
	\M{B} \V{u}
\end{equation}

\begin{equation}
	\V{y}
	=
	\M{C}\T \V{x}
\end{equation}

\paragraph{Controllability}
\begin{equation}
	\M{P}_c
	\triangleq
	\begin{bmatrix}
		\M{B} \\ \M{AB} \\ \vdots \\ \M{A}^{n-1}\M{B}
	\end{bmatrix}
\end{equation}

If $\M{P}_c$ is nonsingular, the system is controllable, and can be transformed to the \emph{controllability canonical form} by
\begin{equation}
	\V{x}_c
	=
	\M{P}_c^{-1} \V{x}
\end{equation}

\paragraph{Properness}
A transfer function $\M{G}(s) = \frac{\M{N}(s)}{\M{D}(s)}$ is
\begin{itemize}
	\item \emph{proper} if $\deg(\M{N}) \leq \deg(\M{D})$,
	\item \emph{biproper} if $\deg(\M{N}) = \deg(\M{D})$,
	\item \emph{stricly proper} if $\deg(\M{N}) < \deg(\M{D})$.
\end{itemize}

\subsection{Transfer function properties}
Consider the polynomial
\begin{equation}
	X(s) = \alpha_n s^n + \alpha_{n-1} s^{n-1} + \cdots + \alpha_0
\end{equation}
and the transfer function
\begin{equation}
	G(s) = \frac{Z(s)}{R(s)}
	.
\end{equation}

\paragraph{Monic:} $X(s)$ is \emph{monic} iff $\alpha_n = 1$.

\paragraph{Hurwitz:} $X(s)$ is \emph{Hurwitz} if all roots of $X(s) = 0$ are in the left half plane.

\paragraph{Minimum phase:} A system defined by the t.f. $G(s)$ is \emph{minimum phase} iff $Z(s)$ is Hurwitz.

\paragraph{Stability:} A system defined by the t.f. $G(s)$ is \emph{stable} if $R(s)$ is Hurwitz.

\paragraph{Coprime:} Two polynomials are \emph{coprime} if they have no common factors other than a constant.

\subsection{(Strictly) positive real transfer functions}

\subsection{Positive Real (PR)}
\paragraph{Definition 3.5.1}
A rational function $G(s)$ of the complex variable $s=\sigma + j\omega$ is called PR if
\begin{itemize}
	\item $G(s)$ is real for real $s$
	\item $\Re[G(s)] \geq 0 \quad \forall \quad \sigma > 0$
\end{itemize}

\paragraph{Lemma 3.5.1}
A rational proper transfer function $G(s)$ is PR iff
\begin{itemize}
	\item $G(s)$ is real for real $s$
	\item $G(s)$ is analytic in $\Re[s] \geq 0$, and the poles on the $j\omega$-axis are simple and such that the associated residues are real and positive.
	\item For all real value $\omega$ for which $s = j \omega$ is not a pole of $G(s)$, one has $\Re[G(j\omega)]$
\end{itemize}

\subsection{Strict Positive Real (SPR)}
\paragraph{Definition 3.5.2}
Assume $G(s)$ is not identically zero for all $s$. Then $G(s)$ is SPR if $G(s-\epsilon)$ is PR for some $\epsilon > 0$.

\paragraph{Theorem 3.5.2}
(Necessary and sufficient conditions.) Assume that a rational function $G(s)$ of the complex variable $s = \sigma + j\omega$ is real for real $s$ and is not identically zero for all $s$. Let $n^*$ be the relative degree of $G(s) = Z(s) / R(s)$ with $|n^*| \leq 1$. Then, $G(s)$ is SPR iff
\begin{itemize}
	\item $G(s)$ is analytic for $\sigma \geq 0$
	\item $\Re[G(j\omega)] > 0 \quad \forall \quad \omega$
	\item When $n^* = 1, \quad \lim_{|\omega| \to \infty} \omega \Re[G(j\omega)] > 0 $
	\item When $n^* = -1,\quad \lim_{|\omega| \to \infty} \frac{G(j\omega)}{j\omega} >0 $
\end{itemize}

\paragraph{Corollary 3.5.1}
\begin{itemize}
	\item $G(s)$ is PR/SPR iff $1/G(s)$ is PR/SPR
	\item If $G(s)$ is SPR, then, $|n^*| \leq 1$, and the zeros and poles of $G(s)$ lie in $\Re[s]<0$.
	\item When $n^* = 1, \quad \lim_{|\omega| \to \infty} \omega \Re[G(j\omega)] > 0 $
	\item When $n^* = -1,\quad \lim_{|\omega| \to \infty} \frac{G(j\omega)}{j\omega} >0 $
\end{itemize}

\paragraph{KYP Lemma (3.5.2)}
Given a square matrix $A$ with eigenvalues $\Re(\lambda) \leq 0$, a vector $B$ such that $(A,B)$ controllable, a vector $C$, and scalar $d \geq 0$, then the t.f.
\begin{equation}
	G(s) = d + C\T (s \mathrm{I} - A)^{-1} B
\end{equation}
is PR iff $\exists$ a symmetric pos. def. matrix $P$ and a vector $q$ such that
\begin{gather}
	A\T P + P A = -q q\T \\
	P B - C = \pm \sqrt{2d} \cdot q
	.
\end{gather}

\paragraph{LKY Lemma (3.5.3)}
Given a stable matrix $A$, a vector $B$ such that $(A,B)$ controllable, a vector $C$ and a scalar $d \geq 0$, then the t.f.
\begin{equation}
	G(s) = d + C\T (s \mathrm{I} - A)^{-1} B
\end{equation}
is SPR iff for any pos. def. matrix $L$, $\exists$ a symmetric pos. def. matrix $P$, a scalar $\nu > 0$ and a vector $q$ such that
\begin{gather}
	A\T P + P A = -q q\T - \nu L \\
	P B - C = \pm q \sqrt{2d}
	.
\end{gather}

\paragraph{MKY Lemma (3.5.4)}
Given a stable matrix $A$, vectors $B,C$, and a scalar $d \geq 0$, we have: If
\begin{equation}
	G(s) = d + C\T (s \mathrm{I} - A)^{-1} B
\end{equation}
is SPR, then for any $L = L\T > 0$, $\exists$ a scalar $\nu > 0$, a vector $q$ and a $P = P\T > 0$ such that
\begin{gather}
	A\T P + P A = -q q\T - \nu L \\
	P B - C = \pm q \sqrt{2d}
	.
\end{gather}

TTK4215 System identification and adaptive control/sec-preliminaries.tex


\section{Norm and distance}
\subsection{Norm}
\textit{Euclidean norm} (or just norm) is\\
$||x||= \sqrt{x_1^2+x_2^2+...+x_n^2}= \sqrt{x^Tx}$\\
\textbf{Properties}\\
\textbullet homogeneity: $||\beta x|| = |\beta ||x|||$\\
\textbullet triangle inequality: $||x + y|| \leq ||x|| + ||y||$\\
\textbullet non negativity: $||x|| \geq 0$\\
\textbullet definiteness: $||x|| = 0$ only if x = 0\\
\textit{positive definiteness} = non negativity + definiteness\\
\textbf{rms}(x) = $\sqrt{\frac{x_1^2+...+x_n^2}{n}} = \frac{||x||}{\sqrt{n}}$ \\
\textbullet \textit{Norm of a sum}:\\
$\Vert a + b \Vert^2 = (x + y)^T(x + y) = \Vert x \Vert^2 + 2x^Ty + \Vert b \Vert^2$\\
\textbf{Norm of block vectors}
$||(a,b,c)|| = \sqrt{||a||^2 + ||b||^2 + ||c||^2} = ||(||a||, ||b||, ||c||)||$\\
\textbf{Chebyshev inequality}
k of its entries satisfy $|x_i| \geq a$, \\then $\frac{k}{n} \leq (\frac{\textbf{rms}(x)}{a})^2$
\subsection{Distance}
$\textbf{dist}(a,b) = ||a - b||$\\
\textit{Triangle Inequality}: $||a - c||^2 = ||(a - b) + (b - c)|| \leq ||a-b|| + ||b-c||$\\
$z_j$ is the nearest neighbor of x if $||x-z_j|| \leq ||x-z_i||, i=1,..,m$\\

\subsection{Standard Deviation}
de-meaned vector: $\tilde{x} = x - \textbf{avg}(x)\textbf{1}$\\
standard deviation: \\\textbf{std}(x) = \textbf{rms}($\tilde{x}$) = $\frac{||x - (\textbf{1}^Tx/n)\textbf{1}||}{\sqrt{n}}$\\
${\textbf{rms}(x)}^2 = {\textbf{avg}(x)}^2 + {\textbf{std}(x)}^2$\\
By Chebyshev inequality, $|x_i - \textbf{avg}(x)| \geq \alpha \textbf{std}(x)$ then $k/n \leq (std(x)/a)^2$. (This inequality is only interesting for $a > std(x)$) \\
\textit{Cauchy–Schwarz inequality}: $|a^Tb| \leq \Vert a\Vert \Vert b \Vert$
\subsection{Angle}
angle between two nonzero vectors a, b defined as\\
$\angle(a,b) = arccos(\frac{a^Tb}{\Vert a \Vert \Vert b \Vert})$\\
$a^Tb = \Vert a \Vert \Vert b \Vert cos (\angle(a,b))$\\
\textbf{Classification of angles}\\

$\theta = \pi/2$: $a \perp b$\\
$\theta = 0$: $a^Tb = \Vert a \Vert \Vert b \Vert$\\
$\theta = \pi = 180^{\circ}$ : $a^Tb = -\Vert a \Vert \Vert b\Vert$\\
$\theta \leq \pi/2 = 90^{\circ} = a^Tb \geq 0$\\
$\theta \geq \pi/2 = 90^{\circ} = a^Tb \leq 0$\\
\textbf{Correlation Coeficient ($\rho$)}
$\rho = \frac{\tilde{a}^T\tilde{b}}{\Vert\tilde{a}\Vert\Vert\tilde{b}\Vert}$\\
With $u = \tilde{a}/\textbf{std}(a)$ \& $u = \tilde{b}/\textbf{std}(b)$, \\
$\rho = u^Tv/n$ where $\Vert u \Vert = \Vert v \Vert = n$\\

$std(a+b) = \\
\sqrt{std(a)^2+ 2\rho std(a)std(b) +std(b)^2}$\\
\textbf{Properties of standard deviation}\\
\textbullet  $\textbf{std}(x+a1) = \textbf{std}(x)$\\
\textbullet $\textbf{std}(ax) = |a| std(x)$\\
\textbf{Standardization}
$z = \frac{1}{\textbf{std}(x)}(x - \textbf{avg}(x)\textbf{1})$

\subsection{Complexity}
\textbullet \textit{norm}: 2n\\
\textbullet \textit{\textbf{rms}}: 2n\\
\textbullet \textit{\textbf{dist}(a,b)}: 3n\\
\textbullet \textit{$\angle(a,b)$}: 6n\\


\section{First and Second Derivative Tests}
\subsection{First Derivative Test}
As we saw in previous sections, we can use the first derivative to find critical values, which allow us to find extrema.
\begin{theorem}[First Derivative Test]
	Let $f(x)$ be a continuous function.
	At a critical point $c$,
	\begin{enumerate}
		\item If $f^\prime$ changes sign from positive to negative ($f^\prime(x) > 0$ for $x < c$ and $f^\prime(x) < 0$ for $x > c$), then $f$ has a local maximum at $c$.
		\item If $f^\prime$ changes sign from negative to positive ($f^\prime(x) < 0$ for $x < c$ and $f^\prime(x) > 0$ for $x > c$), then $f$ has a local minimum at $c$.
		\item If $f^\prime$ does not change sign at $c$, then $f$ does not have a local extrema at $c$.
		\item At a left endpoint $a$, if ($f^\prime < 0$ / $f^\prime > 0$), then $f$ has a local (maximum / minimum) at $a$.
		\item At a right endpoint $b$, if ($f^\prime < 0$ / $f^\prime > 0$), then $f$ has a local (minimum / maximum) at $b$.
	\end{enumerate}
\end{theorem}

\begin{example}
	Find the local extrema of $f(x) = x^3 - 12x - 5$.
	Identify any absolute extrema.
\end{example}
\begin{answer}
	Taking the derivative,
	\begin{equation*}
		f^\prime(x) = 3x^2 - 12 = 3(x+2)(x-2).
	\end{equation*}

	So, the critical values are $x=-2$ and $x=2$.
	$f^\prime$ is negative between these values and positive outside of them, so $x=-2$ is a local maximum, while $x=2$ is a local minimum.
	\begin{equation*}
		f(-2) = 11 \text{ and } f(2) = -21.
	\end{equation*}

	As $x$ approaches $\infty$ (the right endpoint), $f$ also approaches $\infty$, so there is no absolute maximum.
	Similarly, as $x$ approaches $-\infty$ (the left endpoint), $f$ also approaches $-\infty$, so there is no absolute minimum.
\end{answer}

\subsection{Second Derivative Test}
The second derivative tells us how the derivative is changing.
Whether the derivative is increasing or decreasing describes the concavity.
\begin{definition}
	On some open interval $I$, the graph of a twice differentiable function $f(x)$ is
	\begin{enumerate}
		\item Concave up if $f^{\prime\prime} > 0$ on $I$.
		\item Concave down if $f^{\prime\prime} < 0$ on $I$.
	\end{enumerate}
\end{definition}

Concave up portions of a graph tend to look like valleys, while concave down portions tend to look like hills.
Unlike the first derivative, we can tell if a function is increasing or decreasing using concavity.
We can however say if the function is increasing or decreasing more or less rapidly.

\begin{figure}[H]
	\label{mvt}
	\centering
	\includegraphics[width = 0.5\textwidth]{./applications_derivative/concavity.png}
	\caption{\hyperref{https://tutorial.math.lamar.edu/classes/calci/shapeofgraphptii.aspx}{}{}{Paul's Online Notes - The Shape of a Graph, Part II}}
\end{figure}

The points where a function changes concavity are called "inflection points".
At these points, the function is increasing or decreasing most rapidly, depending on the sign of the first derivative.

Rather than looking at the sign of $f^\prime$ around critical values, we can look at the concavity at the critical point.
\begin{theorem}[Second Derivative Test]
	Let $c$ be a critical value of $f$.
	\begin{enumerate}
		\item If $f^{\prime\prime}(c) < 0$, then $c$ is a local maximum.
		\item If $f^{\prime\prime}(c) > 0$, then $c$ is a local minimum.
		\item If $f^{\prime\prime}(c) = 0$, then the test is inclusive.
	\end{enumerate}
\end{theorem}

In other words, if we're at a critical point that's turning into a valley, then the critical point must be the top of a hill.
If we're at a critical point that's turning into a hill, then the critical point must be the bottom of a valley.
This test is particularly useful because you only need to know $f^{\prime\prime}$ at $c$ rather than an entire interval\footnote{It also extends to higher dimensions much better than the first derivative test.}.

\begin{example}
	Use the second derivative test to find the local extreme values of $f(x) = x^3 - 12x - 5$.
\end{example}
\begin{answer}
	Finding critical values,
	\begin{align*}
		f^\prime(x) &= 3x^2 - 12 = 3(x+2)(x-2). \\
		f^\prime(x) &= 0\text{ at } x=-2, x=2.
	\end{align*}

	Taking the second derivative,
	\begin{equation*}
		f^{\prime\prime}(x) = 6x.
	\end{equation*}

	Evaluating the second derivative at the critical values,
	\begin{equation*}
		f^{\prime\prime}(-2) = -12 \text{ and } f^{\prime\prime}(2) = 12.
	\end{equation*}

	So, $x=-2$ is a local maximum and $x=2$ is a local minimum.
\end{answer}


\section*{Exercise 20.3-5}
\subsection*{Running times for the modified operations}

The recursive procedures that implement the vEB tree operations have running times characterized by the recurrence
$$
T(u) \leq T(u^{1/k}) + T(u^{1-1/k}) + O(1),
$$
where the first part is due to the number of clusters, and the second part is due to the size of the clusters, which are enumerated.
\\
By letting $m=\lg u$, such that $u=2^m$, then
$$
T(2^m) \leq T(2^{m/k}) + T(u^{m-m/k}) + O(1)
$$
We then rename $S(m)=T(2^m)$, which results in
$$
S(m) \leq S(m/k) + S(m-m/k) + O(1) = S(m/k) + S\left(\frac{m(k-1)}{k}\right) + O(1)
$$
For $k>2$, the second part will dominate, so in terms of running time, we can "ignore" the first term. In this case, case 2 of the master theorem applies, and $S(m)$ has the solution $S(m)=O(\lg m)$.
By changing back, we can conclude, that
$$
T(u) = T(2^m) = S(m) = O(\lg m) = O(\lg\lg u)
$$


\chapter{Metric Definitions}

\section{Pixel Error}
Given two images, pixel error is defined as the mean of the absolute numerical distance between corresponding pixels in both images. Formally, we define:

$$P(X, Y) = \frac{\sum_{\bm{i} \in S}{|X_{\bm{i}} - Y_{\bm{i}}|}}{|S|}$$
for tensors $X$ and $Y$ over all indexes $\bm{i}$ in the index space $S$ for tensors $X$ and $Y$.

\section{Rand Score}

The version of the Rand Score used in this thesis is the Rand F Score. The Rand F Score essentially counts all the non-distinct pairs in an image that are correctly labeled as belonging to the same or different grouping with respect to a reference image. Formally, we define $S_1, S_2, ... S_n \subseteq S$ to be the set of all groupings $S_i$ in an input volume (i.e. distinct labels in a segmentation)m and $T_1, T_2, ... T_n \subseteq T$ to be the set of all groupings $T_i$ in a ground-truth volume. Let:

$$ t_i = |T_i| $$
$$ s_i = |S_i| $$
$$ c_{i,j} = |S_i \cap T_j| $$

Then, the Rand F Score can be defined as:

$$R_{\mathsf{Full}}(S,T) = \frac{\sum_{i,j}{c_{i,j}}}{\alpha \sum_{i}{s_i^2} + (1 - \alpha)\sum_j{t_i^2}}$$

$$R_{\mathsf{Merge}}(S,T) = \frac{\sum_{i,j}{c_{i,j}}}{\sum_{i}{s_i^2}}$$

$$R_{\mathsf{Split}}(S,T) = \frac{\sum_{i,j}{c_{i,j}}}{\sum_{j}{t_j^2}}$$

where $N$ is the number of voxels in a volume. See \url{https://github.com/seung-lab/segascorus/blob/master/segerror-manual.pdf} for more details of the Rand F Score.

\section{Cross Correlation}

Given two images, cross-correlation is defined as the mean of the prodiuct of corresponding pixels in both images. Formally, we define:

$$C(X, Y) = \frac{\sum_{\bm{i} \in S}{|X_{\bm{i}} \times Y_{\bm{i}}|}}{|S|}$$
for tensors $X$ and $Y$ over all indexes $\bm{i}$ in the index space $S$ for tensors $X$ and $Y$.

\subsection{Smoothed Version}

Because standard Cross Correlation is not particularly continuous (i.e. two images will typically be highly correlated if they match exactly, and loosely correlated if they are translated or rotated), we define a somewhat more smoothed version that attempts to make the function more continuous. We define the function:

$$C_s(X, Y) = \frac{\sum_{\bm{i} \in S}{|X_{\bm{i}} \times (G_{\theta}(Y))_{\bm{i}}|}}{|S|}$$
for tensors $X$ and $Y$ over all indexes $\bm{i}$ in the index space $S$ for tensors $X$ and $Y$. The function $G_{\theta}(\cdot)$ is a smoothing function, which applies a Gaussian filter to its arguments based on parameters $\theta$. The amount of smoothing that occurs determines how dontinuous the smoothed cross-correlation function is. Because the continuity of this function can be controlled, it makes for a more-useful loss function than standard Cross Correlation.


\section{Theory}
\label{sec:theory}

\subsection{Gradient Descent}
The aim of Stochastic Gradient Descent is to minimize the neural network' cost function with respect to its weights.
The algorithm initializes the weights to random values.
Then, it iteratively select some training example $\nu$ from the training dataset $\mathbb{D}_{train}$ and updates the weights by some fraction $\eta$ (usually called learning rate) of the gradient $\nabla_{w_j} e^\nu$ of the cost function $e^\nu$ with respect to the network's weights $w_j$:
\begin{equation}
    \label{eq:weights-update}
    w_j \leftarrow w_j - \eta \cdot \nabla_{w_j} e^\nu.
\end{equation}

In order to use this iterative procedure, we need to define a proper cost function for the regression problem.
A common choice is to use the squared error for the current example $\varepsilon^\nu$:
\begin{equation}
    \label{eq:cost}
    e^\nu = \frac{1}{2} (\sigma(\varepsilon^\nu) - \tau(\varepsilon^\nu))^2,
\end{equation}
where $\tau(\varepsilon^\nu))$ denotes the correct prediction for the current example.

Another possibility is to use all training examples (or a batch of them) to perform a single update, i.e. using the following cost function instead of $e$:
\begin{equation}
    \label{eq:cost-total}
    E = \frac{1}{2} \frac{1}{M} \sum_{\mu = 1}^{M} (\sigma(\varepsilon^\mu) - \tau(\varepsilon^\mu))^2.
\end{equation}
In this case, we usually talk of Gradient Descent (or Batch Gradient Descent) instead of Stochastic Gradient Descent.

\subsection{Gradient Computation}
The hidden units in out network take as input the training examples $\varepsilon$ and use the hyperbolic tangent as the activation function:
\begin{equation}
    g(\varepsilon) = tanh(w \cdot \varepsilon)
\end{equation}

Since we have only $2$ hidden units and the output unit simply computes the sum of the outputs of the hidden layer, the final predicted value is computed as:
\begin{equation}
    \sigma (\varepsilon) = tanh(w_1 \cdot \varepsilon) + tanh(w_2 \cdot \varepsilon),
\end{equation}
where $w_1$ and $w_2$ are the weights' vectors.

The gradient of the cost function defined in \cref{eq:cost} with respect to the weights $w_j$ for a single hidden unit $j$ can be computed as follows:
\begin{equation}
    \label{eq:gradient-gen-weight}
    \begin{split}
        \nabla_{w_j} e^\nu =\; & (tanh(w_1 \cdot \varepsilon^\nu) + tanh(w_2 \cdot \varepsilon^\nu) - \tau(\varepsilon^\nu)) \; \cdot \\
        & \cdot (1 - tanh^2(w_j \cdot \varepsilon^\nu)) \cdot \varepsilon^\nu
    \end{split}
\end{equation}


\chapter{The Dot Product}
% Reference for diagrams:https://www.mathsisfun.com/algebra/vectors-dot-product.html

If you have two vectors $u = [u_1, u_2, \dots, w_n]$ and $v = [v_1, v_2,\dots, v_n]$ ,
we define the \newterm{dot product} $u \cdot v$ as
\begin{equation*}
     u \cdot v = (u_1 \times v_1) + (u_2 \times v_2) + \dots + (u_n \times v_n)
\end{equation*}

So, for example,
\begin{equation*}
    [2,4, -3] \cdot [5, -1, 1] = 2 \times 5 + 4 \times -1 + -3 \times 1 = 3
\end{equation*}\index{dot product}

This may not seem like a very powerful idea, but dot products are \emph{incredibly} useful.
The enormous GPUs(Graphics Processing Unit) that let video games render scenes so quickly?
They are primarily function by computing huge numbers of dot products at mind-boggling speeds.

\begin{Exercise}[title={Basic dot products}, label=dot_products]
    Compute the dot product of each pair of vectors:
    \begin{itemize}
        \item $[1, 2, 3]$, $[4, 5, -6]$
        \item $[\pi, 2\pi]$, $[2, -1]$
        \item $[0,0,0,0]$, $[10,10,10,10]$
    \end{itemize}
\end{Exercise}
\begin{Answer}[ref=dot_products]
        \begin{itemize}
            \item $[1, 2, 3] \cdot [4, 5, -6] = 4 + 10 - 18 = -4$
            \item $[\pi, 2\pi] \cdot [2, -1] = 2\pi - 2\pi = 0$
            \item $[0,0,0,0] \cdot [10,10,10,10] = 0 + 0 + 0 + 0 = 0$
        \end{itemize}
\end{Answer}

\section{Properties of the dot product}

Sometimes we need an easy way to say ``The vector of appropriate length is filled with zeros.''
We use the notation $\vec{0}$ to represent this. Then, for any vector $v$, this is true:

$$v \cdot \vec{0} = 0$$

The dot product is commutative:

$$v \cdot u = u \cdot v$$

The dot product of a vector with itself is its magnitude squared:

$$ v \cdot v = |v|^2 $$

If you have a scalar $a$ then:

    $$(v) \cdot (a u) = a (v \cdot u)$$

So, if $v$ and $w$ are vectors that go in the same direction,

    $$v \cdot w = |v| |w|$$

If $v$ and $w$ are vectors that go in opposite directions,

    $$v \cdot w = -|v| |w|$$

\section{Cosines and dot products}

Furthermore, dot products' interaction with cosine makes them even more useful is what makes them so useful:
If you have two vectors $v$ and $u$,

$$v \cdot u = |v| |u| \cos \theta$$

where $\theta$ is the angle between them.

So, for example, if two vectors $v$ and $u$ are perpendicular, the angle between them is $\pi/2$.
The cosine of $\pi/2$ is 0: The dot product of any two perpendicular vectors is always 0. In fact, if
the dot product of two non-zero vectors is 0, the vectors \textit{must be} perpendicular.
% diagram needed

\begin{Exercise}[title={Using dot products}, label=cos_dot_products]
    What is the angle between these each pair of vectors:
    \begin{itemize}
        \item $[1, 0]$, $[0, 1]$
        \item $[3,4]$, $[4,3]$
    \end{itemize}
\end{Exercise}
\begin{Answer}[ref=cos_dot_products]
        \begin{itemize}
            \item $[1,0] \cdot [0,1] = 0$.  The angle must be $\pi/2$.
            \item $[3,4] \cdot [4, 3] = 24$. $|[3,4]| |[4,3]| \cos(\theta) = 24$.
            $\cos(\theta) = \frac{24}{(5)(5)}$. $\theta = \arccos(\frac{24}{25}) \approx 0.284 \text{ radians}$.
        \end{itemize}
\end{Answer}

If you have two non-zero vectors $v$ and $u$, you can always compute the angle between them:\index{vectors!angle between}

$$\theta = \arccos(\frac{v \cdot u}{|v| |u|})$$
% Breif Description of arccos
\section{Dot products in Python}

NumPy will let you do dot products using the the symbol @.  Open \filename{first\_vectors.py}
and add the following to the end of the script:

\begin{Verbatim}
    # Take the dot product
    d = v @ u
    print("v @ u =", d)

    # Get the angle between the vectors
    a = np.arccos(d / (mv * mu))
    print(f"The angle between u and v is {a * 180 / np.pi:.2f} degrees")
\end{Verbatim}

When you run it you should get:
\begin{Verbatim}
v @ u = 4
The angle between u and v is 78.55 degrees
\end{Verbatim}

\section{Work and Power}
%diagram here
Earlier, we mentioned that mechanical work is the product of the
force you apply to something and the amount it moves. For example, if you
push a train with a force of 10 newtons as it moves 5 meters, you have done 50 joules of work.

What if you try to push the train sideways? That is, it moves down the track 5 meters,
but you push it as if you were trying to derail it -- perpendicular to its motion.
You have done no work because the train didn't move at all in the direction you were pushing.

Now that you know about dot products: The work you do is the dot
product of the force vector you apply and the displacement vector of the train. (The displacement
vector is the vector that tells how the train moved while you pushed it.) \index{work}

Similary, we mentioned that power is the product of the force you apply and the velocity of the
mass you are applying it to. It is actually the dot product of the force vector and the velocity vector.\index{power}

For example, if you are pushing sled with a force of 10 newtons and it is moving 2 meters per second,
but your push is 20 degrees off, you aren't transferring 20 watts of power to the sled.
You are transferring $10 \times 2 \times \cos(20 \text{ degrees}) \approx 18.8$ watts of power.
%add ramps and sin


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 3}

Prove that if the subtree rooted at node $i$ has $n$ nodes, the subtree rooted at either of its children has size at most $2n/3$.

\subsection*{Solution}

The objective is to find an upper bound on the number nodes of a subtree based on number of nodes of a subtree rooted at its parent.
We define $H$ as the height of the latter subtree rooted at node $i$.

To find the upper bound, we try to place as many nodes as possible on a subtree $L$ rooted at a child of $i$ while trying to maintain as few nodes as possible on subtree $R$ at the other child of $i$.
This practice is however restricted by definition of the heap that requires all higher levels of the tree be completely filled.
Thus the difference would be on the lowest level where we are allowed to add nodes from the left.

The imbalance between subtrees $L$ and $R$ will be most significant where last row is half full, in which case subtree $L$ will have height $(H-1)$ while subtree $R$ will have height $(H-2)$, due to the fact that both are completely filled.

If number of nodes of left and right subtrees be defined as $n(L)$ and $n(R)$, respectively, the following would hold.

\begin{equation}\label{eq31}
n(L) = 2^{(H-1)+1}-1 = 2^H - 1
\end{equation}

\begin{equation}\label{eq32}
n(R) = 2^{(H-2)+1}-1 = 2^{H-1} - 1
\end{equation}

since roots of subtrees $L$ and $R$ are immediate children of node $i$ we will also have

\begin{equation}\label{eq33}
n = 1 + n(L) + n(R)
\end{equation}

where $n$ is total number of nodes of a subtree with root at node $i$.

Substituting \ref{eq31} and \ref{eq32} into \ref{eq33},

\begin{equation}\label{eq34}
\begin{aligned}
n &= 2^H + 2^{H-1} - 1\\
&= 2^{H} (1 + \frac{1}{2}) - 1\\
&= \frac{3}{2} n(L) + \frac{1}{2}
\end{aligned}
\end{equation}

which proves that $n(L) < \frac{2}{3}n$.


\input{6DL/BarronSpace}
\subsection{Cosine function as activation function}
	Consider the Fourier transform:
	\begin{equation}
	  \label{Fourier}
	  \hat f(\omega)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d}e^{-i\omega\cdot x}f(x)dx
	  \quad \forall \omega \in \mathbb R^d.
	\end{equation}
	We write
	$\hat{f}(\omega)=e^{i\beta(\omega)}|\hat{f}(\omega)|$. By Fourier inversion formula,
	\begin{equation}
		\label{eqn1}
		f(x)=\int_{\mathbb{R}^d}e^{i\omega\cdot x}\hat{f}(\omega)d\omega
		=\int_{\mathbb{R}^d}e^{i(\omega\cdot x+\beta(\omega))}|\hat{f}(\omega)|d\omega.
	\end{equation}
Since $f(x)$ is real-valued, it implies that, for $x$
	  \begin{equation}
		\label{key}
		\begin{aligned}
			f(x)
			&={\rm Re}\int_{\mathbb{R}^d}
			e^{i\omega\cdot x}
			\hat{f}(\omega)d\omega \\
			&={\rm Re}\int_{\mathbb{R}^d}
			e^{i\omega\cdot x}
			e^{i\beta
				(\omega)}|\hat{f}(\omega)|d\omega \\
			&=\int_{\mathbb{R}^d}\cos(\omega\cdot
			x+\beta(\omega))|\hat{f}(\omega)|d\omega.
		\end{aligned}
	\end{equation}
	Then we have
	\begin{equation}\label{key}
	f(x) =\int_{\mathbb{R}^d}k(x,\omega)d\omega,
	\end{equation}
	with
	\begin{equation}\label{key}
	k(x,\omega) = \cos(\omega\cdot
	x+\beta(\omega))|\hat{f}(\omega)|.
	\end{equation}
	Let
	$$
	\rho(\omega) = |\hat f(\omega)|,\qquad \lambda(\omega)={\rho(\omega)\over \|\rho(\omega)\|_{L^1}}={\rho(\omega)\over \|\hat f\|_{L^1}}.
	$$

\begin{theorem}
There exist $\omega_i \in \mathbb{R}^d$, s.t., $G = \mathbb{R}$ and
\begin{equation}\label{key}
\| f(x) - f_n(x)\|_{L^2} \le \frac{1}{\sqrt{n}} \int_{\mathbb{R}^d} |\hat f(\omega)| d \omega,
\end{equation}
where
\begin{equation}\label{key}
f_n(x)  = \frac{\|\hat f\|_{L^1}}{n} \sum_{i=1}^n \frac{ cos(\omega^*_i \cdot x + \beta^*_i)}{\rho(\omega_i^*)}.
\end{equation}
\end{theorem}
Note that
\begin{equation}\label{key}
f_n = \frac{\|\hat f\|_{L^1}}{n}\sum_{i=1}^n \frac{ cos(\omega^*_i \cdot x + \beta^*_i)}{\rho(\omega_i^*)} \in \dnn(\cos, n).
\end{equation}


\subsection{Definition}
Let’s add another function: addition. Defined by:

\(\forall a \in \mathbb{N} (a+0=a)\)

\(\forall a b \in \mathbb{N} (a+s(b)=s(a+b))\)

That is, adding zero to a number doesn’t change it, and \((a+b)+1=a+(b+1)\).

\subsection{Example}

Let’s use this to solve \(1+2\):

\(1+2=1+s(1)\)

\(1+s(1)=s(1+1)\)

\(s(1+1)=s(1+s(0))\)

\(s(1+s(0))=s(s(1+0))\)

\(s(s(1+0))=s(s(1))\)

\(s(s(1))=s(2)\)

\(s(2)=3\)

\(1+2=3\)

All addition can be done iteratively like this.

\subsection{Commutative property of addition}

Addition is commutative:

\(x+y=y+x\)

\subsection{Associative property of addition}

Addition is associative:

\(x+(y+z)=(x+y)+z\)