\section{Matrices}
\subsection{Matrices}
The set of real m × n matrices is denoted $R^{m \times n}$
\subsection{Zero and identity matrices}
\textbullet \textit{Zero}: All elements equals 0.\\
\textbullet \textit{Identity}: All elements equals 0 and diagonal element equals 1.\\
\textbullet \textit{Sparse}: If many entries are 0\\
\textbullet \textit{Diagonal}: off-diagonal entries are zero\\
\textbullet \textit{Triangular}: \textit{upper triangular} if $A_{ij} = 0 \text{ for } i>j$, and it is \textit{lower triangular} if $A_{ij} =0 \text{ for } i<j$\\
\textbullet \textit{Adjacency Matrix}:
For, $R={(1,2), (1,3), (2,1), (2,4), (3,4), (4,1)}$\\
$
A_{ij} = \left\{
  \begin{array}{@{}ll@{}}
    1, & (i,j) \in R\\
    0, & (i, j) \not\in R
  \end{array}\right.
$\\
A relation R on ${1,...,n}$ is represented by the n×n matrix A with $A_{ij} = 1$, if there exists a edge else , $A_{ij} = 0$
\subsection{Transpose, addition and norm}
\textbf{Block matrix Transpose}\\
$
\begin{bmatrix}
    A & B \\
    C & D

\end{bmatrix}^{T}=
\begin{bmatrix}
    A^T & C^T \\
    B^T & D^T
\end{bmatrix}
$\\
\textbf{Symmetric matrix}: $A = A^T$\\
\textbf{Properties of matrix addition}\\
\textbullet Commutativity: $A+B=B+A$\\
\textbullet Associativity: $(A + B) + C = A + (B + C)$\\
\textbullet Addition with zero matrix: $A+0=0+A=A$\\
\textbullet Transpose of sum:  $(A+B)^T = A^T +B^T$\\
If A is a matrix and $\beta$ , $\gamma$ are scalars
$(\beta + \gamma)A = \beta A + \gamma A,
(\beta \gamma)A = \beta(\gamma A)
$\\
\textbf{Matrix norm}
$\Vert A \Vert = \sqrt{\sum_{i=1}^n\sum_{j=1}^m} A^2_{ij}$
matrix norm satisfies the properties of any norm

\subsection{Matrix-vector multiplication}
A is an $m\times n$ matrix and x is an n-vector, then the matrix-vector product $y = Ax$\\

$y_i = \sum_{k=1}^n A_{ik} x_k =A_{i1}x_1 +...+A_{in}x_n$ for $i=1...m$\\
\textbullet \textbf{Row and column interpretations.}\\
$y = Ax$ can be expressed as $y_i=b^T_ix, i=1,..,m$ where $b^T_1,...,b^T_m$ are rows of A\\
\textbullet $y = Ax$ could also be expressd in terms of column
$y = x_1a_1 + x_2a_2 +...+ x_na_n$
\subsection{Complexity}
\textit{addition}: $mn$\\
\textit{sparse matrix addition}: If A or B or both are sparse $min\{nnz(A), nnz(B)\}$\\
\textit{vector multiplication $A_{mxn}$ with n-vector}: $m(2n-1) \approx 2mn$\\
\textit{Matrix Transpose}: 0 flops


\lab{Applications}{Linear Algebra in Statistics}{Statistics}
\label{Stats1}

\objective{This section will teach about using Python to solve problems in statistics. This will include finding means, correlation matrices, and solving least squares problems.}

\section*{Shifting Data by the Mean}

Consider the table below representing students scores in a class.\\

\begin{figure}[h!]
\begin{center}
\begin{tabular}{|c|r|r|r|r|}
	\hline
Student & Homework & Exam 1  & Exam 2 & Final \\
\hline
S1  & 89 & 91 & 77 & 75 \\
S2  & 67 & 72 & 76 & 66 \\
S3  & 72 & 77 & 69 & 70 \\
S4  & 56 & 60 & 55 & 61 \\
S5  & 92 & 98 & 89 & 86 \\
S6  & 83 & 88 & 90 & 84 \\
S7  & 45 & 60 & 55 & 48 \\
\hline
Average  & 72 & 78 & 73 & 70\\
\hline
\end{tabular}\\
\end{center}
\end{figure}

We can shift our data set by subtracting each column by its average value.  This makes it so that the average of each column in the matrix below is zero.  If $W$ represents the matrix of scores, the following Python command will subtract out the average.  Why does this work?
\begin{lstlisting}[style=python]
:X = W - sp.dot(sp.ones((7,1),dtype=sp.float_),sp.mean(W,axis=0).reshape(1,4));X
array([[ 17.,  13.,   4.,   5.],
       [ -5.,  -6.,   3.,  -4.],
       [  0.,  -1.,  -4.,   0.],
       [-16., -18., -18.,  -9.],
       [ 20.,  20.,  16.,  16.],
       [ 11.,  10.,  17.,  14.],
       [-27., -18., -18., -22.]])
\end{lstlisting}

\section*{Angles Between Vectors}

Inner products give information about lengths of vectors and angles between vectors.  Recall that the angle $\theta$ between two vectors is given by
\[
\cos{\theta} = \frac{\ipt{x}{y}}{\norm{x}\norm{y}}
\]
where the norm (or length) of a vector is $\norm{x} = \sqrt{\ipt{x}{x}}$.  Note that the usual inner product in $\mathbb{R}^n$ is given by
\[
\ipt{x}{y} = x^T y
\]
Alternatively, if we can first divide our vectors by their length (these are called unit vectors) and then take the inner product
\[
\cos{\theta} = \left\langle\frac{x}{\norm{x}},\frac{y}{\norm{y}}\right\rangle
\]
Hence, we can find the cosine of the angles between our data columns in $X$ by dividing each column by its length.  There are a couple of ways of doing this.  The following is an interesting way of doing it.  Be sure to explore why it works:
\begin{lstlisting}[style=python]
: Y = X / sp.dot(sp.ones((7,1),dtype=sp.float_),sp.sqrt(sp.diag(sp.dot(X.T,X)).T).reshape(1,4));Y
array([[ 0.39848615,  0.35329218,  0.11386819,  0.15371887],
       [-0.11720181, -0.16305793,  0.08540114, -0.12297509],
       [ 0.        , -0.02717632, -0.11386819,  0.        ],
       [-0.37504578, -0.48917378, -0.51240685, -0.27669396],
       [ 0.46880723,  0.54352643,  0.45547275,  0.49190037],
       [ 0.25784398,  0.27176321,  0.4839398 ,  0.43041282],
       [-0.63288976, -0.48917378, -0.51240685, -0.67636301]])
\end{lstlisting}
Hence we have:
\[
Y=
\begin{bmatrix}
0.3985 & 0.3533 & 0.1139 & 0.1537\\
-0.1172 & -0.1631 & 0.0854 & -0.1230\\
0 &-0.0272 &-0.1139 & 0\\
-0.3750 & -0.4892 & -0.5124 & -0.2767\\
0.4688 & 0.5435 & 0.4555 & 0.4919\\
0.2578 & 0.2718 & 0.4839 & 0.4304\\
-0.6329 & -0.4892 & -0.5124 & -0.6764
\end{bmatrix}
\]
Finally, we get the cosines of the angles between columns by computing $Y^T Y$.  In Python, that's just
\begin{lstlisting}[style=python]
: sp.dot(Y.T,Y)
array([[ 1.        ,  0.97782999,  0.89014869,  0.94908967],
       [ 0.97782999,  1.        ,  0.90978843,  0.92490144],
       [ 0.89014869,  0.90978843,  1.        ,  0.9276955 ],
       [ 0.94908967,  0.92490144,  0.9276955 ,  1.        ]])
\end{lstlisting}
This yields
\[
Y^T Y =
\begin{bmatrix}
1.0000 & 0.9778 & 0.8901 & 0.9491\\
0.9778 & 1.0000 & 0.9098 & 0.9249\\
0.8901 & 0.9098 & 1.0000 & 0.9277\\
0.9491 & 0.9249 & 0.9277 & 1.0000
\end{bmatrix}.
\]
Note that the $(j,k)$ entry of $Y^T Y$ corresponds to the cosine of the angle between the $j^{th}$ and $k^{th}$ columns.  We remark that the diagonals are always equal to one because the angle between a vector and itself is zero and the cosine of zero is one.

\section*{Correlation}

We remark that the cosine of the angle between two vectors is sometimes called the correlation coefficient.  Two columns are said to be
\begin{itemize}
\item Perfectly correlated if the cosine of the angle between them is one.
\item Positively correlated if the cosine of the angle between them is between zero and one.
\item Uncorrelated if the cosine of the angle between them is zero.
\item Negatively correlated if the cosine of the angle between them is between negative one and zero.
\item Perfectly anticorrelated if the cosine of the angle between them is negative one.
\end{itemize}

The notion of correlation is important in establishing the relationships between measurements.  For example, there is a high correlation between those who smoke and those who get lung cancer.  It's important to understand that the high correlation alone does not necessarily imply that smoking causes lung cancer, it only links them statistically.  This is the famous issue of causation versus correlation.  For example, there is a high correlation between crime rates and sales of ice cream.  This doesn't mean that ice cream causes crime or that increases in crime makes people want to eat more ice cream, but both rates do go up in the summer.

\begin{problem}
Download the file lab8.txt from the following link:
\url{http://www.math.byu.edu/~jeffh/teaching/m343h/lab8.txt}
You can load this datafile by typing
\li{lab8 = sp.genfromtxt("lab8.txt")}
The data is now available in the array \li{lab8} (also, make sure Python is working in the directory of lab8.txt).  \li{lab8} consists of two columns.  Find the correlation coefficient of the two columns.  Then plot the original data and see if the value that you got is reasonable.
\end{problem}

\section*{Least Squares}

It is well known that the displacement of a spring is proportional to the force acting upon it, that is, $F = k x$.  The proportionality constant $k$ is called Hooke's spring constant.  Consider a laboratory experiment where different loads are placed on a spring and the displacement is measured and recorded in the table below:
\vspace{5mm}\\
\begin{center}
\begin{tabular}{|c|c|}
	\hline
x & F \\
(cm) & (dyne)\\
\hline
1.04  & 3.11 \\
2.03  &  6.01\\
2.95  &  9.07\\
3.92  &  11.99\\
5.06  &  15.02\\
6.00  &  17.91\\
7.07  &  21.12\\
\hline
\end{tabular}
\end{center}
\vspace{5mm}
To find the spring constant $k$, we simply need to solve the following linear system
\[
\begin{pmatrix}
1.04\\
2.03\\
2.95\\
3.92\\
5.06\\
6.00\\
7.07\\
\end{pmatrix}
\begin{pmatrix}k\end{pmatrix} =
\begin{pmatrix}
3.11 \\
6.01\\
9.07\\
11.99\\
15.02\\
17.91\\
21.12\\
\end{pmatrix}.
\]
However, there is no solution to this system because it is overdetermined.  Instead, we seek the ``best'' $k$ that fits the data.  Least squares (which we mentioned in a previous exercise) allows us to find that ``best'' solution. We can find the least squares solution by computing the following in Python:
\begin{lstlisting}[style=python]
: A = sp.vstack([1.04,2.03,2.95,3.92,5.06,6.00,7.07])
: b = sp.vstack([3.11,6.01,9.07,11.99,15.02,17.91,21.12])
: k = sp.dot(sp.dot(la.inv(sp.dot(A.T,A)),A.T),b);k
: # Using the built in least squares function this is just: k=la.lstsq(A,b);k
array([[ 2.99568294]])
\end{lstlisting}
Hence, we find the spring constant to be $k = 2.9957$.  We plot the data against the best fit as follows:
\begin{figure}[h!]
\label{fig1}
\begin{center}
\includegraphics[scale = .6]{lab9_line_py}
\caption{The graph of the spring data together with its linear fit}
\label{Fig:SpringFit}
\end{center}
\end{figure}

\begin{lstlisting}[style=python]
: x0 = sp.linspace(0,8,100)
: y0 = k[0]*x0
: from matplotlib import pyplot as plt
: plt.plot(A,b,'*',x0,y0)
: plt.show()
\end{lstlisting}
See Figure \ref{Fig:SpringFit} to see how well the line fits the data.

\section*{General Line Fitting}

Suppose that we wish to fit a general line, that is $y=m x+b$, to the data set $\{(x_k,y_k)\}^n_{k=1}$.  Assume that the line does not cross through the origin, as in the previous example.  Then we seek both a slope and a $y$-intercept.  In this case, we set up the following linear system $A x = b$, or more precisely
\[
\begin{pmatrix}
x_1 & 1\\
x_2 & 1\\
x_3 & 1\\
\vdots & \vdots\\
x_n & 1
\end{pmatrix}
\begin{pmatrix}
m\\
b
\end{pmatrix}=
\begin{pmatrix}
y_1\\
y_2\\
y_3\\
\vdots\\
y_n
\end{pmatrix}.
\]
Note that $A$ has rank $2$ as long as not all of the $x_k$ values are the same.  Hence, the least squares solution will fit the best line for this data.

\begin{problem}
Download the file lab9.txt from the following link:
\url{http://www.math.byu.edu/~jeffh/teaching/m343h/lab9.txt}
You can load this datafile by typing
\li{lab9 = sp.genfromtxt("lab9.txt")}
Now the data is available in the array \li{lab9}.
This consists of two columns corresponding to the $x$ and $y$ values of a given data set.  Use least squares to find the slope and $y$-intercept that best fits the data.  Then plot the data points and the line on the same graph.  Finish off the problem with a discussion of what you've learned.
\end{problem}


In Bayesian statistics, we are interested in performing inference on the posterior distribution of a parameter, $\theta$.  This is calculated using Bayes rule
$$
\pi(\theta | x) = \frac{f(x | \theta) p(\theta)}{f(x)}
$$
where $f(x|\theta)$ is the likelihood function of the data, and $p(\theta)$ is the prior distribution on the parameter.  The term on the denominator, $f(x)=\int f(x)|\theta) p(\theta) d\theta$, is a normalising constant, such that $\pi$ is a probability distribution.  In general, and particularly in high dimensions, this normalising constant is difficult to calculate analytically, and so the posterior is only known up to proportionality.
$$
\pi(\theta | x) \propto f(x | \theta) p(\theta)
$$
This means we cannot easily make inferential statements about the parameter $\theta$.  To solve this problem, we use Markov chain Monte Carlo methods to draw samples from the posterior distribution, and use these samples to make inferences on the parameter.

\subsection{Markov Chain Monte Carlo}
Monte Carlo methods are a class of algorithms that replace difficult or impossible analytical probability calculations with numerical approximations.  Given a random variable $X$ on a probability space $(\Omega,\mathcal{F}, \pi)$ with distribution given by $\pi$ consider the problem of calculating

$$
\E_\pi[g(X)] = \int_\Omega g(x) d \pi
$$
If the distribution $\pi$ does not take a well-known form, then analytically solving this expectation is very difficult.  Instead, Monte Carlo integration uses the strong law of large numbers to approximate the integral.  If $X_n$ are a sequence of i.i.d. random variables distributed according to $\pi$, then
$$
\frac{1}{N} \sum_{n=1}^N g(X_n) \to \E_\pi[g(X)] \quad \text{a.s. as $N \to \infty$.}
$$
Hence we can approximate integrals by taking independent samples from the distribution $\pi$.  However, for an arbitrary distribution, it is not necessarily possible to find samples independently in an efficient way.  Markov chain Monte Carlo allows us to dispense with this assumption.  Rather than sampling independently, we can instead construct a Markov chain with $\pi$ as its invariant distribution.  This chain can then be used to generate dependent samples, which can be used for Monte Carlo integration provided the chain is ergodic with respect to $\pi$.

\begin{defn}[Ergodicity]
Let $T:\Omega \to \Omega$ be a probability-preserving transformation on a probability space $(\Omega,\mathcal{F}, \P)$.  Then we say $T$ is \emph{ergodic} if for every $F \in \mathcal{F}$ with $T^{-1}(F)=F$, either $\P(F)=0$ or $\P(F)=1$.
\end{defn}

Intuitively, this condition means that the process explores the whole space, without becoming stuck in a subregion.  The ergodic theorem then provides the theoretical justification that permits the use of dependent samples for Monte Carlo integration.

\begin{theorem}[Ergodic Theorem]\label{thm:ergodic}
 Let $f$ be measurable, $\E_\pi(|f|) < \infty$, and $T$ be an ergodic probability-preserving transformation. Then with probability 1:
 $$
\frac{1}{N} \sum_{n=1}^N f(T^n(x)) \to \E_\pi[f] \quad \text{a.s. as $N \to \infty$.}
$$
\end{theorem}
\noindent In other words, if $T$ is ergodic, time averages converge to space averages.  Hence, if we can construct a Markov chain which is ergodic with respect to a target measure $\pi$, it can be used to generate Monte Carlo samples.  One method of constructing such a Markov chain is by using Langevin dynamics.


\subsection{\(\sigma \)-algebra}

\subsubsection{Review of algebra on a set}

An algebra, \(\Sigma \), on set \(s\) is a set of subsets of \(s\) such that:

\begin{itemize}
\item Closed under intersection: If \(a\) and \(b\) are in \(\Sigma \) then \(a\land b\) must also be in \(\Sigma \)
\item \(\forall ab [(a \in \Sigma \land b \in \Sigma )\rightarrow (a\land b \in \Sigma)]\)
\item Closed under union: If \(a\) and \(b\) are in \(\Sigma \) then \(a\lor b\) must also be in \(\Sigma \).
\item \(\forall ab [(a \in \Sigma \land b \in \Sigma )\rightarrow (a\lor b \in \Sigma)]\)
\end{itemize}

If both of these are true, then the following is also true:

\begin{itemize}
\item Closed under complement: If \(a\) is in \(\Sigma \) then \(s \backslash a\) must also be in \(\Sigma \)
\end{itemize}

We also require that the null set (and therefore the original set, null's complement) is part of the algebra.

\subsubsection{\(\sigma \)-algebra}

A \(\sigma \)-algebra is an algebra with an additional condition:

All countable unions of sets in \(A\) are also in \(A\).

This adds a constraint. Consider the real numbers with an algebra of all finite sets.

This contains all finite subsets, and their complements. It does not contain \(\mathbb{N}\).

However a \(\sigma \)-algebra requires all countable unions to be including, and the natural numbers are a countable union.

The power set is a \(\sigma \)-algebra. All other \(\sigma \)-algebras are subsets of the power set.


\textbf{Solve the boundary value problem $u_{xx}+4u_{x}+e^xu=\sin{(8x)}$ numerically on $[-1,1]$ with boundary conditions $u(\pm1)=0$. To 10 digits of accuracy, what is $u(0)$?}
\newline

We can express the BVP as
\begin{align*}
D^2u+4Du+\text{diag}(e^x)u &= f,\\
\left[D^2+4D+\text{diag}(e^x)\right]u &= f,\\
Mu &= f.
\end{align*}
where $D$ is the Chebyshev differentiation matrix and $f$ is the right hand side of the equation. Hence, we can calculate the solution to our BVP as
\begin{align*}
u &= M^{-1}f.
\end{align*}

The solution, using $N+1=51$ points (odd number of points to have a node exactly at $x=0$), is plotted in the figure 1. In the figure 2 we plot the value of $u(x=0)$ for different values of $N$. We see how with just $10+$ nodes we achieve high accuracy. The value of $u(x=0)$ obtained is
\begin{align*}
u(0) =  0.0095978572.
\end{align*}
The number of nodes needed to achieve 10 digits of accuracy was $N+1 = 23$.
\begin{figure}[H]
\centering
\includegraphics[scale=0.9]{P1_b.png}\caption{Solution to the BVP using Chebyshev matrices and 51 nodes.}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[scale=0.9]{P1_a.png}\caption{Value of $u(0)$ for different number of nodes.}
\end{figure}

\subsection*{Matlab code for this problem}

\begin{verbatim}
%% Homework 4, Problem 1 - Francisco Castillo
clear all; close all; clc;
labelfontsize = 14;

NN = 4:2:50;
for j=1:length(NN)
    N=NN(j);
    [D,x] = cheb(N); % D:(N+1)x(N+1), x:(N+1)x1
    D2 = D^2;
    D = D(2:N,2:N);
    D2 = D2(2:N,2:N);       % Zero Dirichlet BCs
    f = sin(8*x(2:N));
    E = diag(exp(x(2:N)));
    u = (D2+4*D+E)\f;
    u = [0;u;0];
    u0(j) = u(N/2+1);
end

figure
plot(NN,u0,'r*')
grid on
xlabel('$N$','fontsize',labelfontsize,'interpreter','latex')
ylabel('$u(0)$','fontsize',labelfontsize,'interpreter','latex')
set(get(gca,'ylabel'),'rotation',0)
txt = 'Latex/FIGURES/P1_a';
saveas(gcf,txt,'png')

xx = [x(end):.01:x(1)]';
uu = polyval(polyfit(x,u,N),xx);

figure
plot(x,u,'b*')
hold on
plot(xx,uu,'b')
grid on
xlabel('$x$','fontsize',labelfontsize,'interpreter','latex')
ylabel('$u(x)$','fontsize',labelfontsize,'interpreter','latex')
set(get(gca,'ylabel'),'rotation',0)
txt = 'Latex/FIGURES/P1_b';
saveas(gcf,txt,'png')
\end{verbatim}


\textbf{Derive Newtons iteration for this function. Show that the iteration is well-defined so long as $x_k\neq 1$ and that the convergence rate is expected to be similar to that of the bisection method (and certainly not quadratic).}

\vspace{0.3in}
Newton's method for finding successively better approximations to the roots of a real-valued function is the following,
\begin{align*}
x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}.
\end{align*}
Given the function
\begin{align*}
f(x)=(x-1)^2e^x,
\end{align*}
and its derivative
\begin{align*}
f'(x)=(x^2-1)e^x,
\end{align*}
we can obtain the iteration for this problem
\begin{align*}
x_{n+1}=x_n-\frac{(x_n-1)^2e^{x_n}}{(x_n^2-1)e^{x_n}}=x_n-\frac{(x_n-1)}{(x_n+1)}.
\end{align*}
Note that this is not defined for $x_n=-1$. Clearly the root of the function is $x^*=1$. To find the rate of convergence let $\epsilon_n=x_n-x^*=x_n-1$, and $g(x_n)=\frac{f(x_n)}{f'(x_n)}$. Then, rewriting the first equation in this terms we get
\begin{align*}
\epsilon_{n+1}=\epsilon_n-g(x_n).
\end{align*}
Further,
\begin{align*}
g(x_n)=g(\epsilon_n+1)=g(1)+g'(1)\epsilon_n+\frac{\epsilon_n^2}{2}g''(\xi),
\end{align*}
by Taylor expansion. The values of $g$ and $g'$ are known,
\begin{align*}
g(1)=\frac{f(1)}{f'(1)}=0,~~~~~~g'(x)=\frac{2}{(x+1)^2}\Rightarrow g'(1)=\frac{1}{2}.
\end{align*}
Hence,
\begin{align*}
g(x_n)=\frac{1}{2}\epsilon_n+\frac{\epsilon_n^2}{2}g''(\xi),
\end{align*}
and
\begin{align*}
\lim_{\epsilon\to 0}\frac{\epsilon_{n+1}}{\epsilon_{n}}=\frac{1}{2}.
\end{align*}
Note that the convergence rate is linear and equal to the one of the bisection method. It differes from the quadratic convergence rate talked in class because this function and its derivative have the same root $x^*=1$. For the convergence rate to be quadratic, the root of the function should not be a root of the derivative.

\vspace{0.3in}

\textbf{Implement Newtons method and observe its performance starting from $x_0 = 2$.}

\vspace{0.3in}

After implementing the method in \textsl{Matlab} the root $x^*=1$ was found in 42 iterations, with a time of the order of $0.009~s$. In the following figure we can see the function given and the roots calculated by the Newton's method until convergence.

\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{prob1roots.eps}}
\caption{Newton's method performance.}
\end{figure}

\vspace{0.3in}

\textbf{How easy would it be to apply the bisection method? Explain.}

\vspace{0.3in}

For this case the bisection method would not have worked since it relies on a change of sign at both sides of the root, and this is not the case with this function which is nonnegative for all values of $x$.

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 1
xx = linspace(-3,3,1e4);
f = @(x) (x - 1).^2.*exp(x);
df = @(x) (x.^2 - 1).*exp(x);

tic
[x,niter,conv]=newton(f,df,2,1e-12,100)
time=toc
% To obtain the roots, modified newton function
[x,niter,conv]=newton_vector(f,df,2,1e-12,100);

figure
plot(xx,f(xx),'linewidth',1.5)
hold on
plot(x,0,'r*')
grid on
axis([-2 2.5 -1 7.5])
set(gca,'fontsize',axisfontsize)
xlabel('$x$','interpreter','latex','fontsize',labelfontsize)
ylabel('$y$','interpreter','latex','fontsize',labelfontsize)
saveas(gcf,'Latex/FIGURES/prob2roots','epsc')
\end{verbatim}


\vspace{-4mm}

\subsection{Projections}
\label{vectors:projections}

\vspace{-2mm}

Dot products are used for computing projections.
Assume you're given two vectors $\vec{u}$ and $\vec{n}$ and you want to find the component
of $\vec{u}$ that points in the $\vec{n}$ direction.
The following formula based on the dot product will give you the answer:
\[
 \Pi_{\vec{n}}( \vec{u} ) \equiv \frac{  \vec{u} \cdot \vec{n}  }{ \| \vec{n} \|^2 } \vec{n}.
\]

\vspace{-2mm}

\noindent
This is how to implement this formula in \texttt{SymPy}:
\small
\begin{verbatimtab}
>>> u = Matrix([4,5,6])
>>> n = Matrix([1,1,1])
>>> (u.dot(n) / n.norm()**2)*n
[5, 5, 5]                      # projection of v in the n dir
\end{verbatimtab}
\normalsize

\noindent
In the case where the direction vector $\hat{n}$ is of unit length $\|\hat{n}\| = 1$,
the projection formula simplifies to $\Pi_{\hat{n}}( \vec{u} ) \equiv (\vec{u}\cdot\hat{n})\hat{n}$.

Consider now the plane $P$ defined by $(1,1,1)\cdot[(x,y,z)-(0,0,0)]=0$.
A plane is a two dimensional subspace of $\mathbb{R}^3$.
We can decompose any vector $\vec{u} \in \mathbb{R}^3$ into two parts $\vec{u}=\vec{v} + \vec{w}$
such that $\vec{v}$ lies inside the plane and $\vec{w}$ is perpendicular to the plane (parallel to $\vec{n}=(1,1,1)$).

To obtain the perpendicular-to-$P$ component of $\vec{u}$,
compute the projection of $\vec{u}$ in the direction $\vec{n}$:
\small
\begin{verbatimtab}
>>> w = (u.dot(n) / n.norm()**2)*n
[5, 5, 5]
\end{verbatimtab}
\normalsize

\noindent
To obtain the in-the-plane-$P$ component of $\vec{u}$,
start with $\vec{u}$ and subtract the perpendicular-to-$P$ part:
\small
\begin{verbatimtab}
>>> v = u - (u.dot(n)/n.norm()**2)*n      # same as u - w
[ -1, 0, 1]
\end{verbatimtab}
\normalsize

\noindent
You should check on your own that $\vec{v}+\vec{w}=\vec{u}$ as claimed.

\vspace{-5mm}


\subsection{Cardinality}

\subsubsection{Cardinality of cartesian product}

What about the cardinality of Cartesian products? So if we have sets:

\(\{1,2,3\}\)

\(\{a,b\}\)

We can have the Cartesian product set:

\(\{(1,a),(2,a),(3,a),(1,b),(2,b),(3,b)\}\)

We can see that:

\(|A.B|=|A|.|B|\)
\subsubsection{Cardinality of union and intersection}

\(|A\lor B| = |A|+|B|-|A\land B|\)
\subsubsection{Cardinality of powerset}

\(|P(s)|=2^{|s|}\)

\subsubsection{Cardinality of complement}

\(|a \setminus b|=|a|-|a\land b|\)

\subsubsection{Cardinality of even/odd natural numbers}

What about the cardinality of even numbers? Well, we can define a bijective function between each:

\(f(n)=2n\)

Similarly for odd numbers:

\(f(n)=2n+1\)

So these both have cardinality \(\aleph_0\).


\subsection{Hyperbolic trigonometric functions}
\label{basics:hyperbolic_trigonometric_functions}

The hyperbolic sine and cosine in \texttt{SymPy} are denoted \texttt{sinh} and \texttt{cosh} respectively
and \texttt{SymPy} is smart enough to recognize them when simplifying expressions:

\small
\begin{verbatimtab}
>>> simplify( (exp(x)+exp(-x))/2 )
cosh(x)
>>> simplify( (exp(x)-exp(-x))/2 )
sinh(x)
\end{verbatimtab}
\normalsize
Recall that $x=\cosh(\mu)$ and $y=\sinh(\mu)$ are defined as $x$ and $y$ coordinates of a point on the
the hyperbola with equation $x^2 - y^2 = 1$ and therefore satisfy the identity $\cosh^2 x - \sinh^2 x =1$:

\small
\begin{verbatimtab}
>>> simplify( cosh(x)**2 - sinh(x)**2 )
1
\end{verbatimtab}
\normalsize


\section{Matrix examples}
\subsection{Geometric transformations}
\textbullet \textit{Scaling}: $y = Ax$ with $A = aI$ stretches a vector by the factor $|a|$ (or shrinks it when $|a| < 1$), and it flips the vector (reverses its direction) if $a < 0$\\
\textbullet \textit{Dilation}: $y = Dx$, where D is a diagonal matrix, $D = diag(d1,d2)$. Stretches the vector x by different factors along the two different axes. (Or shrinks, if $|d_i| < 1$, and flips, if $di < 0$.)\\
\textbullet \textit{Rotation Matrix (counter clockwise)}:
$
y =
\begin{bmatrix}
    cos \theta & -sin \theta \\
    sin \theta & cos \theta
\end{bmatrix}x
$\\
\textbullet \textit{Reflection}
Suppose that y is the vector obtained by reflecting x through the line that passes through the origin, inclined $\theta$ radians with respect to horizontal.\\
$
y =
\begin{bmatrix}
    cos (2\theta) & sin(2\theta) \\
    sin (2\theta) & -cos (2\theta)
\end{bmatrix}x
$\\
\textbullet \textit{Projection into a line}
Projection of point x onto a set is the point in the set that is closest to x.\\
$
y =
\begin{bmatrix}
    (1/2)(1 + cos (2\theta)) & (1/2)sin(2\theta) \\
    (1/2)sin (2\theta) & (1/2)(1 - cos (2\theta))
\end{bmatrix}x
$
\subsection{Selectors}
An $m \times n$ selector matrix A is one in which each row is a unit vector (transposed):
\[
\begin{bmatrix}
    e^T_{k_1}\\
    .\\
    .\\
    e^T_{k_m}\\
\end{bmatrix}
\]
When it multiplies a vector, it simply copies the $k_i$th entry of x into the $i$th entry of $y = Ax$:\\
$y = (x_{k_1},x_{k_2},...,x_{k_m})$\\
\textbf{r:s matrix slicing}\\
$A=[0_{m\times(r-1)} I_{m\times m} 0_{m \times (n-s)}]$\\
where $m=s-r+1$
\subsection{Incidence matrix}
\textbf{Directed graph}: A \textit{directed graph} consists of a set of \textit{vertices} (or nodes), labeled 1,...,n, and a set of \textit{directed edges} (or branches), labeled 1,...,m. \\
$
A_{ij} = \left\{
  \begin{array}{@{}ll@{}}
    1, & \text{edge j points to node i} \\
    -1, & \text{edge j points from node i}\\
    0, & \text{otherwise}
  \end{array}\right.
$\\
\subsection{Convolution}
The convolution of an n-vector a and an m-vector b is the (n + m - 1)-vector denoted c = a * b\\
$c_k = \sum_{i+j=k+1} a_ib_j, k=1,..,n+m-1$\\
\textbf{Properties of convolution}\\
\textbullet symmetric: $a * b = b * a$\\
\textbullet associative: $(a*b)*c = a*(b*c)$\\
\textbullet $a*b = 0$ implies that either $a = 0$ or $b = 0$\\
\textbullet A basic property is that for fixed a, the convolution a * b is a linear function of b; and for fixed b, it is a linear function of a, $a * b = T (b)a = T (a)b$ where where T(b) is the (n + m - 1) × n matrix with entries\\
$
T(b)_{ij} = \left\{
  \begin{array}{@{}ll@{}}
    b_{i-j+1}, & 1\leq i-j+1 \leq m\\
    0, & \text{otherwise}
  \end{array}\right.
$\\
\textbf{Complexity of convolution}\\
\textbullet $c= a * b$: 2mn flops\\
\textbullet $T(a)b or T(b)a$: 2mn flops\\
\textbullet Convolution could be calculated faster using \textit{fast Fourier transform (FFT)} : $5(m + n) log_2(m + n) flops$\\


\section{Method of Moments}

Let $X_1, \ldots , X_ n \stackrel{iid}{\sim } \mathbf{P}_{\theta ^*}$ associated with model $(\mathbb {E}, \{ \mathbf{P}_{\theta }\} _{\theta \in \Theta })$, with $\mathbb {E} \subseteq \mathbb {R}$ and $\Theta \subseteq \mathbb {R^d}$, for some d $\geq 1$

Population moments:\\

$m_k(\theta) = \mathbb{E}_{\theta}[X^k_1],  1 \leq k \leq d$\\

Empirical moments:\\

$\widehat{m_k}(\theta) = \bar{X_n^k} = \frac{1}{n} \sum _{i=1}^{n}X_i^k$

Convergence of empirical moments:\\

$\widehat{m_k} \xrightarrow[n \rightarrow \infty]{P, a.s.} m_k$\\

$(\widehat{m_1}, \ldots, \widehat{m_d}) \xrightarrow[n \rightarrow \infty]{P, a.s.} (m_1, \ldots, m_d)$

MOM Estimator $M$ is a map from the parameters of a model to the moments of its distribution. This map is invertible, (ie. it results into a system of equations that can be solved for the true parameter vector $\theta^{*}$). Find the moments (as many as parameters), set up system of equations, solve for parameters, use empirical moments to estimate.

$\displaystyle  \psi : \Theta \displaystyle \to \mathbb {R}^ d$\\

$\displaystyle \theta \displaystyle \mapsto (m_1(\theta ), m_2(\theta ), \ldots , m_ d(\theta ))$\\

$M ^{-1}(m_1(\theta ^*), m_2(\theta ^*), \ldots , m_ d(\theta ^*))$

The MOM estimator uses the empirical moments:\\

$M ^{-1}\left( \frac{1}{n} \sum _{i = 1}^ n X_ i, \frac{1}{n} \sum _{i = 1}^ n X_ i^2, \ldots , \frac{1}{n} \sum _{i = 1}^ n X_ i^ d \right)$\\

Assuming $M^{-1}$ is continuously differentiable at $M(0)$, the asymptotical variance of the MOM estimator is:\\\\

$\sqrt(n)(\widehat{\theta_n^{MM}} - \theta) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\Gamma)$\\

where,

$\Gamma (\theta )=\left[\frac{\partial M^{-1}}{\partial \theta } (M(\theta ))\right]^{T} \Sigma (\theta )\left[\frac{\partial M^{-1}}{\partial \theta } (M(\theta ))\right]$

$\Gamma(\theta) = \nabla_{\theta}(M^{-1})^{T} \Sigma \nabla_{\theta}(M^{-1}) $

$\Sigma_{\theta}$ is the covariance matrix of the random vector of the moments $(X_1^1,X_1^2 \ldots, X_1^d)$.


\chapter{The Cartesian Product}
The \emph{Cartesian product} $\times$ is a binary operation on sets which produces a new set containing all ordered pairs where the first element comes from the left-hand set and the second element comes from the right-hand set. If $A = \{a,b,c,\dots\}$ and $\Gamma = \{\alpha,\beta,\gamma,\dots\}$, then $A \times \Gamma = \{(a,\alpha), (a,\beta), \dots, (b,\alpha), \dots\}$.
Specifically, the Cartesian product is defined as
\[ A \times B = \{(a,b) : a \in A, b \in B\}. \]
While we call it a binary operation, we can have an arbitrary number of sets upon which it can act; consider the generalization to $n$ sets
\[ A \times B \times C \times \cdots = \{(a,b,c,\dots) : a \in A, b \in B, c \in C, \dots\}, \]
which produces an $n$-tuple.
We use the Cartesian product to describe the domain and range of new binary functions that we create. If we imagine a new binary operation $\odot$ which acts upon sets $A$ and $B$ and maps to $C$, we could define it as
\[ \odot : A \times B \to C. \]
For the same of brevity, we often use an exponent to represent the number of times a Cartesian product has been taken over the same vector space, so
\[ A \times A \times A = A^3 \]
and
\[A  \times A \times B \times B \times B = A^2 \times B^3. \]


\section{Decision Tree Learning (20 points)}

In this problem, we want to construct a decision tree to find out if a person will enjoy beer.

\paragraph{Definitions.} Let there be $k$ binary-valued attributes in the data.

We pick an attribute that maximizes the gain at each node:
\begin{equation}\label{G}
%	G = \max [I(D) - (I(D_L) + I(D_R))];
	G = I(D) - (I(D_L) + I(D_R));
\end{equation}
where $D$ is the given dataset, and $D_L$ and $D_R$ are the sets on left and right hand-side branches after division. Ties may be broken arbitrarily.

There are three commonly used impurity measures used in binary decision trees: Entropy, Gini index, and Classification Error. In this problem, we use Gini index and define $I(D)$ as follows\footnote{As an example, if $D$ has 10 items, with 4 positive items (\emph{i.e.} 4 people who enjoy beer), and 6 negative items (\emph{i.e.} 6 who do not), we have $I(D) = 10 \times (1 - (0.16 + 0.36))$.}:
\[
	I(D) = |D| \times \left(1-\sum_i p_i^2\right),
\]
where:
\begin{itemize}
	\item $|D|$ is the number of items in $D$;
	\item $1 - \sum_i p_i^2$ is the gini index;
	\item $p_i$ is the probability distribution of the items in $D$, or in other words, $p_i$ is the fraction of items that take value $i\in\{+,-\}$. Put differently, $p_+$ is the fraction of positive items and $p_-$ is the fraction of negative items in $D$.
\end{itemize}

Note that this intuitively has the feel that the more evenly-distributed the numbers are, the lower the $\sum_i p_i^2$, and the larger the impurity.

\subquestion{(a) [10 Points]}
Let $k = 3$. We have three binary attributes that we could use: ``likes wine'', ``likes running'' and ``likes pizza''.  Suppose the following:
\begin{itemize}
	\item There are 100 people in sample set, 40 of whom like beer and 60 who don't.
	\item Out of the 100 people, 50 like wine; out of those 50 people who like wine, 20 like beer.
	\item Out of the 100 people, 30 like running; out of those 30 people who like running, 20 like beer.
	\item Out of the 100 people, 80 like pizza; out of those 80 people who like pizza, 30 like beer.
	\end{itemize}
\task{What are the values of $G$ (defined in Equation~\ref{G}) for wine, running and pizza attributes? Which attribute would you use to split the data at the root if you were to maximize the gain $G$ using the gini index metric defined above?}

\subquestion{(b) [10 Points]}
Let's consider the following example:
\begin{itemize}
\item There are $100$ attributes with binary values $a_1, a_2, a_3, \ldots, a_{100}$.
\item Let there be one example corresponding to each possible assignment of 0's and 1's to the values $a_1, a_2, a_3 \ldots, a_{100}$. (Note that this gives us $2^{100}$ training examples.)
\item Let the values taken by the target variable $y$ depend on the values of $a_1$ for $99 \%$ of the datapoints. More specifically, of all the datapoints where $a_1=1$, let $99 \%$ of them are labeled $+$. Similarly, of all the datapoints where $a_1 = 0$, let $99 \%$ of them are labeled with $-$.  (Assume that the values taken by $y$ depend on $a_2, a_3, \dots, a_{100}$ for fewer than $99\%$ of the datapoints.)
\item Assume that we build a complete binary decision tree (\emph{i.e.}, we use values of all attributes).
\end{itemize}

\task{Explain what the decision tree will look like. (A one line explanation will suffice.) Also, in 2-3 sentences, identify what the desired decision tree for this situation should look like to avoid overfitting, and why.(The desired decision tree isn't necessarily a complete binary decision tree)}

\subsection*{What to submit}
\begin{enumerate}[(i)]
	\item Values of $G$ for wine, running and pizza attributes. [part (a)]
	\item The attribute you would use for splitting the data at the root. [part (a)]
	\item Explain what the decision tree looks like in the described setting. Explain how a decision tree should look like to avoid overfitting. (1-2 lines each) [part (b)]
\end{enumerate}


\section{ Linear regression }
\subsection{Stochastic gradient descendent}

Stochastic gradient descendent (SGD) is a sequential version of the gradient descent. Instead of considering the full batch gradient on all $N$
training data points, we consider a stochastic version of the gradient. First, pick a training data point $(x_n, y_n)$ uniformly random
(hence the name 'stochastic') and consider only the error on that data point.  \cite{LFD}

The gradient of this single data point's error is used for the weight update in exactly the same that the gradient was used in batch gradient descent.

Another variants are the \textbf{mini-batch gradient descent} and the \textbf{batch gradient descendent}, the differences among them are the size of the batch: one for the pure stochastic gradient descendent, between 32 or 64 for the mini-batch variation and more than that for the batch gradient descendent.

\subsection{Pseudo - inverse algorithm}
Pseudo inverse algorithm also known as \textbf{linear regression algorithm} or \textbf{ordinary least squares}(OLS) is based on minimizing the squared error between the
projection matrix $h(x) = w^T x$ and $y$, the target vector, where $x \in \mathbb R^{N \times (d+1)}$ is the feature matrix and
$N \in \mathbb N$ the training data size.

\begin{equation*}
  E_{in}(w) = \frac{1}{N} \sum_{n=1}{N} (w^T x_n - y)^2
  = \frac{1}{N} \| Xw -y \|^2
\end{equation*}

Where $\|.\|$ is the Euclidean norm of a vector.

Since $E_{in}(w)$ is differentiable we can use standard matrix calculus to find the $w$ that minimizes  $E_{in}$ with respect to w:

$$\nabla E_{in}(w) = \frac{2}{N}(X^TXw - X^T y) =0$$
Finally to get $\nabla  E_{in}(w)$ to be $0$, we should find a $w$ that satisfies

$$X^TXw = X y$$

If $X^TX$ is invertible, $w = X^\dagger y$ where $X^\dagger = (X^T X)^{-1}$ is the \texttt{pseudo-inverse} of $X$. The resulted $w$ is the unique optimal solution that minimizes $E_{in}$.  Otherwise a pseudo-inverse can still be defined, but the solution will not be unique.

In practice, $X^TX$ is invertible in most of the cases since $N$ is often much more bigger than $d+1$, so there will likely be $d+1$ linearly independent vector $x_n$.

\subsection{Exercise 1}

Estimate a linear regression model from the data provided by the
feature vectors (Average intensity, Symmetry)
using both the pseudo-inverse algorithm and the Stochastic Gradient Descent (SGD).
The labels will be $\{-1,1\}$,
one for each feature vector of each number.
Draw the solutions obtained together with the data used in the fitting.
Assess the goodness of the result using $E_{in}$ and $E_{out}$
(for Eout calculate the predictions using the data from the test file).

\subsubsection{Error}

As we have said the error is the mean squared error:
$$E_{out}(h) = \mathbb E [(h(x) -y)^2]$$
 $$E_{in}(w)  = \frac{1}{N} \| Xw -y \|^2$$

A direct implementation is

\begin{minted}{python}
  def Error(x,y,w):
    '''quadratic error
    INPUT
    x: input data matrix
    y: target vector
    w:  vector to

    OUTPUT
    quadratic error >= 0
    '''
    error_times_n = np.linalg.norm(x.dot(w) - y.reshape(-1,1))**2

    return error_times_n/len(x)

  \end{minted}

  For the euclidean norm we have used \texttt{np.linalg.norm} \cite{norm} numpy function.

  The gradient computation is direct too:

  $$\nabla E_{in}(w) = \frac{2}{N}(X^TXw - X^T y)= \frac{2}{N}(X^T(Xw -  y))$$

\begin{minted}{python}
  def dError(x,y,w):
  ''' gradient
  OUTPUT
  column vector
    '''

    return (2/len(x)*(x.T.dot(x.dot(w) - y.reshape(-1,1))))
  \end{minted}

  \subsubsection{Interpretation of the mean squared error, E}
  The mean squared error function $E: \mathbb R ^d \longrightarrow \mathbb R^+_0$ measures the average of the squared difference between the estimated values and the actual value\cite{MSE}. Hence, the nearer to zero, the better.

\subsubsection{Pseudo-inverse algorithm}

As we have described in pseudo inverse introduction, firstly we need to compute the pseudo-inverse. For that we have use  \texttt{np.linalg.pinv} \cite{pseudo-inverse}function from numpy library.

\begin{minted}{python}
  def pseudoInverseMatrix ( X ):
    '''
    INPUT
    X: is a matrix (must be a np.array) to use transpose and dot method
    OUTPUT
    hat matrix
    '''

    '''
    #S =( X^TX ) ^{-1}
    simetric_inverse = np.linalg.inv( X.T.dot(X) )

    # S X^T = ( X^TX ) ^{-1} X^T
    return simetric_inverse.dot(X.T)
    '''
    return np.linalg.pinv(X)
\end{minted}

Finally we have to compute $w = X^\dagger y$

\begin{minted}{python}
  def pseudoInverse(X, Y):
    '''
    INPUT
    X is the feature matrix
    Y is the target vector (y_1, ..., y_m)

    OUTPUT:
    w: weight vector
    '''
    X_pseudo_inverse = pseudoInverseMatrix ( X )
    Y_transposed = Y.reshape(-1, 1)

    w = X_pseudo_inverse.dot( Y_transposed)

    return w

  \end{minted}

  \subsubsection{Pseudo-inverse linear regression model}

  After computing the algorithm we obtain:

\begin{verbatim}
___ Goodness of the Pseudo-inverse fit ___

  Ein:   0.07918658628900395
  Eout:  0.1309538372005258

Evaluating output training data set
Input size:  1561
Bad negatives : 7
Bad positives : 3
Accuracy rate : 99.35938500960923 %

Evaluating output test data set
Input size:  424
Bad negatives : 1
Bad positives : 7
Accuracy rate : 98.11320754716981 %

\end{verbatim}

  Which means that the $w$ computed by our pseudo-inverse algorithm by the training data set has a $E_{in}(w) = 0.079$ and by the test data $E_{out}(w) = 0.131$, for our experiment it is a good fit, because it is closed enough to zero. In addition, if we evaluate the data classification,
  from $1561$ training data it only misclassify $10$ points, whereof 7 was truly positives. The accuracy rate $(\frac{\text{good classified data}}{\text{data set size}})$
  is $99.358 \%$, which continues being really good.

  Initially, we could think that it classifies the positives values better, but if we analyse the output from test data set, there are more negatives values misclassified, so we cannot establish any relation. Moreover here the accuracy rate is $98.113 \%$

  Finally, a graphic representation for the solutions is

  \includegraphics[width=\linewidth]{2_1_pseudo_inverse.png}

  Something really import about this method is that it is equal or better than the gradient descent, therefore in the next experiments
  we are going to compare the results which it.

  \subsubsection{How have we plotted the regression line.}

  Firstable, to draw a line we need two points.

  Therefore we use the sign um to classify the numbers, we are going to find two points in $\mathbb R^2$ that their estimation is zero, which means that they are in the middle of classification (the regression line).

  The obtained weight vector $w^T = (w_1,w_2,w_3)$, means that for a point $(x,y) \in \mathbb R^2$
  its estimation is $h(x,y) = (1,x,y) w = w_1 + w_2x + w_3y$.

  To calculate the two points we are going to equalize $h(x,y)=0$ and from the infinity solutions we would compute two:

  if $x=0$ then $y = \frac{- w_1}{w_3}$, and if  $x=1$ then $y = \frac{- w_1 - w_2}{w_3}.$

  The related code is

  \begin{minted}{python}
    # regression line
    # x= 0
    symmetry_for_cero_intensity = -w[0]/w[2]

    #  x = 1, 0 = w0 + w1 * w2 * x2
    # then y = (-w0 - w1) /w2
    symmetry_for_one_intensity= (-w[0] - w[1])/w[2]

    # plotting order
    plt.plot([0, 1],
             [symmetry_for_cero_intensity,symmetry_for_one_intensity],
             'k-',
             label=(title+ ' regression'))

  \end{minted}

  \subsubsection{Stochastic gradient descendent}

  Based on the description of the algorithm, we have done two implementations:

  This one is strictly based on the classroom's slides, however depending on the \texttt{batch\_size} the exact number of iterations will be
$$\texttt{max\_iter} \times \lfloor \frac{ \texttt{(len(y))}}{\texttt{ batch\_size}} \rfloor$$

  \begin{minted}{python}
def sgd(x,y, eta = 0.01, max_iter = 1000, batch_size = 32, error=10**(-10)):
        '''
        Stochastic gradient descent
        INPUT
        x: data set
        y: target vector
        eta: learning rate
        max_iter

        OUTPUT
        w: weight vector
        '''

        #initialize data
        w = np.zeros((x.shape[1], 1), np.float64)
        n_iterations = 0

        len_x = len(x)
        x_index = np.arange( len_x )
        batch_start = 0
        w_error = Error(x,y,w)

        while n_iterations < max_iter and w_error > error :

                #shuffle and split the same into a sequence of mini-batches
                np.random.shuffle(x_index)
                for batch_start in range(0,  len_x, batch_size):
                        iter_index = x_index[ batch_start : batch_start + batch_size]

                        w = w - eta* dError(x[iter_index, :], y[iter_index], w)

                n_iterations += 1
                w_error = Error(x,y,w)

        return w
      \end{minted}

      In order to control exactly the numbers of iterations we are going to use this function:

      \begin{minted}{python}
        def sgd_exact_number_iter(x,y, eta = 0.01,
        max_iter = 1000, batch_size = 32, error = 10**(-10)):
        '''
        Stochastic gradient descent
        INPUT
        x: data set
        y: target vector
        eta: learning rate
        max_iter
        OUTPUT
        w: weight vector
        '''
        #initialize data
        w = np.zeros((x.shape[1], 1), np.float64)

        n_iterations = 0
        batch_start = 0
        len_x = len(x)

        x_index = np.arange( len_x )
        w_error = Error(x,y,w)

        while n_iterations < max_iter and w_error > error:
                #shuffle and split the same into a sequence of mini-batches
                if batch_start == 0:
                        x_index = np.random.permutation(x_index)
                iter_index = x_index[ batch_start : batch_start + batch_size]

                w = w - eta* dError(x[iter_index, :], y[iter_index], w)

                n_iterations += 1

                batch_start += batch_size
                if batch_start >= len_x: # if end, restart
                        batch_start = 0

                w_error = Error(x,y,w)

        return w

      \end{minted}

      Due to the fact that this is a stochastic method and the gradient's descent does not reduce the error in every step,  there are other variations, for example we can save and return only the $w$ found which has the less error.

       \begin{minted}{python}
         def sgd_save_w(x,y, eta = 0.01, max_iter = 1000,
                                   batch_size = 32, error = 10**(-10)):
        '''
        Stochastic gradient descent
        INPUT
        x: data set
        y: target vector
        eta: learning rate
        max_iter
        OUTPUT
        w: weight vector
        '''
        #initialize data
        w = np.zeros((x.shape[1], 1), np.float64)

        n_iterations = 0
        batch_start = 0
        len_x = len(x)

        x_index = np.arange( len_x )
        w_error = Error(x,y,w)

        #IMPROVEMENT
        best_error = w_error
        best_w = w

        while n_iterations < max_iter and w_error > error:
                #shuffle and split the same into a sequence of mini-batches
                if batch_start == 0:
                        x_index = np.random.permutation(x_index)
                iter_index = x_index[ batch_start : batch_start + batch_size]

                w = w - eta* dError(x[iter_index, :], y[iter_index], w)

                n_iterations += 1

                batch_start += batch_size
                if batch_start >= len_x: # if end, restart
                        batch_start = 0

                w_error = Error(x,y,w)

                # IMPROVEMENT
                if w_error < best_error:
                        best_w = w
                        best_error = w_error

        return best_w

      \end{minted}

    This algorithm is interesting because it returns the best $w$ found and has (in order) the same computational cost.

    \paragraph{Initial point}

    Another consideration is that as we know, it is really important the chosen of the initial point in gradient descendent. However, due to the fact that we are working with the mean quadratic error, we know that only exists  one global minimum, so here the relevance of the initial point is to reduce iterations. Therefore we theoretically do not have more information,  we have chosen the $w_0 = 0 \in \mathbb R^d.$

    \paragraph{The experiment result}  We are going to execute the \texttt{sgd} algorithm (our version)  with $\eta = 0.01$ batch sizes $1,32, 200 \text{ and } \texttt{len(y)}$ and the numbers of steps $50 \text{ and } 300$.

\begin{table}[H]
\begin{tabular}{|c|c|c|c|c|c|}
\hline
   Batch size &  Iterations & $E_{in}$      & $E_{out}$                  & Training accuracy rate $(\%)$   & Test accuracy rate $(\%)$   \\ \hline
  1          &     50     &     0.0798    & 0.131                      &        99.423           & 98.349    \\ \hline
 32          &     50     &   0.081       &      0.133                 & 99.295            &   98.113       \\ \hline
200          &         50 &     0.082     &    0.135                   &      99.424          & 98.113       \\ \hline
 1561        & 50         & 0.404         &  0.428                      &   99.167        & 95.28            \\ \hline
\end{tabular}
\end{table}

As we can see, as bigger is the batch size the worse is the error.
Something really interesting is that we are minimizing based on the quadratic error, but it does not mean we are minimizing the accuracy error ( there is a correlation but not a coincidence. A good example of that is if we compare bath's size 200 with bath's size 32.
As we increase the numbers of iterations the general error decrease (see 300 steps).

Another interesting observation is that for batch's size one,   200 iterations is worse than 50, this is because we are oscillating over the solution. The algorithm we have described which saves the best $w$, \texttt{sgd\_save\_w} would solve this problem.

\includegraphics[angle=90,width=\linewidth]{2_1_comparatives_50.png}

\includegraphics[angle=90,origin=c,width=\linewidth]{2_1_comparatives_300.png}

   \paragraph{50 steps}
    \begin{verbatim}

___ Goodness of the Stochastic Gradient Descendt (SGD) fit ___

	SGD, batch size 1
Ein:  0.07981803904587495
Eout:  0.13188855705205335

Evaluating output training data set
For w^T = [[-1.15690746 -1.20909308 -0.50379833]]
Input size:  1561
Bad negatives : 6
Bad positives : 3
Accuracy rate : 99.4234465086483 %

Evaluating output test data set
For w^T = [[-1.15690746 -1.20909308 -0.50379833]]
Input size:  424
Bad negatives : 0
Bad positives : 7
Accuracy rate : 98.34905660377359 %

--- End of a section, press any enter to continue ---

	SGD, batch size 32
Ein:  0.08183880846320654
Eout:  0.13300563169544255

Evaluating output training data set
For w^T = [[-1.23840622 -0.24445313 -0.45432575]]
Input size:  1561
Bad negatives : 8
Bad positives : 3
Accuracy rate : 99.29532351057014 %

Evaluating output test data set
For w^T = [[-1.23840622 -0.24445313 -0.45432575]]
Input size:  424
Bad negatives : 1
Bad positives : 7
Accuracy rate : 98.11320754716981 %

--- End of a section, press any enter to continue ---

	SGD, batch size 200
Ein:  0.0824097083428238
Eout:  0.13514678139988967

Evaluating output training data set
For w^T = [[-1.21138905 -0.14846307 -0.44806566]]
Input size:  1561
Bad negatives : 6
Bad positives : 3
Accuracy rate : 99.4234465086483 %

Evaluating output test data set
For w^T = [[-1.21138905 -0.14846307 -0.44806566]]
Input size:  424
Bad negatives : 1
Bad positives : 7
Accuracy rate : 98.11320754716981 %

--- End of a section, press any enter to continue ---

	SGD, batch size 1561
Ein:  0.40484272492435486
Eout:  0.42805781278130317

Evaluating output training data set
For w^T = [[-0.42953442 -0.04548081 -0.23773542]]
Input size:  1561
Bad negatives : 1
Bad positives : 12
Accuracy rate : 99.16720051249199 %

Evaluating output test data set
For w^T = [[-0.42953442 -0.04548081 -0.23773542]]
Input size:  424
Bad negatives : 0
Bad positives : 20
Accuracy rate : 95.28301886792453 %

--- End of a section, press any enter to continue ---

\end{verbatim}

      \paragraph{300 steps}
\begin{verbatim}

	SGD, batch size 1
Ein:  0.07991429824341009
Eout:  0.13043428762931666

Evaluating output training data set
For w^T = [[-1.14204678 -1.25437994 -0.4976879 ]]
Input size:  1561
Bad negatives : 7
Bad positives : 3
Accuracy rate : 99.35938500960923 %

Evaluating output test data set
For w^T = [[-1.14204678 -1.25437994 -0.4976879 ]]
Input size:  424
Bad negatives : 0
Bad positives : 7
Accuracy rate : 98.34905660377359 %

--- End of a section, press any enter to continue ---

	SGD, batch size 32
Ein:  0.08063407701065878
Eout:  0.13581316178349648

Evaluating output training data set
For w^T = [[-1.18587205 -0.6497891  -0.48419008]]
Input size:  1561
Bad negatives : 5
Bad positives : 3
Accuracy rate : 99.48750800768738 %

Evaluating output test data set
For w^T = [[-1.18587205 -0.6497891  -0.48419008]]
Input size:  424
Bad negatives : 0
Bad positives : 7
Accuracy rate : 98.34905660377359 %

--- End of a section, press any enter to continue ---

	SGD, batch size 200
Ein:  0.08138110980377243
Eout:  0.1344240103546277

Evaluating output training data set
For w^T = [[-1.23721465 -0.24176584 -0.46011533]]
Input size:  1561
Bad negatives : 7
Bad positives : 3
Accuracy rate : 99.35938500960923 %

Evaluating output test data set
For w^T = [[-1.23721465 -0.24176584 -0.46011533]]
Input size:  424
Bad negatives : 1
Bad positives : 7
Accuracy rate : 98.11320754716981 %

--- End of a section, press any enter to continue ---

	SGD, batch size 1561
Ein:  0.085568968925448
Eout:  0.13713701679830562

Evaluating output training data set
For w^T = [[-1.15962485 -0.13812568 -0.43405538]]
Input size:  1561
Bad negatives : 5
Bad positives : 3
Accuracy rate : 99.48750800768738 %

Evaluating output test data set
For w^T = [[-1.15962485 -0.13812568 -0.43405538]]
Input size:  424
Bad negatives : 0
Bad positives : 7
Accuracy rate : 98.34905660377359 %

--- End of a section, press any enter to continue ---

\end{verbatim}

\section{Experiment }

\subsection{a) Generate a training sample}

We are going to use a uniform generation:

\begin{minted}{python}
def simula_unif(N, d, size):
        ''' generate a trining sample of N  points
in the square [-size,size]x[-size,size]
'''
        return np.random.uniform(-size,size,(N,d))

\end{minted}

After fixed random seed to $1$.

The final 2D map  is

\includegraphics[width=\linewidth]{2_2_a_training_sample.png}

\subsection{b) Labels, noise and map}

b) Let's consider the function $f(x_1, x_2) = sign((x_1 - 0.2)^2 + x_2^2 - 0.6)$ that we will use to assign a label to each point of the previous sample. We introduce noise on the labels, randomly changing the sign of 10 \% of them.
Draw the obtained labels map.

Before plotting, it is important to have in mind that a circumference with radius $r$ and center $(c_1, c_2) \in \mathbb R^2$ are the points $(x_1, x_2) \in \mathbb R^2$ that verify

\[ (x_1 - c_1)^2 + (x_2 - c_2)^2 = r^2\]

Therefore looking at $f$ it is easy to think that we are going to see a circle of radius $\sqrt{0.6}$ and center $(0.2, 0)$.

The plotting is

\includegraphics[width=\linewidth]{/2_2_b_labelled_before_noise.png}

In order to introduce noise on the label we are going to change randomly the sign of the $10\%$ of  the labels obtained by $b$.

\begin{minted}{python}
#labels
y = np.array( [f(x[0],x[1]) for x in training_sample ])

index = list(range(size_training_example))
np.random.shuffle(index)

percent_noisy_data = 10.0
size_noisy_data = int((size_training_example *percent_noisy_data)/ 100 )

noisy_y = np.copy(y)
for i in index[:size_noisy_data]:
    noisy_y[i] *= -1

  \end{minted}

  As we can see the idea behind the snippet is simple: The noised labels would be a copy of the original ones and $10\%$ of the data would change their sign.

  The final map is :

  \includegraphics[width=\linewidth]{2_2_b_labelled_after_noise.png}

  \subsection{ Estimate the fitting error of $E_{in}$ using SGD}

  \subsubsection{Experiment c}
  Using $(1, x_1, x_2)$ as feature vector, fit a linear regression model to the generated datasets and estimate the weights w. Estimate the fitting error of Ein using Stochastic Gradient Descent (SGD).

  Having in mind the observation in the last subsection that the labels follow a circumference's equation with a bit of noise, a linear regression model it is not going to be the best approach.

  The experiment results are:
\begin{verbatim}
EXPERIMENT (c)

	SGD, batch size 5
Ein:  0.9038395322567018

Evaluating output training data set
For w^T = [[ 0.05824863 -0.51637342  0.06313758]]
Input size:  1000
Bad negatives : 163
Bad positives : 212
Accuracy rate : 62.5 %
\end{verbatim}

Remember that the bad negatives are the points that the regression classify in negative but they are positives and the bad positives are the negative ones that are classify as a positives.

Finally the accuracy rate was the corrected classify data divided by the input size, so it is not a good model due to the fact that the accuracy rate is so close to a random one, that it would theoretically have a $50\%$ of accuracy rate.

A visual representation of the projection is

\includegraphics[width=\linewidth]{2_2_c_linear_regression.png}

\subsubsection{d) Repetition of the experiment}

In order to see that the last result was not hazaour we are going to repeat the experiment 1000 times, the result is:

\begin{verbatim}

 EXPERIMENT (d), lineal regression

The mean value of E_in in all 1000 experiments is: 0.9270571984798377
The mean value of E_out in all 1000 experiments is: 0.9330084274789892

\end{verbatim}

So as we can se the mean error is even worse hence, our linear regression is not a good one.

\subsubsection{e) Quadratic adjustment}

e) Assess how good you consider the fit with this linear model is according to the mean values obtained for Ein and Eout. Repeat the same previous experiment but using non-linear characteristics. Now, we will use the following feature vector: $\phi_2(x) =(1,x_1,x_2,x_1 x_2, x_1^2, x_2^2)$. Fit the new linear regression model and calculate the new vector of weights w. Calculate the average errors of Ein and Eout. Which model do you consider the most appropriate according to the average errors for Ein and Eout?

Now, instead of using a linear features vector, we are going to use a quadratic one

\begin{minted}{python}
  def quadraticFeatureVector(x_n):
        '''
        INPUT
         xn = (x1,x2) vector of coordinates

        '''
        return np.array([ 1,
                   x_n[0],
                   x_n[1],
                   x_n[0]*x_n[1],
                   x_n[0]* x_n[0],
                   x_n[1]* x_n[1]  ])

                 \end{minted}

                 We know that the function is a circumference with a 10\% of noise, so apriori we now that our target function is going to be

                 $$h(x,y) = (x-0.2)^2 + y^2 -0.6 = x^2 - 0.4x + 0.04 + y^2 - 0.6 = x^2 + y^2 - 0.4x - 0.56$$

                 What means that our target weight vector is goint to be

                 $$w_t = (-0.56, -0.4, 0,0,1,1)$$

                 Moreover due to the fact that we are introducing 10\% of noisy our accuracy level must be around $90\%$

                After $1000$ iterations we obtain:

 \begin{verbatim}
 SGD, batch size 5, number iterations 1000
Ein:  0.5724680127717665

Evaluating output training data set
For w^T =
[[-0.67345368 -0.45209455 -0.0521589   0.07569398  0.92011803  1.23673771]]
Input size:  1000
Bad negatives : 94
Bad positives : 55
Accuracy rate : 85.1 %

\end{verbatim}

                \includegraphics[width=\linewidth]{2_3_parabolle.png}

                The results are close to our first approach,

                but apriori, for $1000$ iterations they seen a bit bad.

                However if we analyse how the error, accuracy rate and $w$ are evoluting over $10, 50, 100, 200, 500, 700, 1000$ we clearly see that the error is improving, so the iterations were not enought.

                Something interesting is that the $700$ iterations have better accuracy rate than 1000, this is for the same reason we explained at exercise 1; we are minimazing the error, not the accuracy rate.

 \begin{verbatim}

For one experiment:

 SGD, batch size 5, number iterations 10
Ein:  0.96683465968506

Evaluating output training data set
For w^T =
[[ 0.02353559 -0.04212481 -0.00067979  0.00785992  0.02654321  0.03756063]]
Input size:  1000
Bad negatives : 0
Bad positives : 474
Accuracy rate : 52.6 %

For one experiment:

 SGD, batch size 5, number iterations 50
Ein:  0.8964601273439083

Evaluating output training data set
For w^T =
 [[-0.00103819 -0.136871   -0.01033269  0.00363211  0.1056979   0.14178948]]
Input size:  1000
Bad negatives : 34
Bad positives : 266
Accuracy rate : 70.0 %

For one experiment:

 SGD, batch size 5, number iterations 100
Ein:  0.8337613668693657

Evaluating output training data set
For w^T = [[-0.04576936 -0.244725   -0.00396614  0.015827    0.19350407  0.24068118]]
Input size:  1000
Bad negatives : 64
Bad positives : 198
Accuracy rate : 73.8 %

For one experiment:

 SGD, batch size 5, number iterations 200
Ein:  0.751970586436338

Evaluating output training data set
For w^T = [[-0.16858105 -0.3725849  -0.01499836  0.02664101  0.30993404  0.4367047 ]]
Input size:  1000
Bad negatives : 93
Bad positives : 121
Accuracy rate : 78.6 %

For one experiment:

 SGD, batch size 5, number iterations 500
Ein:  0.6352766784644138

Evaluating output training data set
For w^T =
[[-0.41975341 -0.46092311 -0.02902233  0.06514428  0.6286317   0.84761192]]
Input size:  1000
Bad negatives : 80
Bad positives : 72
Accuracy rate : 84.8 %

For one experiment:

 SGD, batch size 5, number iterations 700
Ein:  0.5997408279310205

Evaluating output training data set
For w^T =
[[-0.53430804 -0.46110075 -0.0464511   0.06892706  0.76915291  1.04094009]]
Input size:  1000
Bad negatives : 78
Bad positives : 63
Accuracy rate : 85.9 %

For one experiment:

 SGD, batch size 5, number iterations 1000
Ein:  0.5724680127717665

Evaluating output training data set
For w^T =
[[-0.67345368 -0.45209455 -0.0521589   0.07569398  0.92011803  1.23673771]]
Input size:  1000
Bad negatives : 94
Bad positives : 55
Accuracy rate : 85.1 %

\end{verbatim}

\subsubsection{Mean error and conclusion}

In order to see the last result was not hazaour we are going to repeat the experiment 1000 times, the result is:

\begin{verbatim}
The mean value of E_in in all 1000 experiments is: 0.600107053636983
The mean value of E_out in all 1000 experiments is: 0.6058641910896161
\end{verbatim}

More or less the error is the same, so we can trust that a good adjustment is a quadratic one.

Something which we have known a priori, because we knew our target vector (something that in real life does not happen).


\subsection{Taylor series}

\(f(x)\) can be estimated at point \(c\) by identifying its repeated differentials at point \(c\).

The coefficients of an infinate number of polynomials at point \(c\) allow this.

\(f(x)=\sum_{i=0}^{\infty }a_i(x-c)^i\)

\(f'(x)=\sum_{i=1}^{\infty }a_i(x-c)^{i-1}i\)

\(f''(x)=\sum_{i=2}^{\infty }a_i(x-c)^{i-2}i(i-1)\)

\(f^j(x)=\sum_{i=j}^{\infty }a_i(x-c)^{i-j}\dfrac{i!}{(i-j)!}\)

For \(x=c\) only the first term in the series is non-zero.

\(f^j(c)=\sum_{i=j}^{\infty }a_i(c-c)^{i-j}\dfrac{i!}{(i-j)!}\)

\(f^j(c)=a_ij!\)

So:

\(a_j=\dfrac{f^j(c)}{j!}\)

So:

\(f(x)=\sum_{i=0}^\infty (x-c)^i \dfrac{f^i(c)}{i!}\)


\subsection{Diagonalisable matrices and eigendecomposition}

If matrix \(M\) is diagonalisable if there exists matrix \(P\) and diagonal matrix \(A\) such that:

\(M=P^{-1}AP\)

\subsubsection{Diagonalisiable matrices and powers}

If these exist then we can more easily work out matrix powers.

\(M^n=(P^{-1}AP)^n=P^{-1}A^nP\)

\(A^n\) is easy to calculate, as each entry in the diagonal taken to the power of \(n\).

\subsubsection{Defective matrices}

Defective matrices are those which cannot be diagonalised.

Non-singular matries can be defective or not defective, for example the identiy matrix.

Singular matrices can also be defective or not defective, for example the empty matrix.

\subsubsection{Eigen-decomposition}

Consider an eigenvector \(v\) and eigenvalue \(\lambda \) of matrix \(M\).

We known that \(Mv=\lambda v\).

If \(M\) is full rank then we can generalise for all eigenvectors and eigenvalues:

\(MQ=Q\Lambda\)

Where \(Q\) is the eigenvectors as columns, and \(\Lambda \) is a diagonal matrix with the corresponding eigenvalues. We can then show that:

\(M=Q\Lambda Q^{-1}\)

This is only possible to calculate if the matrix of eigenvectors is non-singular. Otherwise the matrix is defective.

If there are linearly dependent eigenvectors then we cannot use eigen-decomposition.

\subsection{Using the eigen-decomposition to invert a matrix}

This can be used to invert \(M\).

We know that:

\(M^{-1}=(Q\Lambda Q^{-1})^{-1}\)

\(M^{-1}=Q^{-1}\Lambda^{-1}Q\)

We know \(\Lambda \) can be easily inverted by taking the reciprocal of each diagonal element. We already know both \(Q\) and its inverse from the decomposition.

If any eigenvalues are \(0\) then \(\Lambda \) cannot be inverted. These are singular matrices.


Let's modify the permutation problem, so that we do not care in which order the $s$ objects are chosen from the $n$ objects. This amounts to the following problem: How many ways are there to choose a size $s$ subset of objects from a size $n$ set of objects? (Remember that order does not matter when listing a subset.)

In order to indicate the general solution we will solve a more specific example.
\begin{example}
 Suppose I have $6$ different colors of paint, and I must choose $4$ of them to mix together (so that the order in which I choose the paint does not matter). How many ways are there for me to make such a choice?
\end{example}
\begin{proof}[solution]
 From the permutation problem, we have seen that there are $P(6, 4) = 6!/2! = 360$ ways to choose $4$ colors in order. However, this is not the answer to the problem, because we count certain combinations as different when they are really the same. For example, the ordered selection (red, blue, green, yellow) will be counted differently from the ordered selection (blue, green, red, yellow). This means for each distinct combination, we count it $4! = 24$ times (since this is the number of ways to arrange $n!$ in a line.

 This means that the number of permutations is the number of combinations times $4!$. So there are $\frac{1}{4!}P(6,4) = 15$ ways to choose $4$ colors without regard to order.
\end{proof}

We can directly generalize this problem to solve the general combination problem. We call a collection of $r$ objects chosen from a collection of $n$ objects, where the order does not matter, a \textit{combination} of $r$ objects from $n$. From the permutation problem there are $P(n, r) = n!/(n-r)!$ ways to choose $r$ objects in a certain order. For each \textit{combination} of $r$ objects there are $r!$ different orderings associated with it. Since each distinct combination gives rise to $r!$ different permutations, and each permutation can be realized as a combination, we conclude that there must
\[\frac{1}{r!}P(n, r) = \frac{n!}{r!(n-r)!}\]
distinct combinations of $r$ objects from $n$ objects. This number is important enough that we give it a special notation. We define the ``binomial coefficient'' $\binom{n}{r}$ (read ``$n$ choose $r$'') to be the value $\frac{n!}{r!(n-r)!}$.

The method of solving the combination problem from the permutation problem is an important (often only implicitly discussed!) technique, which we call \textbf{the principle of division}.

\begin{tcolorbox}
 Suppose I want to count the number a certain collection of objects $S$, and that I know the number of objects to a certain other collection of objects $T$ with the following properties:
 \begin{itemize}
     \item For each element $s \in S$ we can associate it with exactly $r$ objects in $T$.
     \item  Each object in $T$ can only be associated with a unique object in $S$.
 \end{itemize}

Then it follows that the number of objects in the collection $S$ is $|T|/r$, or the number of objects in $T$ divided by $r$.
\end{tcolorbox}

\section{Probability in terms of Combinatorics}

With knowledge of combinatorics, it is possible to give the answer to some basic questions about probability. Here is the basic idea:

\begin{tcolorbox}
 Suppose that I have a \textit{finite} (this is important!) collection $S$ of objects, and I want to know the probability that I choose some object with a certain property from $S$. Let $T$ denote the subcollection of objects with this property. Then the desired probability is $|T|/|S|$, or the size of $T$ divided by the size of $S$.
\end{tcolorbox}
Remark that this collection is finite is important, because implicitly this probability was attained under the uniform distribution, which does not make sense for infinite collections.


\subsection{Eigenspace}

Kernel of a matrix always forms subspace of domain. If $\lambda$ is an eigenvalue for $A$,
kernel of $A-\lambda I$ is the \textbf{eigenspace} associated with $\lambda$ and
this is denotes as $E_\lambda=\mathrm{ker}(A-\lambda I)$.

If $A-\lambda I$ has column vectors $\tb{v}_1$ and $\tb{v}_2$, then
$E_\lambda=\mathrm{span}\left\{\begin{bmatrix}c_1\\c_2\end{bmatrix}\right\}$ where $c_1\tb{v}_1+c_2\tb{v}_2=\tb{0}$.

\subsection{Multiplicity}

Dimension of eigenspace $E_\lambda$ is \textbf{geometric multiplicity} of $\lambda$.
Multiplicity of root $\lambda$ is \textbf{algebraic multiplicity} in characteristic polynomial $p_A(\lambda)$.
Therefore, geometric multiplicity $\leq$ algebraic multiplicity, considering the case of $p_A(\lambda)=(\lambda-\lambda_0)^2$.

Thus, this represents a $2\times 2$ matrix which fixes one line and moves every other line, known as a shear. All lines but $x$-axis move.
Has characteristic polynomial $p_A(\lambda)=(\lambda-1)^2$, so $E_1=\mathrm{ker}\begin{bmatrix}0&k\\0&0\end{bmatrix}=\mathrm{span}\left\{\begin{bmatrix}1\\0\end{bmatrix}\right\}$.

\subsection{Eigenbasis}

Consists of the eigenvectors of the coefficient matrix. \textbf{An $n\times n$ matrix needs $n$ linearly independent
eigenvectors to have an eigenbasis}. This means that if $A$ has eigenvectors $\lambda_1\neq\lambda_2$, then $E_{\lambda_1}\cap E_{\lambda_2}=\left\{\tb{0}\right\}$.
This is because if some $\tb{v}\in E_{\lambda_1}$, then $A\tb{v}=\lambda_1\tb{v}$ and $A\tb{v}=\lambda_2\tb{v}$.
Thus, $(\lambda_1-\lambda_2)\tb{v}=\tb{0}$, so $\tb{v}=\tb{0}$.

\noindent
Furthermore, if $E_{\lambda_1}$ has basis $\mathfrak{E}_{\lambda_1}$ and $E_{\lambda_2}$ with $\mathfrak{E}_{\lambda_2}$, $\mathfrak{E}_{\lambda_1}\cup \mathfrak{E}_{\lambda_2}$ is linearly independent as well
with the total elements being the sum of the number of elements in each individual basis.
Can then be concluded that:

$$
\boxed{\text{An $n\times n$ matrix has eigenbasis iff sum of geometric multiplicities of eigenvalues is $n$.}}
$$


\textbf{Based on the theorems of chapter 8 of Approximation Theory and Approximation Practice by Trefethen, what can you say about the convergence as $n\rightarrow\infty$ of the Chebyshev interpolants to the following functions? Which is the case that converges much faster than the theorems predict? Can you speculate why?}
\newline

The convergence, predicted by \textsc{Theorem 8.2}, is
\begin{align*}
\|f-p_n\|\leq\frac{4M\rho^{-n}}{\rho-1},
\end{align*}
with
\begin{align*}
\rho = \alpha+\sqrt{\alpha^2-1}~~~~\text{(real singularity at $x=\pm\alpha$)},
\end{align*}
or
\begin{align*}
\rho = \beta+\sqrt{\beta^2+1}~~~~\text{(imaginary singularity at $x=\pm i\beta$)}.
\end{align*}

Applying this to the following functions we obtain:
\begin{enumerate}[label=\alph*)]
\item $f(x) = \tan(x)$. This function has a real singularity at $x=\pm \pi/2$. Hence,
\begin{align*}
\rho = \frac{\pi}{2}+\sqrt{\frac{\pi^2}{4}-1},
\end{align*}
and we have obtained the rate of convergence showed in the next figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{tan(x).png}\caption{Convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to $f(x)=\tan(x)$.}
\end{figure}

\item $f(x) = \tanh(x)$. This function has a imaginary singularity at $x=\pm i\pi/2$. Hence,
\begin{align*}
\rho = \frac{\pi}{2}+\sqrt{\frac{\pi^2}{4}+1},
\end{align*}
and we have obtained the rate of convergence showed in the next figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{tanh(x).png}\caption{Convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to $f(x)=\tanh(x)$.}
\end{figure}

\item $f(x) = \frac{\log(\frac{x+3}{4})}{x-1}$. This function has a real singularity at $x=-3$. Hence,
\begin{align*}
\rho = 3+\sqrt{8},
\end{align*}
and we have obtained the rate of convergence showed in the next figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{log1.png}\caption{Convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to $f(x)= \frac{\log(\frac{x+3}{4})}{x-1}$.}
\end{figure}

\item $f(x) = \int_{-1}^{x}\cos(t^2)dt$. This function is entire, analytic at all finite points of the complex plane. Hence its convergence is much faster as is shown in the following figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{integral.png}\caption{Convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to $f(x)= \frac{\log(\frac{x+3}{4})}{x-1}$.}
\end{figure}

\item $f(x) = \tan(\tan(x))$. This function has a real singularity at $x=\pm \arctan(\pi/2)=\pm k$. Hence,
\begin{align*}
\rho = k+\sqrt{k^2-1},
\end{align*}
and we have obtained the rate of convergence showed in the next figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{tan(tan(x)).png}\caption{Convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to $f(x)= \tan(\tan(x))$.}
\end{figure}

\item $f(x) = (1+x)\log(1+x)$. This function has a real singularity at $x=-1$. Hence,
\begin{align*}
\rho = -1,
\end{align*}
and we have obtained the rate of convergence showed in the next figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{(1+x)log(1+x)wrong.png}\caption{Convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to $f(x)= (1+x)\log(1+x)$.}
\end{figure}
This figure is not right since the function does not satisfies the assumptions of \newline \textsc{Theorem 8.1}, it is not analytic in $[-1,1]$. The rate of convergence found is algebraic (much worse than the previous cases), as shown in the following figure.

\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{(1+x)log(1+x)right.png}\caption{Algebraic convergence as $n\rightarrow\infty$ of the Chebyshev interpolant to \newline $f(x)= (1+x)\log(1+x)$.}
\end{figure}
\end{enumerate}

Overall, it has been tested that the larger the ellipse with foci $\{-1,1\}$ within which the function is analytic, the faster the convergence.
\subsection*{Matlab code for this problem}
\begin{verbatim}
rho = 0.5*pi+sqrt(0.25*pi^2-1);
orderAcuracy('tan(x)',50,2,rho)

rho = (0.5*pi+sqrt(0.25*pi^2+1));
orderAcuracy('tanh(x)',40,2,rho)

rho = (3+sqrt(8));
orderAcuracy('log((x+3)/4)/(x-1)',30,2,rho,'log1')

k = atan(pi/2);
rho = k+sqrt(k^2-1);
orderAcuracy('tan(tan(x))',460,20,rho)

rho = -1;
orderAcuracy('(1+x)*log(1+x)',460,20,rho,'(1+x)log(1+x)wrong')
orderAcuracy('(1+x)*log(1+x)',460,20,rho,'(1+x)log(1+x)right')

function orderAcuracy(func,Nmax,Nstep,rho,namefig)
    labelfontsize = 14;
    figformat = 'png';
    if nargin < 5
        namefig = func;
    end
    f = chebfun(func);
    nn = 0:Nstep:Nmax; ee = 0*nn;
    for j=1:length(nn)
        n = nn(j);
        fn = chebfun(f,n+1);
        ee(j) = norm(f-fn);
    end
    figure
    if strcmp(namefig,'(1+x)log(1+x)right')
        semilogy(nn,2000*nn.^(-3.8),'-b')
    else
        semilogy(nn,rho.^(-nn),'-b')
    end
    hold on
    semilogy(nn,ee,'r*')
    grid on
    xlabel('$N$','interpreter','latex')
    ylabel('$\|f-p_N\|$','interpreter','latex')
    set(gca,'fontsize',labelfontsize)
    txt=['Latex/FIGURES\' namefig];
    saveas(gcf,txt,figformat)
end

% For the integral function
nn = 2:2:20;
err = 0*nn;
for k = 1:length(nn)
    N = nn(k);
    x = -.98:0.02:1;
    F = 0*x;
    FN = F;
    for j = 1:length(x)
        f = chebfun('cos(t^2)', [-1 x(j)]);
        fN = chebfun('cos(t^2)', [-1 x(j)],N);
        F(j) = sum(f);
        FN(j) = sum(fN);
        F = [0 F]; FN = [0 FN]; x = [-1 x];
    end
    err(k) = norm(F-FN);
end
figure
semilogy(nn,err,'r*')
grid on
xlabel('$N$','interpreter','latex')
ylabel('$\|f-p_N\|$','interpreter','latex')
set(gca,'fontsize',labelfontsize)
txt='Latex/FIGURES\integral';
saveas(gcf,txt,figformat)
\end{verbatim}


\section{Generative Learning: Gaussian Discriminant Analysis}
A generative model learns who the data is generated by estimating $P(x|y)$
which is then used to estimate $P(y|x)$ via Bayes rule.
\subsection{Assumptions}
Gaussian Discriminant Analysis assumes the following:
\begin{enumerate}
  \item $y\sim\textrm{Bernoulli}(\phi)$
  \item $x|y=0\sim\mathcal{N}(\mu_0,\Sigma)$
  \item $x|y=1\sim\mathcal{N}(\mu_1,\Sigma)$
\end{enumerate}
Writing out the distributions be have:
\begin{equation}
  P(y) = \phi^y \cdot (1-\phi)^{1-y}
\end{equation}
\begin{equation}
  P(x|y=k) = \frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}} \exp \left(-\frac{1}{2} (x-\mu_k)^T \Sigma^{-1} (x - \mu_k)\right)
\end{equation}
\subsection{Estimation}
The log-likelihood of the data is given by:
\begin{equation}
  \begin{aligned}
    \ell(\phi, \mu_0, \mu_1, \Sigma) \quad =& \quad \log \prod_{i=1}^{m}P(x^{(i)}, y^{(i)}; \phi, \mu_0, \mu_1, \Sigma) \\
    =& \quad \log \prod_{i=1}^m P(x^{(i)}| y^{(i)}; \phi, \mu_0, \mu_1, \Sigma) \cdot P(y^{(i)}; \phi)
  \end{aligned}
\end{equation}
By maximizing $\ell$ with respect to the parameters, we get the following MLE
of the parameters:
\begin{equation}
  \phi = \displaystyle\frac{1}{m}\sum_{i=1}^m1_{\{y^{(i)}=1\}}
\end{equation}
\begin{equation}
  \mu_k = \displaystyle\frac{\sum_{i=1}^m1_{\{y^{(i)}=k\}}x^{(i)}}{\sum_{i=1}^m1_{\{y^{(i)}=k\}}}
\end{equation}
\begin{equation}
  \Sigma = \displaystyle\frac{1}{m}\sum_{i=1}^m(x^{(i)}-\mu_{y^{(i)}})(x^{(i)}-\mu_{y^{(i)}})^T
\end{equation}
\subsection{Prediction}
To classify a point $x$, we find the class $y=k$ which maximizes the probability:
\begin{equation}
  \hat{y} = \underset{k}{\textrm{arg max}} \,\, P(y=k) \cdot\prod_j P(x_j | y=k)
\end{equation}


\textbf{Consider the predictor-corrector time-stepping scheme:}
\begin{align*}
& y_{n+1}^p = y_n + \frac{\Delta t}{12}(23f_n - 16f_{n-1}+5f_{n-2}), \\
&y_{n+1} = y_n + \frac{\Delta t}{12}(5f(t_{n+1},y_{n+1}^p) + 8f)n - f_{n-1})~.
\end{align*}
\textbf{where $f_n = f(t_n,y_n)$ and $y'(t) = f(t,y)$, $y(0) = y_0$. Plot the stability region for this scheme.}

Let $f_n = \lambda y^{n}$ and $a = \frac{\Delta t \lambda}{12}$. We start by simply substituting $y_{n+1}^p$ from the first equation into the second,
\begin{align*}
y^{n+1} = y^n + a \left[ 5(y^n + a(23y^n- 16y^{n-1}+5y^{n-1})) + 8y^n - y^{n-1} \right]~.
\end{align*}
Further let $y^{n+1} = gy^n$,
\begin{align*}
gy^n = y^n +a\left[ 5y^n + 115ay^n - 80\frac{a}{g}y^n + 25\frac{a}{g^2}y^n\right] + 8ay^n - \frac{a}{g}y^n.
\end{align*}
Reorganizing the terms, we obtain a second degree polynomial for $a$,
\begin{align*}
(115g^2 - 80g +25)a^2 + (13g^2-g)a + (g^2-g^3) = 0,
\end{align*}
which gives us the solution
\begin{align*}
a = \frac{-(13g^2 - g) \pm \sqrt{(13g^2-g)^2 - 4(115g^2 - 80g+25)(g^2-g^3)}}{2(115g^2 - 80g +25)}.
\end{align*}
Note that $z =\Delta t\lambda= 12a$,
\begin{align*}
z =12 \frac{-(13g^2 - g) \pm \sqrt{(13g^2-g)^2 - 4(115g^2 - 80g+25)(g^2-g^3)}}{2(115g^2 - 80g +25)}.
\end{align*}
We plot the previos in \textsl{Matlab} and obatin stability region shown in the next figure.

\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{P3.eps}}
\caption{Stability region.}
\end{figure}

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 3
clear variables
close all
clc
N=300;
theta = linspace(0,2*pi,N);
g=exp(1i*theta);
a=115*g.^2-80*g+25;
b=13*g.^2-g;
c=g.^2-g.^3;
sq=sqrt(b.^2-4*a.*c);
zp=12*(-b+sq)./(2*a);
zn=12*(-b-sq)./(2*a);
figure
plot(real(zp),imag(zp),'b.')
hold on
plot(real(zn),imag(zn),'b.')
grid on
axis('image', [-2.5 0.5 -1.5 1.5])
xlabel('$\Re(z)$','interpreter','latex'...
    ,'fontsize',16)
ylabel('$\Im(z)$','interpreter','latex'...
    ,'fontsize',16)
txt='Latex/FIGURES/P3';
saveas(gcf,txt,'epsc')
\end{verbatim}


\begin{solution}
We start by expanding $f(x+h)$ and $f(x-h)$ up to eigth order,
\begin{align*}
f(x\pm h)=&f(x)\pm hf'(x)+\frac{h^2}{2}f''(x)\pm\frac{h^3}{3!}f'''(x)+\frac{h^4}{4!}f^{(IV)}(x)\pm\frac{h^5}{5!}f^{(V))}(x)+\frac{h^6}{6!}f^{(VI)}(x)\\
&\pm\frac{h^7}{7!}f^{(VII))}(x)+\frac{h^8}{8!}f^{(VIII)}(x)+\mathcal{O}(h^9).
\end{align*}
By adding both $f(x+h)$ and $f(x-h)$ together, crossing out many terms,
\begin{align*}
f(x+h)+f(x-h)=2f(x)+h^2f''(x)+\frac{2h^4}{4!}f^{(IV)}+\frac{2h^6}{6!}f^{(VI)}+\frac{2h^8}{8!}f^{(VIII)}+\mathcal{O}(h^9),
\end{align*}
we can now isolate the second derivative
\begin{align*}
f''(x)=\frac{f(x-h)-2f(x)+f(x+h)}{h^2}-\frac{h^2}{12}f^{(IV)}-\frac{2h^4}{6!}f^{(VI)}-\frac{2h^6}{8!}f^{(VIII)}+\mathcal{O}(h^9).
\end{align*}
If we want sixth order precision we need to get rid of the firs two leading terms of the error. In order to do so we use more stencil points. We can express the second derivative as
\begin{align*}
f''(x)=\frac{f(x-2h)-2f(x)+f(x+2h)}{4h^2}-\frac{h^2}{3}f^{(IV)}-\frac{2h^4}{45}f^{(VI)}-\frac{h^6}{315}f^{(VIII)}+\mathcal{O}(h^9),
\end{align*}
and
\begin{align*}
f''(x)=\frac{f(x-3h)-2f(x)+f(x+3h)}{9h^2}-\frac{3h^2}{4}f^{(IV)}-\frac{9h^4}{40}f^{(VI)}-\frac{81h^6}{2240}f^{(VIII)}+\mathcal{O}(h^9).
\end{align*}
Multiplying the three equations by $a,b,c$, respectively, and adding them together we can impose the condition that the coefficients of the second and fourth order error terms must be zero, obtaining a sixth order approximation. The system of equations to solve is then
\begin{align*}
&\frac{a}{12}+\frac{b}{3}+\frac{3c}{4}=0,\\
&\frac{0}{360}+\frac{2b}{45}+\frac{9c}{40}=0.
\end{align*}
The solutions are $a=15c$, $b=-6c$, and $c$ will vanish from the system. Once we put this values in the equations and add them together we get the final result
\begin{align*}
f''(x)=&\frac{15\left[f(x-h)+f(x+h)\right]-\frac{3}{2}\left[f(x-2h)+f(x+2h)\right]+\frac{1}{9}\left[f(x-3h)+f(x+3h)\right]-\frac{245}{9}f(x)}{10h^2}\\&-\frac{h^6}{560}f^{(VIII)}(x)+\mathcal{O}(h^7),
\end{align*}
a sixth order approximation to the second derivative.
In the figure 1 we can see the funtion and its derivatives
\begin{align*}
f(x)=e^x\sin{(3x)},~~~~~~f'(x)=e^x\left(\sin{(3x)}+3\cos{(3x)}\right),~~~~~~f''(x)=e^x\left(6\cos{(3x)}-8\sin{(3x)}\right),
\end{align*}
as well as the value of the second derivative at the point $x_0=0.4$.
\begin{figure}[H]
\centering     %%% not \center
\includegraphics[scale=0.7]{IMAGES/problem2_1.eps}\caption{The function $f(x)$ and its two first derivatives. It is also included the value of the second derivative at $x_0=0.4$ (asterisk) and the approximated value (square) using the approximation derived above.}
\end{figure}
We can see that the approximated value for the three values of $h$ is very accurate as we cannot distinguish any square from the astersik. However, in the following figure we can see the error and, in logarithmic scale, seems to have a linear dependency on $h$.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.7]{IMAGES/problem2_2.eps}}
\caption{Error of the approximation.}
\end{figure}
We made a linear fitting and found out that the slope was $m=5.990378959014029$. This proves that the approximation is, indeed, of sixth order.
\subsection*{Matlab code for this problem}
\begin{verbatim}
x=-1:0.005:1;
x_0=0.4;
f = chebfun(@(x) exp(x)*sin(3*x));
df = chebfun(@(x) exp(x)*(sin(3*x)+3*cos(3*x)));
ddf = chebfun(@(x) exp(x)*(6*cos(3*x)-8*sin(3*x)));

H=[0.1;0.05;0.025];
ddf_approx=zeros(3,1);
for k=1:3
    h=H(k);
    ddf_approx(k)=(15*(f(x_0-h)+f(x_0+h))-1.5*(f(x_0-2*h)+f(x_0+2*h))
    +(1/9)*(f(x_0-3*h)+f(x_0+3*h))-(245/9)*f(x_0))/(10*h^2);
end

figure(1)
plot(x,f(x),x,df(x),x,ddf(x),x_0,ddf(x_0),'*',x_0,ddf_approx(1),'s',
x_0,ddf_approx(2),'s',x_0,ddf_approx(3),'s')
grid on
legend({'$f(x)$','$f''(x)$','$f''''(x)$'},'Interpreter','latex')
xlabel('$x$','Interpreter','latex')
set(gca,'fontsize',18)
saveas(gcf,'IMAGES/problem2_1','epsc')

P=polyfit(log(H),log(abs(ddf(x_0)-ddf_approx)),1);
h=0.01:1e-3:0.15;
figure(2)
loglog(H,abs(ddf(x_0)-ddf_approx),'*')
grid on
xlabel('$h$ (log scale)','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
set(gca,'fontsize',14)
hold on
loglog(h,exp(P(1)*log(h)+P(2)))
grid on
axis([2e-2 1.2e-1 1e-12 1e-2])
saveas(gcf,'IMAGES/problem2_2','epsc')
P =

   5.990378959014029   3.227778361027343

\end{verbatim}
\end{solution}


\textit{CLRS section 3.2 p. 53}
\subsection{Monotonicity}
A function $ f(n) $ is
\begin{description}
    \item[monotonically increasing] if $ m \leq n $ implies $ f(m) \leq f(n) $.
    \item[monotonically decreasing] if $ m \leq n $ implies $ f(m) \geq f(n) $.
    \item[strictly increasing] if $ m < n $ implies $ f(m) < f(n) $.
    \item[strictly decreasing] if $ m < n $ implies $ f(m) > f(n) $.
\end{description}

\subsection{Exponentials}
For all real $ a > 0$, $ b > 0 $, $ m $, and $ n $, we have the following identities:\\
\begin{align*}
a^0 &= 1\\
a^1 &= a\\
a^{-1} &= \frac{1}{a}\\
\left( a^m\right) ^n &= a^{nm}\\
\left( a^m\right) ^n &= \left( a^n\right) ^m\\
a^ma^n &= a^{n+m}\\
0^0&=1\\
a^mb^m&=(ab)^m\\
a^\frac{m}{n} &= \left( \sqrt[n]{a}\right) ^m\\
a^{-m}&=\dfrac{1}{a^m}\\
a^{m-n}&=\dfrac{a^m}{a^n}
\end{align*}
\subsection{Logarithms}
\emph{Logarithm functions apply only to the next term in the formula}, so that $ \lg n + k $ will mean $ \left( \lg n\right) +k $ and not $ \lg(n+k) $.
CLRS uses the following notations:
\begin{align*}
\lg n &= \log_2 n &\text{(binary logarithm)}\\
\ln n &= \log_e n &\text{(natural logarithm)}\\
\lg^kn &= (\lg n)^k &\text{(exponentiation)}\\
\lg\lg n &= \lg (\lg n) &\text{(composition)}
\end{align*}
For all real $ a > 0 $, $ b > 0 $, $ c > 0 $, and $ n $,
\begin{align*}
a &= b^{\log_ba}\\
\log_c(ab) &= \log_ca+\log_cb\\
\log_ba^n &= n\log_ba\\
\log_ba &= \dfrac{\log_ca}{\log_cb}\\
\log_b\left( \frac{1}{a}\right) &= -\log_ba\\
\log_ba &= \dfrac{1}{\log_ab}\\
a^{\log_bc}&=c^{\log_ba}\\
\log_b 1 &= 0\\
\log_bb &= 1\\
\log_b\left( \frac{a}{c}\right) &= \log_b a - \log_b c
\end{align*}
where, in each equation above, logarithm bases are not 1.\\

There is a simple series expansion for $ \ln (1+x) $ when $ |x| < 1 $:
$$ \ln(1+x) = x - \dfrac{x^2}{2} + \dfrac{x^3}{3} - \dfrac{x^4}{4} + \dfrac{x^5}{5} - \cdots $$

We also have the following inequalities for $ x > -1 $:
$$ \dfrac{x}{1+x} \leq \ln(1+x) \leq x $$
where equality holds only for $ x = 0 $.

Algorithms and Data Structures - Reference/notation.tex


\subsection{Definition}

A series is the summation of a sequence. For a series \(a_n\) there is a corresponding series:

\(s_n=\sum_{i=0}^na_n\)

Where:

\(\sum_{i=0}^na_i=a_0+a_1+a_2+...+a_n\)

\subsection{Multiplication of summations}

If all members of a sequence are multiplied by a constant, so is each member of the series.

We can take constants out of the series:

\(s_n=\sum_{i=0}^na_i\)

\(s_n=\sum_{i=0}^ncb_i\)

\(s_n=a\sum_{i=0}^nb_i\)

\subsection{Summation of constants}

If all elements of a sequence are the same, then the series is a multiple of that constant.

\(s_n=\sum_{i=0}^na_i\)

\(s_n=\sum_{i=0}^nc\)

\(s_n=nc\)

\subsection{Addition of summations}

Consider a sequence \(a_i=b_i+c_i\).

\(s_n=\sum_{i=0}^na_i\)

\(s_n=\sum_{i=0}^n(b_i+c_i)\)

We can then split this out.

\(s_n=\sum_{i=0}^nb_i+\sum_{i=0}^nc_i\)

\subsection{Summation from a different start point}

\(\sum_{i=0}^na_i=a_0+\sum_{i=1}^na_i\)

\subsection{Multiple summations}

\(\sum^n_{i=0} \sum^m_{j=0} a_i=n\sum^m_{j=0} a_i\)

\(\sum^n_{i=0} \sum^m_{j=0} a_i b_j=\sum^n_{i=0}a_i\sum^m_{j=0}b_i\)


\subsubsection{Repeated Eigenvalues}
\noindent
If $A$ has an eigenvalue with multiplicity $k$, that eigenvalue needs to generate $k$ fundamental solutions.
If this eigenvalue generates $k$ linearly independent eigenvectors, then the process is much like with distinct eigenvalues.
Otherwise, the matrix $A$ is defective.
\begin{theorem}
	If $\lambda$ is an eigenvalue with multiplicity $k$, and $\left\{\vec{v_1}, \ldots, \vec{v_k}\right\}$ are corresponding linearly independent eigenvectors, the the set of fundamental solutions generated by $\lambda$ is $\left\{e^{\lambda t}\vec{v_1}, \ldots, e^{\lambda t}\vec{v_k}\right\}$.
\end{theorem}

\begin{example}
	Find the general solution to the system
	\begin{equation*}
		\vec{x}' = \begin{bmatrix}
			0 & 1 & 1 \\
			1 & 0 & 1 \\
			1 & 1 & 0
		\end{bmatrix} \vec{x}.
	\end{equation*}
\end{example}
\noindent
Finding the eigenvalues by finding the roots of the characteristic polynomial of $A$,
\begin{equation*}
	p(\lambda) = \det{(A - \lambda I)} = -\lambda^3 + 3\lambda + 2 \implies \lambda = 2, -1, -1.
\end{equation*}
Finding the eigenvector for $\lambda = 2$,
\begin{equation*}
	(A - 2I)\vec{v} = \vec{0} \implies \vec{v} = C_1\begin{bmatrix}
		1 \\
		1 \\
		1
	\end{bmatrix}.
\end{equation*}
Finding the eigenvectors for $\lambda = -1$,
\begin{equation*}
	(A + I)\vec{v} = \vec{0} \implies \vec{v} = C_2\begin{bmatrix}
		-1 \\
		1 \\
		0
	\end{bmatrix} + C_3\begin{bmatrix}
		-1 \\
		0 \\
		1
	\end{bmatrix}.
\end{equation*}
So, our solution is
\begin{equation*}
	\vec{x} = C_1e^{2t}\begin{bmatrix}
		1 \\
		1 \\
		1
	\end{bmatrix} + C_2e^{-t}\begin{bmatrix}
		-1 \\
		1 \\
		0
	\end{bmatrix} + C_3e^{-t}\begin{bmatrix}
		-1 \\
		0 \\
		1
	\end{bmatrix}.
\end{equation*}


\documentclass[11pt,twoside]{article}

\usepackage[utopia]{mathdesign}
\usepackage[headings]{fullpage}
\usepackage{color}
\usepackage{enumerate}

\pagestyle{myheadings}
\markboth{}{Blur}

\input{../../fncextra}

\begin{document}

\begin{center}
  \bf Blurred lines
\end{center}

In a computer, an image is ultimately a table of numbers representing pixel intensity values. We can regard these values as a matrix and model some common image operations using linear
algebra. A prominent example of such operations is blurring due to poor focus.

Consider first an $m\times 1$ vector $\bfx$ of pixel intensities, mapped to a
vector $\bfz$ of intensities after blurring. We model each pixel
in the new vector as a weighted average,
\[
z_i = \frac{1}{4}x_{i-1} + \frac{1}{2}x_i + \frac{1}{4} x_{i+1}.
\]
Pixel values that are beyond the boundaries, namely $x_0$ and $x_{m+1}$, are taken to be zero.  We can express the relationship between original and blurred image as a matrix-vector product:
\begin{equation}
\label{eq:blurmatrix}
\bfz =
\begin{bmatrix}
  \frac{1}{2} & \frac{1}{4} & 0           & \cdots      & 0           \\[2mm]
  \frac{1}{4} & \frac{1}{2} & \frac{1}{4} & \cdots      & 0           \\[2mm]
              & \ddots      & \ddots      & \ddots      &             \\
  0           & \cdots      & \frac{1}{4} & \frac{1}{2} & \frac{1}{4} \\[2mm]
  0           & 0           & \cdots      & \frac{1}{4} & \frac{1}{2}
\end{bmatrix}
\bfx = \mB_m \bfx,
\end{equation}
where the $m$ subscript indicates an $m\times m$ matrix.

Things are not much more complicated for actual 2D images. Let $\mX$
be an $m\times n$ image and $\mZ$ be a blurred version of
it. We can first blur in the vertical direction by recalling the columnwise multiplication identity,
\[
\mB_m \mX =
\begin{bmatrix}
  \mB_m \mX_1 & \mB_m \mX_2 & \cdots & \mB_m \mX_n
\end{bmatrix}.
\]
Thus, $\mB_m\mX$ applies vertical blur to each column of the image.

To blur rows, we go in three steps. First, transpose the image to turn rows into columns. Second, apply blurring to each column as shown above. Finally, transpose the result back to restore the original orientation. Hence the fully blurred image is given by
\begin{equation}
\label{eq:blurop}
  \mZ = \Bigl[ \mB_n(\mB_m\mX)^T \Bigr]^T = \mB_m \mX \, \mB_n^T = \mB_m \mX \, \mB_n,
\end{equation}
where the last step uses the symmetry apparent in~\eqref{eq:blurmatrix}.

Finally, we can simulate a larger amount of blur by applying our simple blur multiple times, as in $ (\mB_m)^k \mX \, (\mB_n)^k$ for some integer $k>1$.

\subsection*{Preparation}

Read Section 2.2. Then read the online documentation for \texttt{diag}, paying particular attention to the examples.

\subsection*{Goals}

You will write a function that produces a blur matrix of requested size. This will be used to blur an image that you import as a matrix.

%\end{document}

\subsection*{Procedure}

Download the template script \texttt{Blur} and the template file \texttt{blurmatrix.m}.

\begin{enumerate}
\item The file \texttt{blurmatrix.m} has the first line
\begin{verbatim}
function B = blurmatrix(n)
\end{verbatim}
Within the function, \texttt{n} is an input parameter describing the size of the matrix, and \texttt{B} is the result that is returned to the caller. Using the functions \texttt{ones} and \texttt{diag}, complete the definition of the function so that it returns $\mB_n$ as in~\eqref{eq:blurmatrix}.
\item In the script, construct the matrix $\mB_7$ using a call to \texttt{blurmatrix} and have it printed out (no semicolon) to show that it's correct.
\item Save an image to your working folder. (It should have between 400 and 1000 pixels in each dimension.) The template script has code for importing it, converting it to a floating-point matrix $\mX$, and displaying it.
\item Apply blurring to your image 60 times in the vertical direction only, and display the result as in step~3.
\item Blur the image 60 times in each direction and display it.
\end{enumerate}

\subsection*{Extras}
\begin{enumerate}
\item[E1.] Look at the output of the final step. What creates the ``black lines'' around the edges? What happens to them as the number of blur applications increases?
\end{enumerate}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End: