\section{Angles between planes and lines}\label{s:angles}
In this section, we figure out how to check whether lines and planes are parallel or perpendicular, and then how to find the line perpendicular to a plane going through a given point, and vice versa. All those operations are very easy with our representation of planes and lines.

\subsection{Between two planes}
The angle between two planes is equal to the angle between their normals. Since usually two angles of distinct amplitudes $\theta$ and $\pi-\theta$ are formed, we take the smaller of the two, in $[0,\frac{\pi}{2}]$.

\centerFig{para-perp-0}

We can find it with the following code. We take the minimum with 1 to avoid \lstinline|nan| in case of imprecisions.
\begin{lstlisting}
double smallAngle(p3 v, p3 w) {
    return acos(min(abs(v|w)/abs(v)/abs(w), 1.0));
}
double angle(plane p1, plane p2) {
    return smallAngle(p1.n, p2.n);
}
\end{lstlisting}

In particular, we can check whether two planes are parallel/perpendicular by checking if their normals are parallel/perpendicular:
\begin{lstlisting}
bool isParallel(plane p1, plane p2) {
    return p1.n*p2.n == zero;
}
bool isPerpendicular(plane p1, plane p2) {
    return (p1.n|p2.n) == 0;
}
\end{lstlisting}

\subsection{Between two lines}
The situation with lines is exactly the same: their angle is equal to the angle between their direction vectors. Note that the lines aren't necessarily in the same plane, so the angle is taken as if they were moved until they touch.

\centerFig{para-perp-1}

\begin{lstlisting}
double angle(line3d l1, line3d l2) {
    return smallAngle(l1.d, l2.d);
}
bool isParallel(line3d l1, line3d l2) {
    return l1.d*l2.d == zero;
}
bool isPerpendicular(line3d l1, line3d l2) {
    return (l1.d|l2.d) == 0;
}
\end{lstlisting}

\subsection{Between a plane and a line}
The situation when considering a plane and a line is a bit different.
Let's consider a plane $\Pi$ of normal $\vv{n}$ and a line $l$ of direction vector $\vv{d}$.
When they are perpendicular, $\vv{n}$ is parallel to $\vv{d}$, and inversely when they are parallel, $\vv{n}$ is perpendicular to $\vv{d}$. In general, if the angle between $\vv{n}$ and $\vv{d}$ is $\theta \in [0,\frac{\pi}{2}]$, then the angle between the plane and the line is $\frac{\pi}{2}-\theta$.

\centerFig{para-perp-2}

\begin{lstlisting}
double angle(plane p, line3d l) {
    return M_PI/2 - smallAngle(p.n, l.d);
}
bool isParallel(plane p, line3d l) {
    return (p.n|l.d) == 0;
}
bool isPerpendicular(plane p, line3d l) {
    return p.n*l.d == zero;
}
\end{lstlisting}

\subsection{Perpendicular through a point}
The line perpendicular to a plane $\Pi$ of normal $\vv{n}$ and going through a point $O$ is simply the line going through $O$ and whose direction vector is $\vv{n}$, or equivalently the line going through $O$ and $O+\vv{n}$.

\centerFig{para-perp-3}

\begin{lstlisting}
line3d perpThrough(plane p, p3 o) {return line(o, o+p.n);}
\end{lstlisting}

The plane perpendicular to a line $l$ of direction vector $\vv{d}$ and going through a point $O$ is simply the plane containing $O$ and whose normal is $\vv{d}$.

\centerFig{para-perp-4}

\begin{lstlisting}
plane perpThrough(line3d l, p3 o) {return plane(l.d, o);}
\end{lstlisting}


\section*{Exercises}

\begin{ex} Determine which matrices are in {\rref}.

  \begin{enumerate}
  \item $\begin{mymatrix}{rrr}
      1 & 2 & 0 \\
      0 & 1 & 7
    \end{mymatrix}$

  \item $\begin{mymatrix}{rrrr}
      1 & 0 & 0 & 0 \\
      0 & 0 & 1 & 2 \\
      0 & 0 & 0 & 0
    \end{mymatrix}$

  \item $\begin{mymatrix}{rrrrrr}
      1 & 1 & 0 & 0 & 0 & 5 \\
      0 & 0 & 1 & 2 & 0 & 4 \\
      0 & 0 & 0 & 0 & 1 & 3
    \end{mymatrix}$
  \end{enumerate}
  \begin{sol}
    \begin{enumerate}
    \item This one is not.
    \item This one is.
    \item This one is.
    \end{enumerate}
  \end{sol}
\end{ex}

\begin{ex}
  Reduce each of the matrices from Exercise~\ref{ex:rr-ef} to {\rref}.
\end{ex}

\begin{ex} Use Gauss-Jordan elimination to solve the system of equations
  $-8x+2y+5z=18,-8x+3y+5z=13$, and $-4x+y+5z=19$.
  \begin{sol}
    Solution is: $\mat{x=-1,y=-5,z=4}$
  \end{sol}
\end{ex}

\begin{ex} Use Gauss-Jordan elimination to solve the system of equations $3x-y-2z=3$,
  $y-4z=0$, and $-2x+y=-2$.
  \begin{sol}
    Solution is: $\mat{x=2t+1,y=4t,z=t}$
  \end{sol}
\end{ex}

\begin{ex} Use Gauss-Jordan elimination to solve the system of equations
  $-9x+15y=66,-11x+18y=79$, $-x+y=4$, and $z=3$.
  \begin{sol}
    Solution is: $\mat{x=1,y=5,z=3}$
  \end{sol}
\end{ex}

\begin{ex} Use Gauss-Jordan elimination to solve the system of equations $-19x+8y=-108$,
  $-71x+30y=-404$, $-2x+y=-12$, $4x+z=14$.
  \begin{sol}
    Solution is: $\mat{x=4,y=-4,z=-2}$
  \end{sol}
\end{ex}

\begin{ex}
  Solve the following two systems of equations simultaneously, by
  using a single augmented matrix with two constant vectors.
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l@{~}}
      x  + 2y -  z &=& 0 \\
      2x + 3y +  z &=& 3 \\
      x  -  y + 2z &=& 3 \\
    \end{array}
    \quad\quad
    \begin{array}{r@{~}c@{~}l@{~}}
      x  + 2y -  z &=& 1 \\
      2x + 3y +  z &=& 7 \\
      x  -  y + 2z &=& 4 \\
    \end{array}
  \end{equation*}

  \begin{sol}
    The rank of the coefficient matrix is $3$, so both systems have a
    unique solution. The solution of the first system is
    $(x,y,z)=(1,0,1)$, and the solution of the second system is
    $(x,y,z)=(1,1,2)$.
  \end{sol}
\end{ex}


\chapter{Probability Distributions}

\section*{Exercise 2.1 $\star$}
Verify that the Bernoulli distribution (2.2) satisfies
the following properties
\begin{equation}\label{eq:2.257}\tag{2.257}
    \sum_{x=0}^{1} p(x | \mu) = 1
\end{equation}
\vspace{-1em}
\begin{equation}\label{eq:2.258}\tag{2.258}
    \mathbb{E}[x] = \mu
\end{equation}
\vspace{-1em}
\begin{equation}\label{eq:2.259}\tag{2.259}
    \text{var}[x] = \mu(1 - \mu)
\end{equation}
Show that the entropy $H[x]$ of a Bernoulli distributed
random binary variable $x$ is given by
\begin{equation}\label{eq:2.260}\tag{2.260}
    H[x] = -\mu \ln \mu - (1 - \mu) \ln(1 - \mu)
\end{equation}

\vspace{1em}

\begin{proof}
    The Bernoulli distribution is given by
    \begin{equation}\label{eq:2.2}\tag{2.2}
        \text{Bern}(x | \mu) = \mu^x (1 - \mu)^{1-x}
    \end{equation}
    The properties are easily verified:
    \[
        \sum_{x=0}^{1} p(x | \mu) = p(x = 0 | \mu) + p(x = 1 | \mu)
        = \mu^0(1 - \mu)^1 + \mu^1(1 - \mu)^0 = 1 \tag{2.257}
    \]
    \[
        \mathbb{E}[x] = \sum_{x=0}^{1} xp(x | \mu)
        = 0 \cdot p(x = 0 | \mu) + 1 \cdot p(x = 1 | \mu)
        = \mu \tag{2.258}
    \]
    \[
        \text{var}[x] = \mathbb{E}[x^2] - \mathbb{E}[x]^2
        = \sum_{x=0}^{1} x^2 p(x|\mu) - \mu^2
        = 0^2 \cdot p(x = 0 | \mu) + 1^2 \cdot p(x = 1 | \mu) - \mu^2
        = \mu(1 - \mu) \tag{2.259}
    \]
    The entropy is also straightforward to derive:
    \begin{align*}
        H[x] = -\sum_{x=0}^{1} p(x | \mu) \ln p(x | \mu)
        &= -p(x = 0 | \mu)\ln p(x = 0 | \mu) - p(x = 1 | \mu) \ln p(x = 1 | \mu) \\
        &= -\mu \ln \mu - (1 - \mu) \ln(1 - \mu)
    \end{align*}
\end{proof}

\section*{Exercise 2.2 $\star \star$}
The form of the Bernoulli distribution given by ($\ref{eq:2.2}$)
is not symmetric between the two values of $x$. In some situations,
it will be more convenient to use an equivalent formulation for which
$x \in \{-1 , 1\}$, in which case the distribution can be written
\begin{equation}\label{eq:2.261}\tag{2.261}
    p(x | \mu) = \bigg(\frac{1 - \mu}{2}\bigg)^{(1 - x)/2} \bigg(\frac{1 + \mu}{2}\bigg)^{(1 + x)/2}
\end{equation}
where $\mu \in [-1, 1].$ Show that the distribution ($\ref{eq:2.261}$) is normalized,
and evaluate its mean, variance and entropy.

\vspace{1em}

\begin{proof}
    The distribution is normalized since
    \[
        \sum_{x} p(x | \mu) = p(x = -1 | \mu) + p(x = 1 | \mu)
        = \frac{1 - \mu}{2} + \frac{1 + \mu}{2} = 1
    \]
    The other properties are also easily derived:
    \[
        \mathbb{E}[x] = \sum_{x} xp(x | \mu)
        = p(x = 1 | \mu) - p(x = -1 | \mu)
        = \frac{1 + \mu}{2} - \frac{1 - \mu}{2}
        = \mu
    \]
    \begin{align*}
        \text{var}[x] = \mathbb{E}[x^2] - \mathbb{E}[x]^2
        &= \sum_{x} x^2 p(x | \mu) - \mu^2
        = p(x = -1 | \mu) + p(x = 1 | \mu) - \mu^2 \\
        &= \frac{1+\mu}{2} + \frac{1-\mu}{2} - \mu^2
        = (1 - \mu)(1 + \mu)
    \end{align*}

    \begin{align*}
        H[x] = -\sum_{x} p(x | \mu) \ln p(x | \mu)
        &= -p(x = -1 | \mu) \ln p(x = -1 | \mu) - p(x = 1 | \mu) \ln p(x = 1 | \mu) \\
        &= -\frac{1 - \mu}{2} \ln\bigg(\frac{1 - \mu}{2}\bigg)
        - \frac{1 + \mu}{2} \ln\bigg(\frac{1 + \mu}{2}\bigg)
    \end{align*}
\end{proof}

\section*{Exercise 2.3 $\star \star$}
In this exercise, we prove that the binomial distribution ($\ref{eq:2.9}$)
is normalized. First use the definition (2.10) of the number of combinations of
$m$ identical objects chosen from a total of $N$ to show that
\begin{equation}\label{eq:2.262}\tag{2.262}
    \binom{N}{m} + \binom{N}{m-1} = \binom{N+1}{m}
\end{equation}
Use this result to prove by induction the following result
\begin{equation}\label{eq:2.263}\tag{2.263}
    (1 + x)^N = \sum_{m=0}^{N} \binom{N}{m}x^m
\end{equation}
which is known as the $\emph{binomial theorem}$, and which is valid
for all real values of $x$.
Finally, show that the binomial distribution is normalized, so that
\begin{equation}\label{eq:2.264}\tag{2.264}
    \sum_{m=0}^{N} \binom{N}{m} \mu^m(1 - \mu)^{N - m} = 1
\end{equation}
which can be done by first pulling out a factor $(1 - \mu)^N$ out of the summation
and then making use of the binomial theorem.

\begin{proof}
    The binomial distribution is given by
    \begin{equation}\label{eq:2.9}\tag{2.9}
        \text{Bin}(m | N, \mu) = \binom{N}{m} \mu^m (1-\mu)^{N - m}
    \end{equation}
    By using $(2.10)$, we prove ($\ref{eq:2.262}$)
    \begin{align*}
        \binom{N}{m} + \binom{N}{m - 1}
        &= \frac{N!}{(N - m)!m!} + \frac{N!}{(N - m + 1)!(m - 1)!} \\
        &= \frac{(N - m + 1)N!}{(N - m + 1)!m!} + \frac{mN!}{(N - m + 1)!m!} \\
        &= \frac{(N + 1)!}{(N - m + 1)!m!} \\
        &= \binom{N + 1}{m} \tag{2.262}
    \end{align*}
    We aim to prove ($\ref{eq:2.263}$) by induction. The base
    case for $N=1$ is obviously true since $$1+x = \binom{1}{0} + \binom{1}{1}x$$
    Now, suppose that the case for $N=k \in \mathbb{N}^*$ is true, i.e.
    \[
        (1 + x)^k = \sum_{m=0}^{k} \binom{k}{m} x^m
    \]
    By using this and $(\ref{eq:2.262})$, we show that
    \begin{align*}
        (1 + x)^{k + 1}
        &= (1 + x) \sum_{m=0}^{k} \binom{k}{m} x^m \\
        &= \sum_{m=0}^{k} \binom{k}{m} x^m + \sum_{m=0}^{k} \binom{k}{m} x^{m+1} \\
        &= 1 + \sum_{m=1}^{k} \binom{k}{m} x^m + \sum_{m=1}^{k + 1} \binom{k}{m - 1} x^m \\
        &= \binom{k+1}{0} + \binom{k+1}{k+1}x^{k + 1} + \sum_{m=1}^{k} \bigg\{\binom{k}{m} + \binom{k}{m-1}\bigg\}x^m \\
        &= \sum_{m=0}^{k+1} \binom{k+1}{m} x^m
    \end{align*}
    which by induction proves that $(\ref{eq:2.263})$ is indeed true.

    Finally, we use this result to show that the Binomial distribution is normalized:
    \begin{align*}
        \sum_{m=0}^{N} \binom{N}{m} \mu^m(1 - \mu)^{N - m}
        &= (1 - \mu)^N \sum_{m=0}^{N} \binom{N}{m} \bigg(\frac{\mu}{1 - \mu}\bigg)^m \\
        &= (1 - \mu)^N \bigg(1 + \frac{\mu}{1 - \mu}\bigg)^N \\
        &= 1 \tag{2.264}
    \end{align*}
\end{proof}

\section*{Exercise 2.4 $\star \star$}
Show that the mean of the binomial distribution is given by
($\ref{eq:2.11}$). To do this, differentiate both sides of the normalization condition
($\ref{eq:2.264}$) with respect to $\mu$ and then rearrange to obtain an
expression for the mean of $m$. Similarly, by differentiating ($\ref{eq:2.264}$)
twice with respect to $\mu$ and making use of the result ($\ref{eq:2.11}$) for the mean
of the binomial distribution prove the result (2.12) for the variance of the binomial.

\begin{proof}
    We start by differentiating both sides of ($\ref{eq:2.264}$)
    with respect to $\mu$:
    \begin{align*}
        \pdv{\mu} \sum_{m=0}^{N} \binom{N}{m} \mu^m(1-\mu)^{N - m} &= 0 \\
        \sum_{m=0}^{N} \binom{N}{m} \mu^m(1 - \mu)^{N - m}
        \bigg(\frac{m}{\mu} + \frac{m-N}{1-\mu}\bigg) &= 0 \\
        \bigg(\frac{1}{\mu} + \frac{1}{1 - \mu}\bigg)
        \sum_{m=0}^{N} m \binom{N}{m} \mu^m(1 - \mu)^{N - m}
        - \frac{N}{1-\mu} \sum_{m=0}^{N} \binom{N}{m} \mu^m(1 - \mu)^{N - m} &= 0
    \end{align*}
    We recognize the expression of the binomial distribution and
    use the fact that it is normalized, to obtain:
    \begin{align*}
        \bigg(\frac{1}{\mu} + \frac{1}{1 - \mu}\bigg)
        \sum_{m=0}^{N} m\text{Bin}(m | N, \mu)
        &= \frac{N}{1-\mu} \sum_{m=0}^{N} \text{Bin}(m | N, \mu) \\
        \bigg(\frac{1 - \mu}{\mu} + 1\bigg) \mathbb{E}[m] &= N
    \end{align*}
    which directly gives us the desired result, that is
    \begin{equation}\label{eq:2.11}\tag{2.11}
        \mathbb{E}[m] = \sum_{m=0}^{N} m\text{Bin}(m | N, \mu) = N\mu
    \end{equation}

    To derive the variance, we differentiate twice both sides
    of ($\ref{eq:2.264}$), so
    \begin{align*}
        \pdv[2]{\mu} \sum_{m=0}^{N} \binom{N}{m} \mu^m(1 - \mu)^{N - m} &= 0 \\
        \frac{1}{\mu^2(1 - \mu)^2} \sum_{m=0}^{N} \text{Bin}(m | N, \mu)
            \{m^2 + m(2\mu - 2N\mu - 1) + (N - 1)N\mu^2\} &=0 \\
        \sum_{m=0}^{N} \text{Bin}(m | N, \mu) (m - N\mu)^2
            + (2\mu - 1)\sum_{m=0}^{N} m\text{Bin}(m | N, \mu)
            - N\mu^2 \sum_{m=0}^{N} \text{Bin}(m | N, \mu) &= 0 \\
        \text{var}[m] + (2\mu - 1)\mathbb{E}[m] - N\mu^2 &= 0
    \end{align*}
    which gives us the desired result, i.e.
    \begin{equation}\label{eq:2.12}\tag{2.12}
        \text{var}[m] \equiv \sum_{m=0}^{N} (m - \mathbb{E}[m])^2 \text{Bin}(m | N, \mu) = N\mu(1 - \mu)
    \end{equation}
\end{proof}

\section*{Exercise 2.5 $\star \star$}
In this exercise, we prove that the beta distribution, given by
($\ref{eq:2.13}$), is correctly normalized, so that (2.14) holds.
This is equivalent to showing that
\begin{equation}\label{eq:2.265}\tag{2.265}
    \int_{0}^{1} \mu^{a-1}(1 - \mu)^{b - 1} \diff \mu = \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}
\end{equation}
From the definition ($\ref{eq:1.141}$) of the gamma function,
we have
\begin{equation}\label{eq:2.266}\tag{2.266}
    \Gamma(a)\Gamma(b) = \int_{0}^{\infty} \exp(-x)x^{a - 1} \diff x +
    \int_{0}^{\infty} \exp(-y)y^{b - 1} \diff y
\end{equation}
Use this expression to prove ($\ref{eq:2.265}$) as follows. First
bring the integral over $y$ inside the integrand of the integral
over $x$, next make the change of variable $t = y + x$, where $x$
is fixed, then interchange the order of the $x$ and $t$ integrations,
and finally make the change of variable $x = t\mu$ where $t$ is fixed.

\vspace{1em}

\begin{proof}
    The problem is easily solved by following the provided steps.
    By bringing the integral over $y$ inside the integrand of
    the integral over $x$ we obtain that
    \[
        \Gamma(a)\Gamma(b) = \int_{0}^{\infty} \int_{0}^{\infty}
        \exp\{-(x + y)\} x^{a - 1} y^{b - 1} \diff y \diff x
    \]
    We know use the change of variable $t = y+x$ with $x$ fixed
    to get
    \[
        \Gamma(a)\Gamma(b) = \int_{0}^{\infty} \int_{0}^{\infty}
        \exp(-t) x^{a - 1} (x - t)^{b - 1} \diff t \diff x
    \]
    Interchanging the order of integrations yields
    \[
        \Gamma(a)\Gamma(b) = \int_{0}^{\infty} \int_{0}^{\infty}
        \exp(-t) x^{a - 1} (x - t)^{b - 1} \diff x \diff t
    \]
    which by making the change of variable $x = t\mu$ with $t$ fixed
    becomes
    \[
        \Gamma(a)\Gamma(b) = \int_{0}^{\infty} \int_{0}^{\infty}
        \exp(-t) (t\mu)^{a - 1} (t\mu - t)^{b - 1} t \diff \mu \diff t
    \]
    By separating the $t$ terms from the first integral, we have that
     \[
         \Gamma(a)\Gamma(b) = \int_{0}^{\infty}
         \exp(-t) t^{a + b - 1} \diff t
         \int_{0}^{\infty} \mu^{a - 1}(1 - \mu)^{b - 1} \diff \mu
    \]
    Finally, we notice that the first integral is equal to $\Gamma(a+b)$
    and by noting the fact that $\mu$ is a probability, so its
    range is $[0, 1]$, we obtain the desired result:
    \begin{equation}\tag{2.265}
        \int_{0}^{1} \mu^{a-1}(1 - \mu)^{b - 1} \diff \mu = \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}
    \end{equation}
\end{proof}

\section*{Exercise 2.6 $\star$}
Make use of the result ($\ref{eq:2.265}$) to show that the mean,
variance, and mode of the beta distribution ($\ref{eq:2.13}$) are given
respectively by
\begin{equation}\label{eq:2.267}\tag{2.267}
    \mathbb{E}[\mu] = \frac{a}{a + b}
\end{equation}
\begin{equation}\label{eq:2.268}\tag{2.268}
    \text{var}[\mu] = \frac{ab}{(a + b)^2(a + b + 1)}
\end{equation}
\begin{equation}\label{eq:2.269}\tag{2.269}
    \text{mode}[\mu] = \frac{a - 1}{a + b - 2}
\end{equation}

\vspace{1em}

\begin{proof}
    The beta distribution is given by
    \begin{equation}\label{eq:2.13}\tag{2.13}
        \text{Beta}(\mu | a, b) = \frac{\Gamma(a + b)}{\Gamma(a)\Gamma(b)} \mu^{a - 1}(1 - \mu)^{b - 1}
    \end{equation}
    By using $(\ref{eq:2.265})$ and the fact that $\Gamma(x + 1) = x\Gamma(x)$, we
    obtain the mean of the Beta distribution:
    \begin{align*}
        \mathbb{E}[\mu]
        = \int_{0}^{1} \mu \text{Beta}(\mu | a, b) \diff \mu
        = \frac{\Gamma(a + b)}{\Gamma(a)\Gamma(b)} \int_{0}^{1} \mu^a (1 - \mu)^{b - 1} \diff \mu
        = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} \cdot \frac{\Gamma(a + 1)\Gamma(b)}{\Gamma(a + b + 1)}
        = \frac{a}{a + b}
        \tag{2.267}
    \end{align*}
    From this result, we can also easily get the variance:
    \begin{align*}
        \text{var}[\mu]
        &= \int_{0}^{1} \bigg(\mu - \frac{a}{a + b}\bigg)^2 \text{Beta}(\mu | a,b) \diff \mu \\
        &= \int_{0}^{1} \mu^2 \text{Beta}(\mu | a, b) \diff \mu
        - \frac{2a}{a + b} \int_{0}^{1} \mu\text{Beta}(\mu | a, b) \diff \mu
        + \frac{a^2}{(a + b)^2} \int_{0}^{1} \text{Beta}(\mu | a, b) \diff \mu \\
        &= \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} \cdot \frac{\Gamma(a + 2)\Gamma(b)}{\Gamma(a + b + 2)}
        - \frac{2a}{a+b} \cdot \frac{a}{a + b} + \frac{a^2}{(a+b)^2} \\
        &= \frac{a(a + 1)}{(a + b)(a + b + 1)} - \frac{2a^2}{a + b} + \frac{a^2}{(a + b)^2} \\
        &= \frac{ab}{(a + b)^2(a + b + 1)}
        \tag{2.267}
    \end{align*}
    Finally, the mode of the distribution is given by getting the value of
    $\mu$ for which the derivative of the distribution is 0,
    \begin{align*}
        \pdv{\mu} \text{Beta}(\mu | a, b) = 0
        &\iff \pdv{\mu} \mu^{a - 1}(1 - \mu)^{b - 1} = 0 \\
        &\iff (a - 1)\mu^{a - 2}(1 - \mu)^{b - 1} + (b - 1)\mu^{a - 1}(1 - \mu)^{b - 2} = 0 \\
        &\iff \mu^{a - 2}(1 - \mu)^{b - 2}\{(a - 1)(1 - \mu) + (b - 1)\mu\} = 0 \\
        &\iff (a - 1)(1 - \mu) + (b - 1)\mu = 0 \\
        &\iff \mu = \frac{a - 1}{a + b - 2}
    \end{align*}
    so indeed
    \[
        \text{mode}[\mu] = \frac{a - 1}{a + b - 2} \tag{2.268}
    \]
\end{proof}

\section*{Exercise 2.7 $\star \star$}
Consider a binomial random variable $x$ given by $(\ref{eq:2.9})$, with
prior distribution for $\mu$ given by the beta distribution $(\ref{eq:2.13})$,
and suppose we have observed $m$ occurences of $x=1$ and $l$ occurences
of $x=0$. Show that the posterior mean value of $\mu$ lies between the prior
mean and the maximum likelihood estimate for $\mu$. To do this,
show that the posterior mean can be written as $\lambda$ times the prior
mean plus $(1 - \lambda)$ times the maximum likelihood estimate,
where  $0 \leq \lambda \leq 1$. This illustrates the concept of
the posterior distribution being a compromise between the prior
distribution and the maximum likelihood solution.

\begin{proof}
    The prior mean is $\displaystyle \frac{a}{a + b}$, the posterior mean is
    $\displaystyle \frac{a + m}{a + m + b + l}$ and the maximum likelihood
    estimate is $\displaystyle \frac{m}{m + l}$. Suppose that our hypothesis
    is true, i.e. there exists a $\lambda$ such that we can have our equality
    and $0 \leq \lambda \leq 1$. Then we'd have that:
    \begin{align*}
        \frac{a + m}{a + m + b + l} &= \frac{\lambda m}{m + l} + \frac{(1 - \lambda)a}{a + b} \\
        \frac{a + m}{a + m + b + l} - \frac{a}{a + b} &= \lambda\bigg(\frac{m}{m + l} + \frac{a}{a + b}\bigg) \\
        \lambda &= \frac{bm - al}{(a + b)(a + m + b + l)} \cdot \frac{(a + b)(a + m)}{bm - al} \\
        \lambda &= \frac{l + m}{a + m + b + l}
    \end{align*}
    This $\lambda$ obviously exists and $0 \leq \lambda \leq 1$, so our hypothesis is
    true and the posterior mean value of $x$ lies between the prior mean
    and the maximum likelihood estimate for $\mu$.
\end{proof}

\section*{Exercise 2.8 $\star$}
Consider two variables $x$ and $y$ with joint distribution $p(x, y)$.
Prove the following two results
\begin{equation}\label{eq:2.270}\tag{2.270}
    \mathbb{E}[x] = \mathbb{E}_y[\mathbb{E}_x[x | y]]
\end{equation}
\vspace{-1em}
\begin{equation}\label{eq:2.271}\tag{2.271}
    \text{var}[x] = \mathbb{E}_y[\text{var}_x[x | y]] + \text{var}_y[\mathbb{E}_x[x|y]]
\end{equation}
Here $\mathbb{E}_x[x | y]$ denotes the expectation of $x$ under the conditional
distribution $p(x | y)$, with a similar notation for the conditional
variance.

\vspace{1em}

\begin{proof}
    The first is straightforward to derive:
    \begin{align*}
        \mathbb{E}[x]
        = \iint x p(x, y) \diff x \diff y
        = \iint x p(x | y) p(y) \diff x \diff y
        &= \int\bigg(\int x p(x | y) \diff x\bigg) p(y) \diff y \\
        &= \int \mathbb{E}_x[x | y] p(y) \diff y
        = \mathbb{E}_y[\mathbb{E}_x[x | y]]
        \tag{2.270}
    \end{align*}
    However, proving ($\ref{eq:2.271}$) is slightly more complicated.
    We'll compute each term separately:

\end{proof}

\section*{Exercise 2.10 $\star \star$}
Using the property $\Gamma(x+1) = x\Gamma(x)$ of the gamma function,
derive the following results for the mean, variance, and covariance
of the Dirichlet distribution given by $(\ref{eq:2.38})$
\begin{equation}\label{eq:2.273}\tag{2.273}
    \mathbb{E}[\mu_j] = \frac{\alpha_j}{\alpha_j}
\end{equation}
\begin{equation}\label{eq:2.274}\tag{2.274}
    \text{var}[\mu_j] = \frac{\alpha_j(\alpha_0 - \alpha_j)}{\alpha_0^2(\alpha_0 + 1)}
\end{equation}
\begin{equation}\label{eq:2.275}\tag{2.275}
    \text{cov}[\mu_j\mu_l] = -\frac{\alpha_j\alpha_l}{\alpha_0^2(\alpha_0 + 1)},
    \hspace{3em} j \neq l
\end{equation}
where $\alpha_0$ is defined by  (2.39).

\vspace{1em}

\begin{proof}
    The Dirichlet distribution is given by
    \begin{equation}\label{eq:2.38}\tag{2.38}
        \text{Dir}(\bm{\mu} | \bm{\alpha})
        = \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\ldots\Gamma(\alpha_K)}
        \prod_{k = 1}^K \mu_k^{\alpha_k - 1}
    \end{equation}
    Besides the property that $\Gamma(x+1) = x\Gamma(x)$, we'll be using the fact that the distribution is normalized,
    specifically that
    \[
        \int \prod_{k = 1}^K \mu_k^{\alpha_k - 1} \diff \bm{\mu}
        = \frac{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}{\Gamma(\alpha_0)},
    \]
    where $\alpha_0$ is defined by (2.39).

    The expected value is then given by
    \begin{align*}
        \mathbb{E}[\mu_j]
        &= \int \mu_j \text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \int \mu_1^{\alpha_1 - 1}\dots\mu_j^{\alpha_j}\ldots\mu_K^{\alpha_K - 1} \diff \bm{\mu} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \cdot \frac{\Gamma(\alpha_1)\ldots\Gamma(\alpha_j + 1)\ldots\Gamma(\alpha_K)}
        {\Gamma(\alpha_0 + 1)} \\
        &= \frac{\alpha_j}{\alpha_0}
        \tag{2.273}
    \end{align*}
    This can now be used to derive the variance:
    \begin{align*}
        \text{var}[\mu_j]
        &= \int (\mu_j - \mathbb{E}[\mu_j])^2 \text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu} \\
        &= \int \bigg(\mu_j - \frac{\alpha_j}{\alpha_0}\bigg)^2
        \text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu} \\
        &= \int \mu_j^2 \text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu}
        - \frac{2\alpha_j}{\alpha_0}\int \mu_j\text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu}
        + \frac{\alpha_j^2}{\alpha_0^2} \int \text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \int \mu_1^{\alpha_1 - 1}\dots\mu_j^{\alpha_j + 1}\ldots\mu_K^{\alpha_K - 1} \diff \bm{\mu}
        - \frac{2\alpha_j}{\alpha_0} \mathbb{E}[\mu_j] + \frac{\alpha_j^2}{\alpha_0^2} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \cdot \frac{\Gamma(\alpha_1)\ldots\Gamma(\alpha_j + 2)\ldots\Gamma(\alpha_K)}
        {\Gamma(\alpha_0 + 2)} - \frac{\alpha_j^2}{\alpha_0^2} \\
        &= \frac{\alpha_j(\alpha_j + 1)}{\alpha_0(\alpha_0 + 1)} - \frac{\alpha_j^2}{\alpha_0^2} \\
        &= \frac{\alpha_j(\alpha_0 - \alpha_j)}{\alpha_0^2(\alpha_0 + 1)}
        \tag{2.275}
    \end{align*}
    The covariance is given by
    \[
    \text{cov}[\mu_j\mu_l]
    = \mathbb{E}[\mu_j\mu_l] - \mathbb{E}[\mu_j]\mathbb{E}[\mu_l]
    = \mathbb{E}[\mu_j\mu_l] - \frac{\alpha_j\alpha_l}{\alpha_0^2}
    \]
    By computing the expectation separately, we find that
    \begin{align*}
        \mathbb{E}[\mu_j\mu_l]
        &= \int \mu_j\mu_l \text{Dir}(\bm{\mu} | \bm{\alpha}) \diff \bm{\mu} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \int \mu_1^{\alpha_1 - 1}\dots\mu_j^{\alpha_j}\ldots\mu_l^{\alpha_l}
        \ldots\mu_K^{\alpha_K - 1} \diff \bm{\mu} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \cdot \frac{\Gamma(\alpha_1)\ldots\Gamma(\alpha_j + 1)\ldots\Gamma(\alpha_l + 1)\ldots
        \Gamma(\alpha_K)}{\Gamma(\alpha_0 + 2)}\\
        &= \frac{\alpha_j\alpha_l}{\alpha_0(\alpha_0 + 1)}
    \end{align*}
    Finally, the covariance becomes
    \[
        \text{cov}[\mu_j\mu_l]
        = \frac{\alpha_j\alpha_l}{\alpha_0(\alpha_0 + 1)} - \frac{\alpha_j\alpha_l}{\alpha_0^2}
        = -\frac{\alpha_j\alpha_l}{\alpha_0^2(\alpha_0 + 1)}
        \tag{2.275}
    \]
\end{proof}

\section*{Exercise 2.11 $\star$}
By expressing the expectation of $\ln \mu_j$ under the Dirichlet
distribution ($\ref{eq:2.38}$) as a derivative with respect
to $\alpha_j$, show that
\begin{equation}\label{eq:2.276}\tag{2.276}
    \mathbb{E}[\ln \mu_j] = \psi(\alpha_j) - \psi(\alpha_0)
\end{equation}
where $\alpha_0$ is given by (2.39) and
\begin{equation}\label{eq:2.277}\tag{2.277}
    \psi(a) \equiv \dv{a} \ln \Gamma(a)
\end{equation}
is the $\emph{digamma}$ function.

\vspace{1em}

\begin{proof}
    We start by taking the partial derivative of the Dirichlet
    distribution with respect to $\alpha_j$:
    \begin{align*}
        \pdv{\alpha_j} \text{Dir}(\bm{\mu} | \bm{\alpha})
        &= \pdv{\alpha_j}\bigg(\frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \prod_{k = 1}^K \mu_k^{\alpha_k - 1}\bigg) \\
        &= \bigg(\pdv{\alpha_j}\frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}\bigg) \prod_{k = 1}^K \mu_k^{\alpha_k - 1} +
        \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \bigg(\pdv{\alpha_j}\prod_{k = 1}^K \mu_k^{\alpha_k - 1}\bigg)
    \end{align*}
    Our goal is to compute both terms separately. Firstly, since
    a small change in one of the sum terms is equivalent to a small
    change in the sum itself, i.e.
    \[
        \pdv{\alpha_j} \Gamma(\alpha_0) = \pdv{\alpha_0} \Gamma(\alpha_0)
    \]
    we have that
    \begin{align*}
        \pdv{\alpha_j} \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_j)}
        &= \frac{\pdv{\alpha_j}\Gamma(\alpha_0) - \pdv{\alpha_j}\Gamma(\alpha_j)}{\Gamma(\alpha_j)^2} \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_j)} \bigg(\frac{\pdv{\alpha_j} \Gamma(\alpha_0)}
            {\Gamma(\alpha_0)} - \frac{\pdv{\alpha_j} \Gamma(\alpha_j)}{\Gamma(\alpha_j)}\bigg) \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_j)}
        \bigg(\pdv{\alpha_0} \ln \Gamma(\alpha_0) - \pdv{\alpha_j} \ln \Gamma(\alpha_j)\bigg) \\
        &= \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_j)}(\psi(\alpha_0) - \psi(\alpha_j))
    \end{align*}
    and therefore, that
    \[
        \bigg(\pdv{\alpha_j}\frac{\Gamma(\alpha_0)}
        {\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}\bigg)
        \prod_{k = 1}^K \mu_k^{\alpha_k - 1}
        = (\psi(\alpha_0) - \psi(\alpha_j))\text{Dir}(\bm{\mu} | \bm{\alpha})
    \]
    Now, since
    \[
        \pdv{\alpha_j} \prod_{k = 1}^K \mu_k^{\alpha_k - 1}
        = (\mu_1^{\alpha_1 - 1}\ldots\mu_{j - 1}^{\alpha_{j - 1} - 1}
        \mu_{j + 1}^{\alpha_{j + 1} - 1}
        \ldots \mu_K^{\alpha_K - 1}) \pdv{\alpha_j} \mu_j^{\alpha_j - 1}
        = \ln \mu_j \prod_{k = 1}^K \mu_k^{\alpha_k - 1}
    \]
    it follows that
    \[
        \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\ldots\Gamma(\alpha_K)}
        \bigg(\pdv{\alpha_j}\prod_{k = 1}^K \mu_k^{\alpha_k - 1}\bigg)
        = \ln \mu_j \text{Dir}(\bm{\mu} | \bm{\alpha})
    \]
    By substituting into the initial expression,
    \[
        \pdv{\alpha_j} \text{Dir}(\bm{\mu} | \bm{\alpha})
        = \text{Dir}(\bm{\mu} | \bm{\alpha}) (\ln \mu_j + \psi(\alpha_0) - \psi(\alpha_j))
    \]
    and then integrating with respect to $\bm{\mu}$, we obtain the
    desired result:
    \[
        \mathbb{E}[\ln{\mu_j}] = \psi(\alpha_j) - \psi(\alpha_0)
        \tag{2.276}
    \]
\end{proof}


\textbf{Consider again the approximation of $f(x)=3/(5−4cos(x))$, $x\in[-\pi,\pi]$. Let $N$ be the number of nodes in Fourier and polynomial interpolation of this function.
\begin{enumerate}[label=\alph*)]
\item Plot the error as a function of $N$ (on the same figure) for both Chebyshev and Fourier. Notice
that Fourier converges at a faster rate in this case.
\item Now consider that maximum spacing between nodes: $h=max|x_{i+1}-x_i|$. Plot the error for
polynomial and Fourier approximations as a function of $h$ and notice that the rates of convergence
are now nearly the same.
\item Show that the ratio $h_{cheb}/h_{Fourier}$ is about $\pi/2$.
\end{enumerate}
$~$}
\newline

For the first part we look at the next figure. We can see that, in fact, Fourier converges at a faster rate in this function.

\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{P7_a.png}\caption{Convergence of the Chebyshev and Fourier interpolants to $f(x)= \frac{3}{54\cos{x}}$.}
\end{figure}

We continue by scaling the Chebyshev points to be withing $[-\pi,\pi]$ and calculate $h$ for both Chebishev and Fourier, for each $N$. We obtain the following figure, which shows that the rates of convergence are nearly the same.
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{P7_b.png}\caption{Convergence of the Chebyshev and Fourier interpolants to $f(x)= \frac{3}{54\cos{x}}$.}
\end{figure}

Lastly, in the following figure we see that, once is large enough, $h_{Cheb}/h_{Fourier}\approx \pi/2$.

\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{P7_c.png}\caption{$h_{Cheb}/h_{Fourier}$ for $f(x)= \frac{3}{54\cos{x}}$.}
\end{figure}

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 7
close all
f = chebfun('3/(5-4*cos(x))',[-pi,pi]);
plot(f)
grid on
N = 2:2:50;
for k = 1:length(N)
    fcheb = chebfun('3/(5-4*cos(x))',[-pi,pi],N(k));
    ffour = chebfun('3/(5-4*cos(x))',[-pi,pi],N(k),"trig");
    errcheb(k) = norm(f-fcheb,inf);
    errfour(k) = norm(f-ffour,inf);
    % b
    [~,x] = cheb(N(k)); x = pi*x;
    hcheb(k) = max(abs(x(2:end)-x(1:end-1)));
    hfour(k) = 2*pi/(N(k));
end
% a
figure
semilogy(N,errcheb,'b',N,errfour,'r')
hold on
semilogy(N,errcheb,'b*',N,errfour,'r*')
grid on
xlabel('$N$','interpreter','latex')
ylabel('$Error$','interpreter','latex')
set(gca,'fontsize',labelfontsize)
legend('Chebishev', 'Fourier')
txt='Latex/FIGURES/P7_a';
saveas(gcf,txt,figformat)
% b
figure
semilogy(hcheb.^(-1),errcheb,'b',hfour.^(-1),errfour,'r')
hold on
semilogy(hcheb.^(-1),errcheb,'b*',hfour.^(-1),errfour,'r*')
grid on
xlabel('$1/h$','interpreter','latex')
ylabel('$Error$','interpreter','latex')
set(gca,'fontsize',labelfontsize)
legend('Chebishev', 'Fourier')
txt='Latex/FIGURES/P7_b';
saveas(gcf,txt,figformat)
% c
figure
plot(N,hcheb./hfour,'r*')
grid on
axis([0 50 0 pi])
xlabel('$N$','interpreter','latex')
ylabel('$h_{Cheb}/h_{Fourier}$','interpreter','latex')
set(gca,'fontsize',labelfontsize)
legend('Chebishev', 'Fourier')
txt='Latex/FIGURES/P7_c';
saveas(gcf,txt,figformat)
\end{verbatim}


\section*{Exercises}

\begin{ex}
  Find the matrix for the linear transformation that reflects every
  vector in $\R^2$ about the $x$-axis and then reflects about the
  $y$-axis.
\end{ex}

\begin{ex}
  Find the matrix for the linear transformation that rotates every
  vector in $\R^2$ by an angle of $2\pi/3$ and then reflects
  about the $x$-axis.
  \begin{sol}
    \begin{equation*}
      \def\arraystretch{1.2}
      \begin{mymatrix}{rr}
        1 & 0 \\
        0 & -1
      \end{mymatrix} \begin{mymatrix}{cc}
        \cos \paren{\frac{2\pi}{3}}  & -\sin \paren{\frac{2\pi}{3}}
        \\
        \sin \paren{\frac{2\pi}{3}}  & \cos \paren{\frac{2\pi}{3}}
      \end{mymatrix} = \begin{mymatrix}{cc}
        -\frac{1}{2} & -\frac{1}{2}\sqrt{3} \\
        -\frac{1}{2}\sqrt{3} & \frac{1}{2}
      \end{mymatrix}
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Find the matrix for the linear transformation that rotates every
  vector in $\R^2$ by an angle of $\pi /6$ and then reflects
  about the $x$-axis followed by a reflection about the $y$-axis.
  \begin{sol}
    \begin{equation*}
      \def\arraystretch{1.2}
      \begin{mymatrix}{rr}
        -1 & 0 \\
        0 & 1
      \end{mymatrix} \begin{mymatrix}{cc}
        \cos \paren{\frac{\pi}{6}}  & -\sin \paren{\frac{\pi}{6}}  \\
        \sin \paren{\frac{\pi}{6}}  & \cos \paren{\frac{\pi}{6}}
      \end{mymatrix} = \begin{mymatrix}{cc}
        -\frac{1}{2}\sqrt{3} & \frac{1}{2} \\
        \frac{1}{2} & \frac{1}{2}\sqrt{3}
      \end{mymatrix}
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Find the matrix for the linear transformation that reflects every
  vector in $\R^2$ about the $x$-axis and then rotates by an angle
  of $\pi/4$.
  \begin{sol}
    \begin{equation*}
      \def\arraystretch{1.2}
      \begin{mymatrix}{cc}
        \cos \paren{\frac{\pi}{4}}  & -\sin \paren{\frac{\pi}{4}}  \\
        \sin \paren{\frac{\pi}{4}}  & \cos \paren{\frac{\pi}{4}}
      \end{mymatrix} \begin{mymatrix}{rr}
        1 & 0 \\
        0 & -1
      \end{mymatrix} = \begin{mymatrix}{cc}
        \frac{1}{2}\sqrt{2} & \frac{1}{2}\sqrt{2} \\
        \frac{1}{2}\sqrt{2} & -\frac{1}{2}\sqrt{2}
      \end{mymatrix}
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Find the matrix of the linear transformation that rotates every
  vector in $\R^3$ counterclockwise about the $z$-axis when viewed
  from the positive $z$-axis by an angle of 30 degrees and then
  reflects about the $xy$-plane.
  \begin{sol}
    \begin{equation*}
      \def\arraystretch{1.2}
      \begin{mymatrix}{rrr}
        1 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & -1
      \end{mymatrix} \begin{mymatrix}{ccc}
        \cos \paren{\frac{\pi}{6}}  & -\sin \paren{\frac{\pi}{6}}  & 0
        \\
        \sin \paren{\frac{\pi}{6}}  & \cos \paren{\frac{\pi}{6}}  & 0
        \\
        0 & 0 & 1
      \end{mymatrix} = \begin{mymatrix}{ccc}
        \frac{1}{2}\sqrt{3} & -\frac{1}{2} & 0 \\
        \frac{1}{2} & \frac{1}{2}\sqrt{3} & 0 \\
        0 & 0 & -1
      \end{mymatrix}
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Prove the three properties in
  Proposition~\ref{prop:properties-linear-transformation}, using only
  the definition of a linear transformation (i.e., the fact that it
  preserves addition and scalar multiplication).
  \begin{sol}
    (a) $T(\vect{0}) = T(0\vect{0}) = 0T(\vect{0}) = \vect{0}$.
    (b) $T(-\vect{v}) = T((-1)\vect{v}) = (-1)T(\vect{v}) = -T(\vect{v})$.
    (c) $T(a_1\vect{v}_1 + \ldots + a_k \vect{v}_k)
    = T(a_1\vect{v}_1) + \ldots + T(a_k \vect{v}_k)
    = a_1T(\vect{v}_1) + \ldots + a_kT(\vect{v}_k)$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be the linear transformation with matrix
  $A = \begin{mymatrix}{rr}
    3 & 1 \\
    -1 & 2
  \end{mymatrix}$ and $S$ the linear transformation with matrix
  $B = \begin{mymatrix}{rr}
    0 & -2 \\
    4 & 2
  \end{mymatrix}$. Find the matrix of $S \circ T$. Compute
  $(S \circ T) (\vect{v})$ for
  $\vect{v} = \begin{mymatrix}{r}
    2 \\
    -1
  \end{mymatrix}$.
  \begin{sol}
    The matrix of $S \circ T$ is given by
    \begin{equation*}
      BA = \begin{mymatrix}{rr}
        0 & -2 \\
        4 & 2
      \end{mymatrix} \begin{mymatrix}{rr}
        3 & 1 \\
        -1 & 2
      \end{mymatrix} = \begin{mymatrix}{rr}
        2 & -4 \\
        10 & 8
      \end{mymatrix}.
    \end{equation*}
    Now,
    \begin{equation*}
      (S \circ T) (\vect{v}) = BA\vect{v}
      =
      \begin{mymatrix}{rr}
        2 & -4 \\
        10 & 8
      \end{mymatrix}
      \begin{mymatrix}{r}
        2 \\
        -1
      \end{mymatrix}
      =
      \begin{mymatrix}{r}
        8 \\
        12
      \end{mymatrix}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be a linear transformation and suppose
  $T \paren{\begin{mymatrix}{r}
      1 \\
      -4
    \end{mymatrix}} = \begin{mymatrix}{r}
    2 \\
    -3
  \end{mymatrix}$. Suppose $S$ is the linear transformation with
  matrix $B = \begin{mymatrix}{rr}
    1 & 2 \\
    -1 & 3
  \end{mymatrix}$. Find $(S \circ T) (\vect{v})$ for
  $\vect{v} = \begin{mymatrix}{r}
    1 \\
    -4
  \end{mymatrix}$.
  \begin{sol}
    We have
    \begin{equation*}
      (S \circ T) (\vect{v})
      = S(T(\vect{v}))
      = B(T(\vect{v}))
      = \begin{mymatrix}{rr}
        1 & 2 \\
        -1 & 3
      \end{mymatrix}
      \begin{mymatrix}{r}
        2 \\
        -3
      \end{mymatrix}
      = \begin{mymatrix}{r}
        -4 \\
        -11
      \end{mymatrix}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  What is the inverse of a reflection? Rotation? Shearing? Scaling?
  \begin{sol}
    The inverse of a reflection is a reflection, namely, itself. (For
    example, reflecting twice about the $x$-axis returns each vector
    to its original position). The inverse of a rotation is a rotation
    by the same angle in the opposite direction. The inverse of a
    shearing is a shearing in the opposite direction. The inverse of a
    scaling by factor $a$ is a scaling by factor $1/a$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be a linear transformation with matrix
  $A = \begin{mymatrix}{rr}
    2 & 1 \\
    5 & 2
  \end{mymatrix}$. Find the matrix of $T^{-1}$.
  \begin{sol}
    The matrix of $T^{-1}$ is $A^{-1}$.
    \begin{equation*}
      \begin{mymatrix}{rr}
        2 & 1 \\
        5 & 2
      \end{mymatrix}^{-1} =
      \begin{mymatrix}{rr}
        -2 & 1 \\
        5 & -2
      \end{mymatrix}
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Let $T$ be the linear transformation given by
  $T\paren{\begin{mymatrix}{c} x\\y \end{mymatrix}} =
  \begin{mymatrix}{c} 4x-3y\\2x-2y \end{mymatrix}$.  Find the matrix
  of $T^{-1}$.
  \begin{sol}
    The matrix of $T$ is
    $A = \begin{mymatrix}{rr}
    4 & -3 \\
    2 & -2
  \end{mymatrix}$. The matrix of $T^{-1}$ is $A^{-1} =
  \begin{mymatrix}{cc}
    1 & -3/2 \\
    1 & -2
  \end{mymatrix}$.
  \end{sol}
\end{ex}

\begin{ex} Let $T$ be a linear transformation and suppose $T \paren{\begin{mymatrix}{r}
      1 \\
      2
    \end{mymatrix}} = \begin{mymatrix}{r}
    9 \\
    8
  \end{mymatrix}$ and $T \paren{\begin{mymatrix}{r}
      0 \\
      -1
    \end{mymatrix}} = \begin{mymatrix}{r}
    -4 \\
    -3
  \end{mymatrix}$.
  Find the matrix of $T$ and the matrix of $T^{-1}$.
  % \begin{sol}
  % \end{sol}
\end{ex}


\subsection{Partial Adjustment Model}

\subsubsection{Estimating a static model}

We start by estimating a static model.

\(y_t=\alpha + \theta x_t + \gamma_t\)

\subsubsection{Equilibrium}

We then use this form an equilibrium for \(y_t\), \(y_t^*\).

\(y_t^*=\hat \alpha + \hat \theta x_t \)

The process depends on the difference from this equilibrium.

\(y_t-y_{t-1}=\beta (y_{t}^*-y_{t-1})+\epsilon_t \)

\(y_t-y_{t-1}=\beta (\hat \alpha + \hat \theta x_t -y_{t-1})+\epsilon_t \)

\(y_t=\beta \hat \alpha + \beta \hat \theta x_t + (1-\beta )y_{t-1}+\epsilon_t \)

\(y_t=\alpha y_{t-1}+(1-\beta )(y_{t}^*-y_{t-1})+\epsilon \)

The higher \(\beta \), the slower the adjustment.

If stationary, can we can use OLS.


\subsection{Permutations and the symmetric group}

A permutation is defined as a bijection from a set to itself.

For a set of size \(n\), the number of permutations is \(n!\). This is because there are \(n\) possibilities for the first item, \(n-1\) for the second and so on.

\subsubsection{The symmetric group}

The set of all permutations forms a group, the symmetric group. This forms a group because:

\begin{itemize}
\item There is an identity element
\item Each combination of permutations is also in the group.
\item Each permutation has an inverse in the group.
\end{itemize}

\subsubsection{Permutation groups}

A subgroup of the symmetric group is called a permutation group.


\section{The Elementary Theory of Initial-Value Problems}
\begin{defn}[Lipschitz Condition]
A function $f(t,y)$ is said to satisfy a \emph{Lispschitz condition} in the variable $Y$ on a set $D\subset\mathbb{R}^2$ if a constant $L>0$ exists with
\[
\abs{f(t,y_1)-f(t,y_2)}\leq L\abs{y_1-y_2}
\]
whenever $(t_1,y_1)$, $(t_2,y_2)$ are in $D$. The constant $L$ is called a \emph{Lipschitz constant} for $f$.
\end{defn}
\begin{defn}[Convex]
A set $D\subset\mathbb{R}^2$ is said to be convex if whenever $(t_1,y_1),(t_2,y_2)\in D$, then for every $\lambda\in [0,1]$,
\[
((1-\lambda)t_1+\lambda t_2, (1-\lambda)y_1+\lambda y_2) \in D.
\]
\end{defn}
\begin{theo}
Suppose $f(t,y)$ is defined on a convex set $D\subset\mathbb{R}^2$, if a constant $L>0$ exists with
\[
\abs{\frac{\partial f}{\partial y}(t,y)}\leq L
\]
for all $(t,y)\in D$, then $f$ satisfies a Lipschitz condition on $D$ in the variable $y$ with Lipschitz constant $L$.
\end{theo}
\begin{theo}
Suppose that $D=\left\{(t,y)|a\leq t\leq b,y\in\mathbb{R}\right\}$ and $f(t,y)$ is continuous on $D$. If $f$ satisfies a Lipschitz condition on $D$ in the variable $y$, then the initial-value problem
\begin{align*}
  \begin{cases}
    y'(t)=f(t,y) & a\leq t\leq b \\
    y(a)=\alpha
  \end{cases}
\end{align*}
has a unique solution $y(t)$ for $a\leq t\leq b$.
\end{theo}
\subsection{Well-Posed Problems}
\begin{theo}[Well-Posed]
Suppose that $D=\left\{(t,y)|a\leq t\leq b,y\in\mathbb{R}\right\}$ and $f(t,y)$, if $f$ is continuous and satisfies a Lipschitz condition in the variable $y$ on the set $D$, then the initial-value problem
\[
\frac{\D y}{\D t}=f(t,y),\quad a\leq t\leq b,\quad y(a)=\alpha
\]
is well-posed.
\end{theo}


\subsection{Solving Inequalities that Contain Absolute Values}
We start by solving an equality that contains an absolute value.
To do so, we recall that if $a\geq 0$ then the solution to
$|x|=a$ is $x=\pm a$. In cases where we are not sure if the right
side is positive or negative, we must perform a check at the end. \\

\begin{example}{Absolute Value Equality}{AbsoluteValueEquality}
Solve for $x$ in $|2x+3|=2-x$.
\end{example}

\begin{solution}
This means that either:
\[\begin{array}{rclcrcl}
	2x+3 & = & 2-x & \hspace{5mm} \text{or} \hspace{5mm} & \hspace{3mm} 2x+3 & = & -(2-x)\\
	2x+3 & = & 2-x &  & 2x+3 & = & -2+x\\
	3x & = & -1 &  & x & = & -5\\
	x & = &-1/3 &  & x & = & -5\\
\end{array}\]
Since we do not know if the right side $``2-x"$ is positive or negative, we must perform a check of our answers and omit any that are incorrect.\\

If $x=-1/3$, then we have

\hspace{4cm} $\begin{array}{rcl}
\displaystyle{\left| 2 (-1/3) +3 \right|} & = & \displaystyle{2-\left( -1/3 \right)} \\
\displaystyle{\left| -2 /3 \, +3\right|} & = & \displaystyle{2+ \, 1/3 } \\
	\displaystyle{ \left| 7 / 3 \right|} & = & \displaystyle{ 7/3 } \\
	\displaystyle{ 7/3 } & = &  \displaystyle{ 7/3 } \\
\end{array} $

% \left( \frac{-1}{3} \right)+3\right|}

Since the left hand side of the equation equals the right hand side, $x=-1/3$ is a solution.\\

Now checking to see if $x=-5$ satisfies the expression. We have

\hspace{4cm} $\begin{array}{rcl}
\displaystyle{\left| 2 (-5) +3 \right|} & = & \displaystyle{2-\left( -5 \right)} \\
\displaystyle{\left| -10 \, +3\right|} & = & \displaystyle{2+ \, 5 } \\
	\displaystyle{ \left| -7 \right|} & = & \displaystyle{ 7 } \\
	\displaystyle{ 7 } & = &  \displaystyle{ 7 } \\
\end{array} $

Since the left hand side equals the right hand side,  $x=-5$ is a solution.\\
Therefore, the solution to the absolute value equality is $x=-5, \, -1/3$.
\end{solution}

We next look at absolute values and inequalities. \\

\begin{example}{Absolute Value Inequality}{AbsoluteValueInequality}
Solve $|x-5|<7$.
\end{example}

\begin{solution}
This simply means $-7<x-5<7$.
Adding $5$ to each gives $-2<x<12$.
Therefore the solution is the interval $(-2,12)$.
\end{solution}

In some questions you must be careful when multiplying by a negative number as in the next problem. \\

\begin{example}{Absolute Value Inequality}{AbsoluteValueInequality2}
Solve $|2-z|<7$.
\end{example}

\begin{solution}
This simply means $-7<2-z<7$.
Subtracting $2$ gives: $-9<-z<5$.
Now multiplying by $-1$ gives: $9>z>-5$. \ifont{Remember to reverse the inequality signs!}
We can rearrange this as $-5<z<9$.
Therefore the solution is the interval $(-5,9)$.
\end{solution}

\bigskip

\begin{example}{Absolute Value Inequality}{AbsoluteValueInequality3}
Solve $|2-z|\geq 7$.
\end{example}

\begin{solution}
Recall that for $a>0$, $|x|\geq a$ means $x\leq -a$ or $x\geq a$.
Thus, either $2-z\leq -7$ \ifont{or} $2-z\geq 7$.
Either $9\leq z$ \ifont{or} $-5 \geq z$.
Either $z\geq 9$ \ifont{or} $z \leq -5$.
In interval notation, either $z$ is in $[9,\infty)$ \ifont{or} $z$ is in $(-\infty,-5]$.
All together, we get our solution to be: $(-\infty,-5]\cup [9,\infty)$.
\end{solution}

In the previous two examples the \ifont{only} difference is that one had $<$ in
the question and the other had $\geq$. Combining the two solutions gives the
\ifont{entire} real number line! \\

\begin{example}{Absolute Value Inequality}{AbsoluteValueInequality4}
Solve $0<|x-5|\leq 7$.
\end{example}

\begin{solution}
We split this into two cases.

(1) For $0<|x-5|$ note that we always have that an absolute value is positive or zero (i.e., $0\leq |x-5|$ is always true).
So, for this part, we need to avoid $0=|x-5|$ from occurring.
Thus, $x$ \ifont{cannot} be $5$, that is, $x\neq 5$.\\

(2) For $|x-5|\leq 7$, we have $-7\leq x-5\leq 7$.
Adding $5$ to each gives $-2\leq x\leq 12$.
Therefore the solution to $|x-5|\leq 7$ is the interval $[-2,12]$.

To combine (1) and (2) we need combine $x\neq 5$ with $x\in[-2,12]$.
Omitting $5$ from the interval $[-2,12]$ gives our solution to be: $[-2,5)\cup(5,12]$.
\end{solution}


\chapter{Probability}\label{sec:probability}
This course will use probability theory quite a lot, but we will often use
a fairly informal notation. Bayesian
statistics is really just probability theory used for a particular purpose,
to describe uncertainty. The two most important rules of probability are given below,
for reference.

\section{The Product Rule}
The first important rule of probability is the
product rule. This tells us how to calculate the probability that any two
propositions or hypotheses, $A$ and $B$, are {\bf both} true.
The probability of $A$ {\bf and}
$B$, will be denoted $P(A, B)$. This can be calculated by first finding the
probability that $A$ is true, and then multiplying by the probability that $B$
is true {\it given} that $A$ is true.
\begin{eqnarray}
P(A, B) &=& P(A)P(B|A)\label{product1}
\end{eqnarray}
We could also have done this the other way around: first finding the
probability that $B$ is true and then the probability that $A$ is true given
that $B$ is true:
\begin{eqnarray}
P(A, B) &=& P(B)P(A|B).\label{product2}
\end{eqnarray}

When using the product rule (or any rule of probability, for that matter), you
must ensure that the statements to the right of the ``given'' sign (or the
absence of any statements) are consistent throughout. For example,
$P(A, B|C) = P(A|C)P(B|A, C)$ is a valid use of the product rule, since
``given $C$'' is part of the background information in all of the terms.

You may be familiar with the idea of a {\it tree diagram} from earlier
statistics courses or maybe even high school. A tree diagram is a helpful way
to work with the product rule. If you find tree diagrams helpful, feel free to
use them, although tree diagrams themselves will not be examinable.
An example tree diagram is given in Figure~\ref{fig:tree_diagram}.

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.9]{Figures/tree_diagram.pdf}
\caption{\it A tree diagram.\label{fig:tree_diagram}}
\end{center}
\end{figure}

The product rule can also be applied to more than two propositions, like so:
\begin{eqnarray}
P(A, B, C) &=& P(A)P(B|A)P(C|B,A).
\end{eqnarray}
You can also apply the product rule in a situation where there is a common
statement in the ``given part'' of all probabilities in the expression. For
example, the following is also valid:
\begin{eqnarray}
P(A, B, C | D) &=& P(A |D)P(B|A,D)P(C|B,A,D).
\end{eqnarray}
In fact, it's best to regard even ``unconditional'' probabilities such
as $P(A)$ as being conditional on some prior information $I$, which is just
left out to keep the notation simple.

\subsection{Bayes' Rule}
Looking at Equations~\ref{product1} and~\ref{product2}, they are both equations
for the same
thing, $P(A,B)$. Therefore we can equate the right hand sides. Doing this gives
a result known as Bayes' rule:
\begin{eqnarray}
P(A|B) &=& \frac{P(A)P(B|A)}{P(B)}. \label{bayes}
\end{eqnarray}
Bayes' rule will be used extensively throughout this course. You will need to
know it and know how to use it!

\section{The Sum Rule}
The sum rule is the second important rule of probability. A general statement
of the sum rule is
\begin{align}
P(A \vee B) &= P(A) + P(B) - P(A, B).\label{eqn:sum_rule}
\end{align}
where $\vee$ means logical {\em or}.

The sum rule is often used to calculate the
probability of some statement
$A$ when we only know the probability of $A$ conditional on some
some other statement $B$. Then we
can use the sum rule like this:
\begin{eqnarray}
P(A) &=& P(A, B) + P(A, \neg B)\\
&=& P(B)P(A|B) + P(\neg B)P(A|\neg B).
\end{eqnarray}
where the $\neg$ symbol means ``not'', i.e. $\neg B$ is the statement that
$B$ is false. To understand
this formula, imagine we want to know the probability of $A$. There are
two mutually exclusive ways that could happen: via $B$ being true, or via $B$
being false. The first way has probability $P(B)P(A|B)$, and the second way
has probability $P(\neg B)P(A|\neg B)$.

If, instead of just two mutually exclusive and exhaustive
pathways $B$ and $\neg B$, there are many, such as $B_1, ..., B_n$.
Then the sum rule takes the form
\begin{eqnarray}
P(A) &=& \sum_{i=1}^n P(B_i)P(A|B_i).
\end{eqnarray}
As an exercise, you can try proving this version of the sum rule
starting from the simpler version of Equation~\ref{eqn:sum_rule}.

In Bayesian statistics the sum rule is most often used to calculate the
marginal likelihood $P(D)$, and to marginalise out ``nuisance
parameters'' from the posterior distribution. In STATS 331 we will mostly use
MCMC to do the latter.

\section{Random Variables}
Throughout this course I will use the term ``probability distribution'' to
refer to both the probability mass function for a discrete random variable, and
the probability density function for a continuous random variable. I will also
use a common shorthand notation.

\subsection{Discrete Random Variables}
We will also see quite a lot of random variables in this course (although
without using that terminology very much, as I consider the word ``random'' to be
worse than useless). A discrete
random variable is just a quantity $X$ that has a countable number of possible
values. A discrete random variable has a {\it probability mass function}
that tells you the probability as a function of the {\it possible} values $x$.
For example, the equation for the Poisson distribution (a useful discrete
distribution) is:
\begin{eqnarray}
P(X=x) &=& \frac{\lambda^x e^{-\lambda}}{x!}\label{eq:poisson}
\end{eqnarray}
for $x \in \{0, 1, 2, 3, ...\}$. The actual random variable is named $X$, and
$x$ is just used so we can write the probabilities ($P(X=0), P(X=1), ...)$ as
a formula.

\subsection{Continuous Random Variables}
Continuous random variables are those where the set of possibilities is
continuous. For example, with a normal distribution, technically any real value
is possible. Therefore it doesn't make sense to ask, for example, the probability
that $X=1.32$. The answer is zero because the total probability of 1 has to be
spread among an infinite number of possibilities. Instead, we can ask the probability
that $X$ is in some region that has a nonzero size. In general, if $X$ has a
probability density function $f(x)$, then the probability that $X \in [a, b]$ is:
\begin{eqnarray}
P(a \leq X \leq b) &=& \int_a^b f(x) \, dx.
\end{eqnarray}
Note the lower case $x$ in the probability density function. This is analogous
to the lower case $x$ in the probability mass function of a discrete random
variable. Note that I won't often get you to do an integral analytically. One
of the major reasons why MCMC is so awesome is that you can get away without
having to do hard integrals!

\subsection{Shorthand Notation}
The notation of random variables can be cumbersome. For example, consider
inferring a (discrete) parameter $Y$ from (discrete) data $X$. Bayes' rule gives us:
\begin{eqnarray}
P(Y=y | X=x) &=& \frac{P(Y=y)P(X=x|Y=y)}{P(X=x)}.
\end{eqnarray}
That's very verbose, so instead we use the shorthand:
\begin{eqnarray}
p(y | x) &=& \frac{p(y)p(x|y)}{p(x)}.
\end{eqnarray}
In this notation we don't distinguish between the symbol for the random variable
and the dummy variables that allow us to write the probability distribution as
a formula, we just use the lower case for everything. Despite this simplification,
everything still works. Just read
$p(y|x)$ as ``the probability distribution for $y$ given $x$'' and everything
will be fine. Astute readers may have noticed that when I gave the Poisson formula
(Equation~\ref{eq:poisson}), ``given $\lambda$'' was implicit throughout.

To be clear when we are talking about a simple probability and when we are
talking about the probability distribution for a variable, I will use upper case
$P$ for the former and lower-case $p$ for the latter.

\section{Useful Probability Distributions}
In the course we will study Bayesian models which involve the following
discrete probability distributions: general (all probabilities given explicitly,
such as in a Bayes' Box), binomial, Poisson, discrete uniform,
negative binomial, multinomial.
We may also use the following continuous distributions: uniform, normal,
Cauchy, student-$t$, beta, log-uniform, exponential, gamma, Dirichlet.

Wikipedia is an excellent resource for looking up all the properties of these
distributions, but I will also give various properties (e.g. the mean of a beta
distribution) and describe the distributions when we need them.


\section{Linear Mappings}

\subsection{Exercise 1}
(a) $x \in X \implies x = \sum_{i=1}^{n} k_i x_i \implies T(x) = \sum_{i=1}^{n} k_i T(x_i) \in U$.

(b) $T(x), T(y) \in U \implies T(x + y) \in U \implies x + y \in X$.

\subsection*{Theorem 1}
$x \in X, y \in N_T \implies T(x + y) = T(x) + T(y) = T(x)$.

\subsection{Exercise 2}
(a) Differentiation constant and sum rules imply linearity, and multiplication by $s$ is distributive.
Take $p(s) = 1$ to see that $ST \neq TS$.

(b) Rotation by 90 degrees amounts to swapping and negating coordinates, which is linear.
Take $p = (1, 1, 0)$ to see that $ST \neq TS$.

\subsection{Exercise 3}
(i) $T^{-1} (T(a + b)) = T^{-1} (T(a) + T(b)) = a + b = T^{-1} (T(a)) + T^{-1} (T(b))$.

(ii) Composition of isomorphisms is an isomorphism, hence $ST$ is invertible.

\subsection{Exercise 4}
(i) Let $T: X \to U,\: S: U \to V$ and $l_v \in V'$.
Then $(ST)' (l_v) = l_v (ST) = (l_v S) T = (S' l_v) T = T' S' l_v$,
since $S' l_v \in U'$.

(ii) Follows from linearity of transpose (definition).

(iii) Let $T: X \to U$ be an isomorphism. Then $l_x = l_u T \implies l_x T^{-1} = l_u$
for $l_u \in U', \: l_x \in X'$.

\subsection{Exercise 5}
$T''(l_{x'}) = l_{x'}T'$ where $l_{x'} \in X''$ and $l_{x'}T' \in U''$.
Since we can identify elements in $X''$ and $U''$ with elements in $X$ and $ U$
respectively, we have that $T''$ assigns elements of $U$ to $X$.

\subsection*{Theorem 2'}
Since $T': U' \to X'$ we have $l_u \in N_{T'} \implies T'(l_u) = l_u T = 0$.
$N_{T'}^{\perp}$ consists of elements $l_{u'} | l_{u'} (l_u) = 0$.
From $l_u Tx = 0$ we have that each $l_{u'}$ is identified with a $u \in R_T$.

\subsection{Exercise 6}
The first two elements of $x$ are already 0 after applying $P$, so $P^2 = P$.
Linearity follows from linearity of vector addition.

\subsection{Exercise 7}
$P$ is linear since function addition is linear.
$P^2 f = \frac{f(x) + f(-x)}{4} + \frac{f(x) + f(-x)}{4} = P f$.


\section{Fields}
We begin our dive into the rigorous analysis of linear algebra by defining an algebraic construction call a \emph{field}, which we need in order to properly define vector spaces later. In essence, a field has most of the important properties of the real numbers, namely the closure, commutativity, associativity, identity and inverse of addition and multiplication of any two elements in the field (except the product inverse of the field equivalent object for the number $0$). In a later section we will use fields to construct the general notion of \emph{vector spaces}.

\setenumerate[1]{label*=\arabic*.}
\begin{definition}{Field}{}
	A field $\mathbb{F}$ is a set of objects together with two operations called \emph{addition} and \emph{multiplication} (denoted $+$ and $\cdot$, respectively), for which the following axioms hold:

	\begin{descitemize}
		\item[Closure of under addition and multiplication] for any $a,b\in\mathbb{F}$,
			\begin{enumerate}
				\item $(a+b)\in\mathbb{F}$,
				\item $(a\cdot b)\in\mathbb{F}$.
			\end{enumerate}

		\item[Commutativity under addition multiplication] for any $a,b\in\mathbb{F}$,
			\begin{enumerate}
				\item $a+b=b+a$,
				\item $a\cdot b=b\cdot a$.
			\end{enumerate}

		\item[Associativity under addition and multiplication] for any $a,b,c\in\mathbb{F}$,
			\begin{enumerate}
				\item $a+(b+c)=(a+b)+c$,
				\item $a\cdot(b\cdot c)=(a\cdot b)\cdot c$.
			\end{enumerate}

		\item[Additive and multiplicative identity] there exist an element in $\mathbb{F}$ called the \textit{additive identity} and denoted by $0$, for which $a+0=a$ for any $a\in\mathbb{F}$.

			Similarity, there exists an element in $\mathbb{F}$ called the \textit{multiplicative identity} and denoted by $1$, for which $a\cdot1=a$ for any $a\in\mathbb{F}$.

		\item[Additive and multiplicative inverses] for any element $a\in\mathbb{F}$ (except the additive identity) there exists:
			\begin{enumerate}
				\item $b\in\mathbb{F}$ such that $a+b=0$, and
				\item $c\in\mathbb{F}$ such that $a\cdot c=1$.
			\end{enumerate}
			(usually $b$ is denoted as $-a$, while $c$ is denoted as $a^{-1}$)

		\item[Distributivity of multiplication over addition] for any $a,b,c\in\mathbb{F}$,
			\[
				a\cdot(b+c) = (a\cdot b) + (a\cdot c).
			\]
	\end{descitemize}
\end{definition}

\subsection{Infinite fields}
We start with one of the most obvious examples of a field: the real numbers together with the standard addition and product.

\begin{theorem}{$\bm{\mathbb{R}}$ as a field}{R as a field}
	The set of real numbers $\mathbb{R}$ forms a field together with the standard addition and product.
\end{theorem}

We leave the proof of \autoref{theorem:R as a field} to the reader, as it is pretty straight forward using the known properties of the standard addition and product over $\mathbb{R}$ (and rather uninteresting). Instead, we jump forward to using \autoref{theorem:R as a field} for proving the same idea about the complex numbers:

\begin{theorem}{$\bm{\mathbb{C}}$ as a field and more more more}{C as a field}
	The set of complex numbers $\mathbb{C}$ forms a field together with the addition and product operations as defined in \autoref{sec:complex numbers} (namely \autoref{eq:complex_addition}, \autoref{eq:complex_product} and \autoref{eq:complex_product_geometric}).
\end{theorem}

\begin{proof}{$\bm{\mathbb{C}}$ as a field}{C as a field}
	(note: in the following proof, equalities marked with $!$ use the respective property of the real numbers)

	\begin{descitemize}
		\item[Closure under both operations] for any two complex numbers $z_{1}=a+\iu b$ and $z_{2}=c+\iu d$,
			\begin{listitemize}
			\item[Addition] since addition in $\mathbb{R}$ is closed, $(a+c)\in\mathbb{R}$ and $(b+d)\in\mathbb{R}$. Therefore
				\[
					z = z_{1}+z_{2} = a+c + (b+d)\iu
				\]
				is also a complex number with $\Re(z)=a+c$ and $\Im(z)=b+d$.
			\item[Multiplication] since multiplication in $\mathbb{R}$ is also closed, $(ac-bd)\in\mathbb{R}$ and $(ad+bc)\in\mathbb{R}$. Therefore
				\[
					z = z_{1} \cdot z_{2} = ac-bd + (ad+bc)\iu
				\]
				is a complex number with $\Re(z)=ac-bdc$ and $\Im(z)=ad+bc$.
			\end{listitemize}

		\item[Commutativity of both operation] for any two complex numbers $z_{1}=a+\iu b$ and $z_{2}=c+\iu d$,
			\begin{listitemize}
			\item[Addition] since addition in $\mathbb{R}$ is commutative, $a+c=c+a$ and $b+d=d+b$. Therefore
				\[
					z_{1} + z_{2} = a+c + (b+d)\iu \overset{!}{=} c+a + (d+b)\iu = z_{2}+z_{2}.
				\]
			\item[Multiplication] since multiplication in $\mathbb{R}$ is also commutative, $ac-bd = ca-db$ and $ad+bc=da+cb$. Therefore
				\[
					z_{1} \cdot z_{2} = ac-bd + (ad+bc)\iu \overset{!}{=} ca-db + (da+cb)\iu = z_{2} \cdot z_{1}.
				\]
			\end{listitemize}

		\item[Associativity of both operation] for any three complex numbers $z_{1}=a+\iu b,\ z_{2}=c+\iu d$ and $z_{3}=g+\iu h$ (where $a,b,c,d,g,h\in\mathbb{R}$\footnote{The letters $g$ and $h$ are used instead of $e$ and $f$ to avoid confusion with Eurler's constant and the common notation for real functions, respectively.}),
			\begin{listitemize}
			\item[Addition] since addition in $\mathbb{R}$ is associative, $a+(c+g)=(a+c)+g$ and $b+(d+h)=(b+d)+h$. Therefore
				\[
					z_{1} + (z_{2}+z_{3}) = a+(c+g) + [b+(d+h)]\iu \overset{!}{=} (a+c)+g + [(b+d)+h]\iu = (z_{1}+z_{2}) + z_{3}.
				\]
			\item[Multiplication] since multiplication in $\mathbb{R}$ is also associative, the following equalities apply:
				\begin{align*}
					a\cdot(c\cdot g) &= (a\cdot c)\cdot g,\\
					b\cdot(c\cdot h) &= (b\cdot c)\cdot h,\\
					a\cdot(d\cdot h) &= (a\cdot d)\cdot h,\\
					b\cdot(d\cdot g) &= (b\cdot d)\cdot g,\\
					a\cdot(c\cdot h) &= (a\cdot c)\cdot h,\\
					a\cdot(d\cdot g) &= (a\cdot d)\cdot g,\\
					b\cdot(c\cdot g) &= (b\cdot c)\cdot g,\\
					b\cdot(d\cdot h) &= (b\cdot d)\cdot h.\\
				\end{align*}
				Therefore,
				\begin{align*}
					z_{1}\cdot(z_{2} \cdot z_{3}) &= a\cdot(c\cdot g) - a\cdot(d\cdot h) - b\cdot(c\cdot h) - b\cdot(d\cdot g)\\ &+[a\cdot(c\cdot h) + a\cdot(d\cdot g) + b\cdot(c\cdot g) - b\cdot(d\cdot h)]\iu\\
					&\overset{!}{=} (a\cdot c)\cdot g - (a\cdot d)\cdot h - (b\cdot c)\cdot h - (b\cdot d)\cdot g\\ &+[(a\cdot c)\cdot h + (a\cdot d)\cdot g + (b\cdot c)\cdot g - (b\cdot d)\cdot h]\iu\\
					&= (z_{1} \cdot z_{2}) \cdot z_{3}.
				\end{align*}
			\end{listitemize}

		\item[Identity for both operations]~\\
			\begin{listitemize}
			\item[Addition] the complex number $0=0+0i$ is the complex addition identity: for any real number $x\in\mathbb{R},\ x+0=x$. Therefore, for any complex number $z=a+\iu b$,
				\[
					z+0 = a+\iu b + 0+0i = a+0 + (b+0)\iu \overset{!}{=} a+\iu b.
				\]
			\item[Multiplication] the complex number $1=1+0\iu$ is the complex multiplication identity: for any real number $x\in\mathbb{R},\ x\cdot1=x$ and $x\cdot0=0$. Therefore, for any complex number $z=a+\iu b$,
				\[
					z\cdot1 = (a+\iu b)\cdot(1+0\iu) \overset{!}{=}a\cdot1-\cancel{b\cdot0\iu^{2}} + (\cancel{a\cdot0\iu}+b\cdot1)\iu = a+\iu b.
				\]

			\end{listitemize}

		\item[Inverse for both operations]~\\
			\begin{listitemize}
			\item[Addition] for any complex number $z_{1}=a+\iu b$, the number $z_{2}=-a-\iu b$ is also a complex number for which
				\[
					z_{1} + z_{2} = a+\iu b + -a-\iu b \overset{!}{=} a-a + (b-b)\iu = 0 + 0\iu = 0.
				\]
			\item[Multiplication] for any complex number $z=r\eu^{\theta i}$ where $r\neq0$, the number $z^{-1}=\frac{1}{r}\eu^{-\iu\theta}$ is also a complex number for which
				\[
					z \cdot z^{-1} = r\eu^{\iu\theta} \cdot \frac{1}{r}\eu^{-\iu\theta} \overset{!}{=} \frac{r}{r}\eu^{\cancel{\iu\theta - \iu\theta}} = 1\cdot1 = 1.
				\]

			Note: for $z=a+\iu b$,
			\[
				z^{-1} = \frac{1}{r}\eu^{-\iu\theta} = \frac{1}{|z|}\cdot\frac{a-\iu b}{|z|} = \frac{1}{|z|}\cdot\frac{\conj{z}}{|z|} = \frac{\conj{z}}{|z|^{2}}.
			\]
			Therefore, for any $z\neq0,\ z^{-1} = \frac{\conj{z}}{|z|^{2}}$.
			\end{listitemize}

		\item[Distributivity of multiplication over addition] for any $z_{1}=a+\iu b,\ z_{2}=c+\iu d$ and $z_{3}=g+\iu h$,
			\begin{align*}
				z_{1}\cdot(z_{2}+z_{3}) &= (a+\iu b)\cdot(c+\iu d + g+\iu h) = (a+\iu b)\cdot(c+g + [d+h]\iu)\\
				&= ac + ag + (bd)\iu^{2} + (bh)\iu^{2}\\
				&+ (ad)\iu + (ah)\iu + (bc)\iu + (bg)\iu\\
				&= ac + ag - bd - bh + (ad + ah + bc + bg)\iu\\
				&= ac-bd + (ad+bc)\iu + ag-bh + (ah+bg)\iu\\
				&= (z_{1}\cdot z_{2}) + (z_{1}\cdot z_{3}).
			\end{align*}
	\end{descitemize}
\end{proof}

The sets $\mathbb{R}$ and $\mathbb{C}$ are examples of \emph{infinite fields}, since they each have infinite number of elements. The set $\mathbb{Q}$ (rational numbers) can be shown to also be an infinite field, however unlike $\mathbb{R}$ and $\mathbb{C}$ it has \textbf{countable} number of elements, i.e. each number in $\mathbb{Q}$ can be assigned an index $1,2,3,\dots$\footnote{For proof, see \ldots}.

\begin{challenge}{$\bm{\mathbb{Q}}$ as a field}{Q as a field}
	Prove that $\mathbb{Q}$ (together with the usual addition and product operation) is indeed a field.
\end{challenge}

\subsection{Finite fields}
While all three examples of fields we encountered so far have each an infinite number of elements, some fields only have a finite number of elements (called their \emph{order}). For example, consider the set $S=\{0,1,a,b\}$ and the addition and product operations described using the following tables (left table describes addition, right table describes multiplication):

\centering
\begin{tabular}[]{>{\bfseries}c|cccc}
	\rowstyle{\bfseries} % why doesn't this work? :(
	$\bm{+}$ & 0 & 1 & a & b\\
	\hline
	\rule{0em}{2.65ex}%
	0 & 0 & 1 & a & b\\
	1 & 1 & 0 & b & a\\
	a & a & b & 0 & 1\\
	b & b & a & 1 & 0\\
\end{tabular}\hspace{2cm}
\begin{tabular}[]{>{\bfseries}c|cccc}
	\rowstyle{\bfseries}
	$\bm{\cdot}$ & 0 & 1 & a & b\\
	\hline
	\rule{0em}{2.65ex}%
	0 & 0 & 0 & 0 & 0\\
	1 & 0 & 1 & a & b\\
	a & 0 & a & b & 1\\
	b & 0 & b & 1 & a\\
\end{tabular}

\flushleft
By examining the tables above, several points become clear:
\begin{itemize}
	\item all the possible combinations of operands in both addition and multiplication give elements from $S$ itself, meaning that the set is closed under both these operations.
	\item both tables are symmetric around their main diagonal, meaning that both addition and multiplication are commutative operations.
	\item in the addition table, the first row and first column both show that $x+0=x$ for any $x\in S$, meaning that $0$ is the additive identity in $S$.
	\item in the product table, the second row and second column both show that $x\cdot1=x$ for any $x\in S$, meaning that $1$ is the multiplicative identity in $S$.
	\item in the addition table, the element $0$ appears in each row and each column exactly once. This means that every element $x$ has a single additive inverse $y\in S$.
	\item in the product table, the element $1$ appears in each row and each column exactly once, except for the first row and first column. This means that every element $x\neq 0$ has a single multiplicative inverse $z\in S$.
\end{itemize}

We therefore only need to prove two points to show that $S$ is a field together with the operations described by the above tables: associativity of both operations and distributivity of multiplication over addition. We leave these proofs as a challenge to the reader. Such a field is sometime denoted as $\mathbb{F}_{4}$. There are, of course, infinitely many finite fields.

\subsection{Modulo fields}
Another example of finite fields are sets of integers of the form $\{0,1,2,3,\dots,n\}$ where $n$ is a prime, together with \emph{modular addition} and \emph{modular product}. To understand modular arithmetics, we recall the fact that on a circle, an angle can have a negative value but also greater than $\ang{360}$ values are possible (see \autoref{fig:angles equivalency}): $\ang{390}$ is equivalent to $\ang{30}$, $\ang{-30}$ is equivalent to $\ang{330}$, etc. The set of integer values $\ang{0},\ang{1},\ang{2},\dots,\ang{359}$ on a circle is an example of a modular set: if for example we add together two angles of values $\deg{100}$ and $\ang{300}$ we get the equivalent angle $\deg{60}$. If we subtract $\deg{300}$ from $\deg{100}$ the result is an angle of $\deg{160}$.

We say that on a circle, the values $\ang{360}, \ang{720}, -\ang{360}$ etc. are all \emph{congruent} to $0$ modulo $360$. In mathematical notation we represent this fact as e.g.
\begin{equation}
	720 \equiv 0 \pmod{360}.
	\label{eq:mod 360}
\end{equation}

Note that from this point forward we drop the degrees unit, and deal with pure integers.
The notation for the set $\{0,1,2,\dots,359\}$ is $\mathbb{Z}_{360}$. Generally speaking, the set $\{0,1,2,\dots,n\}$ is denoted as $\Z{n}$.

\begin{note}{About modulo set notation}{}
	It is not common to use the notation $\Z{n}$ for the modulo-$n$ set, since it is also used for a different algebraic construct, namely the $n$-adic ring. However, due to the simplicity of the notation, and the fact that we don't discuss rings in this chapter we are using it in this book. Common notations for the set are $\mathbb{Z}\mathbin{/} n\mathbb{Z}$ and $\mathbb{Z}\mathbin{/} n$.
\end{note}

Addition and multiplication on $\Z{n}$ is done by the following rather straight forward definition:
\begin{definition}{Operations in $\bm{\Z{n}}$}{operations in Zn}
	In the set $\Z{n}$ addition and multiplication are defined as the following:
	\begin{descitemize}
	\item[Addition] for any two elements $a,b\in\Z{n},\ a+b\coloneqq(a+b)\pmod{n}$.
	\item[Multiplication] for any two elements $a,b\in\Z{n},\ a\cdot b\coloneqq(a\cdot b)\pmod{n}$.
	\end{descitemize}
\end{definition}

\begin{example}{Operations in $\bm{\Z{n}}$}{}
	The tables below show addition and multiplication results of numbers in different modulo sets $\Z{n}$ for some values of $n$:

	\vspace{1em}
	\centering
	\begin{tabular}[]{lll}
		\toprule
		$n$ & $2+3$ & $2\cdot3$\\
		\midrule
		$4$  & $1$ & $2$\\
		$5$  & $0$ & $1$\\
		$6$  & $5$ & $0$\\
		$7$  & $5$ & $6$\\
		$8$  & $5$ & $6$\\
		$9$  & $5$ & $6$\\
		$10$ & $5$ & $6$\\
		$11$ & $5$ & $6$\\
		\bottomrule
	\end{tabular}
	\hspace{1cm}
	\begin{tabular}[]{lll}
		\toprule
		$n$ & $4+7$ & $4\cdot7$\\
		\midrule
		$8$  & $3$  & $4$\\
		$9$  & $2$  & $1$\\
		$10$ & $1$  & $8$\\
		$15$ & $11$ & $13$\\
		$20$ & $11$ & $8$\\
		$27$ & $11$ & $1$\\
		$28$ & $11$ & $0$\\
		$30$ & $11$ & $28$\\
		\bottomrule
	\end{tabular}
\end{example}

\autoref{fig:Z5 periodic} shows the equivalency between integers and the elements of $\Z{5}$.

\begin{figure}[h]
	\centering
	\begin{tikzpicture}
		\pgfmathsetmacro{\d}{1.75}
		\pgfmathsetmacro{\h}{-1.3}
		\foreach \k in {-10,-9,...,9}{
			\pgfmathsetmacro{\kp}{int(mod(\k+20,5))}
			\node (\k) at ({\k/\d},0) {$\k$};
			\node (z5\k) at ({\k/\d},\h) {$\kp$};
			\draw[->, >=stealth, densely dashed, xblue] (\k.south) -- (z5\k.north);
		}
		\node (Zdots) at ({-11/\d},0) {$\dots$};
		\node (Z5dots) at ({-11/\d},\h) {$\dots$};
		\node at ({10/\d},0) {$\dots$};
		\node at ({10/\d},\h) {$\dots$};
		\node[left of=Zdots] (Z) {$\mathbb{Z}$};
		\node[left of=Z5dots] (Z5) {$\Z{5}$};
		\draw[->, >=stealth, densely dashed, xblue] (Z.south) -- (Z5.north);
	\end{tikzpicture}
	\caption{An example of the periodicity of $\Z{5}$: the top numbers are the ordinary integers, each showing their respective congruent modulo $5$ below (blue dashed arrow).}
	\label{fig:Z5 periodic}
\end{figure}

Only the sets $\Z{n}$ for which $n$ is a prime number are also fields. Let's define this property precisely:

\begin{theorem}{$\bm{\Z{p}}$ is a field}{Zp is a field}
	Any modulo set $\Z{p}$ where $p$ is a prime number greater than $1$ is also a field together with the operations as defined in \autoref{def:operations in Zn}.
\end{theorem}

In order to prove \autoref{theorem:Zp is a field} we use two lemmas: the first is known as \emph{B\'ezout’s lemma}:
\begin{lemma}{B\'ezout's lemma}{bezout}
	For any two positive integers $a,b$ there exist two integers $x,y$ such that
	\[
		\gcd{a}{b} = xa+yb.
	\]
\end{lemma}

\begin{note}{$\bm{\gcd{a}{b}}$}{gcd}
	$\gcd{a}{b}$ is the \emph{greatest common divisor} of the two integers $a$ and $b$. For example, $\gcd{36}{24}=12$ since the divisors of $36$ are $1,\ 2,\ 3,\ 4,\ 6,\ 9,\ 12,\ 18,\ 36$,\ and the divisors of $24$ are $1,\ 2,\ 3,\ 4,\ 6,\ 8,\ 12,\ 24$.
\end{note}

An example of B\'ezout's lemma is the following:

\begin{example}{B\'ezout's lemma in action}{bezout}
	For the two positive integers $a=60,\ y=114$
	\[
		\gcd{60}{114}=6.
	\]
	Therefore, B\'ezout's lemma says that there exist two integers $x,y$ such that
	\[
		6 = 60x + 114y.
	\]
	Indeed, two such integers exist: $x=2$ and $y=-1$.
\end{example}
(SHOULD WE PROVE THE LEMMA?..)

The second lemma we use is the following:
\begin{lemma}{$\bm{\gcd{n}{p}=1}$}{gcd(n,p)=1}
	Given a positive prime number $p$, then for any positive integer $n<p$,
	\[
		\gcd{p}{n}=1.
	\]
\end{lemma}

Proving the lemma:
\begin{proof}{$\bm{\gcd{n}{p}=1}$}{gcd(n,p)=1}
	We assume that $\gcd{p}{n} \neq 1$. Then there exist an integer $a\leq n<p$ which divides both $n$ and $p$, meaning that $p$ has a divider, contrary to the assumption that $p$ is a prime number. Therefore $\gcd{n}{p}$ must equal $1$.
\end{proof}

Now we can proceed to the proof of \autoref{theorem:Zp is a field}:

\begin{proof}{$\bm{\Z{p}}$ is a field}{}
	\begin{descitemize}
	\item[Closure under both operations] the definition of the modulo operator limit any $M\pmod{p}$ (where $M\in\mathbb{Z}$) to be in $[0,p-1]$. Therefore the result of using the operators given in \autoref{def:operations in Zn} must be within the same range, and thus in $\Z{p}$.

	\item[Commutativity and associativity of both operations] for any two numbers $a,b\in\Z{p}$ the result $a+b$ and $a\cdot b$ under $\mathbb{Z}$ is both commutative and associative. Therefore the result modulo $n$ is the same no matter the order of operations.

	\item[Additive identity] the number $0\in\Z{p}$ is the additive identity, since for each $a\in\Z{p},\ a+0=a$.

	\item[Multiplicative identity] the number $1\in\Z{p}$ is the additive identity, since for each $a\in\Z{p},\ a\cdot1=a$.

	\item[Additive inverse] for each $a\in\Z{p}$ the element $n=p-a$ is in $\Z{p}$ since $p>a$. Adding $n$ to $a$ results in $0$:
		\[
			a+n = a+(p-a) = p \equiv 0\pmod{p}.
		\]

	\item[Multiplicative inverse] let $a\in\Z{p}$ and $a\neq0$. Since $p$ is a prime, $\gcd{a}{p}=1$ and from B\'ezout's theorem we know that there exist two integers $x,y$ such that
		\[
			xa + yp = 1.
		\]
		Rearrangement gives $p = \frac{1-xa}{y}$ meaning that $p$ divides $1-xa$, and thus
		\[
			xa\equiv1\pmod{p}.
		\]
		Therefore $x$ is the multiplicative inverse of $a$.

	\item[Distributivity of multiplication over addition] \ldots
	\end{descitemize}
\end{proof}

The only part of the proof that uses the fact that $p$ is a prime number is the multiplicative inverse. When $n$ is not a prime, $\Z{n}$ is not a field.

\begin{challenge}{$\bm{\Z{n}}$ is not a field when $n$ is not a prime number}{}
	Prove that the modulo set $\Z{n}$ where $n$ is \textbf{not} a prime number, is not a field.

	(\textbf{hint}: what property of prime numbers is used in the above proof to show that there is always a multiplicative inverse in $\Z{p}$ where $p$ is prime?)
\end{challenge}


\section{Information Theory}
Information Theory revolves around quantifying how much information is present
in a signal. The basic intuition lies in the fact that learning an unlikely
event has occured is more informative than learning that a likely event has
occured. The basics are:
\begin{enumerate}
  \item Likely events should have low information content, and in the extreme
  case, events that are guaranteed to happen should have no information content
  whatsoever.
  \item Less likely events should have higher information content.
  \item Independent events should have additive information.
\end{enumerate}
We satisfy all three properties by defining self-information of an event $x$
for a probability distribution $P$ as:
\begin{equation}
  I(x) = -\log P(x)
\end{equation}
We can quantify the amount of uncertainty in a
distribution using Shannon entropy:
\begin{equation}
  H ( P ) = \mathbb { E } _ { \mathrm { x } \sim P } [ I ( x ) ] = - \mathbb { E } _ { \mathrm { x } \sim P } [ \log P ( x ) ]
\end{equation}
Which in the discrete setting is written as:
\begin{equation}
  H ( P ) = -\sum_{x} P(x) \log P(x)
\end{equation}
In other words, the Shannon entropy of a distribution is the expected amount
of information in an event drawn from that distribution. It gives a lower bound
on the number of bits needed on average to encode symbols drawn
from a distribution $P$. If we have two separate probability distributions
$P(x)$ and $Q(x)$ over the same random variable $\mathrm{x}$, we can measure how
different these two distributions are using the Kullback-Leibler (KL)
divergence:
\begin{equation}
  \begin{split}
    D _ { \mathrm { KL } } ( P \| Q ) \quad =& \quad \mathbb { E } _ { \mathbf { x } \sim P } \left[ \log \frac { P ( x ) } { Q ( x ) } \right]\\
    \\
    =& \quad \mathbb { E } _ { \mathbf { x } \sim P } \left[ \log P ( x ) - \log Q ( x ) \right] \\
    \\
    =& \quad \sum_x P(x) \frac{\log P(x)}{\log Q(x)} \\
  \end{split}
\end{equation}
In the case of discrete variables, it is the extra amount of information
needed to send a message containing symbols drawn from probability distribution
$P$, when we use a code that was designed to minimize the length of messages
drawn from probability distribution $Q$. The KL divergence is always
non-negative, and is $0$ if and only if $P$ and $Q$ are the same. We can
relate the KL divergence to cross-entropy.
\begin{equation}
  \begin{split}
    H(P, Q) \quad =& \quad H(P) + D _ { \mathrm { KL } } ( P \| Q ) \\
    \\
    =& \quad - \mathbb { E } _ { \mathbf { x } \sim P } \left[ \log Q ( x ) \right]\\
    \\
    =& \quad -\sum_x P(x) \log Q(x) \\
  \end{split}
\end{equation}
Minimizing the cross-entropy with respect to $Q$ is equivalent to
minimizing the KL divergence, because $Q$ does not participate in the
omitted term (entropy is constant).


\chapter{One-Dimensional Discrete Fourier Transforms}
\footnote{For more detail, see Olver and Shakiban~\cite{OlvSha06}.} The discrete Fourier transform (DFT) takes a function sampled at a finite number of points and finds the coefficients for the linear combination of trigonometric polynomials that best approximates the function; the number of trigonometric polynomials used is equal to the number of sample points.  Suppose we have a function $f(x)$ which is defined on the interval $a \leq x \leq b$. Due to memory limitations, a computer can only store values at a finite number of sample points, i.e. $a \leq x_0 < x_1 < ... <x_n \leq b$. For our purposes these points will be equally spaced, for example $x_1-x_0=x_3-x_2$, and so we can write
\begin{eqnarray}
x_j = a +jh, \qquad j=0,1,2,...,n
\end{eqnarray}
where $x_j$ are the \emph{sample points}, $n$ is the number of sample points and
\begin{eqnarray}
 h=\frac{b-a}{n}.
\end{eqnarray}
It is convenient to use the \emph{standard interval}, for which $0 \leq x \leq 2\pi$. Rewriting $x$ in terms of standard interval yields
\begin{eqnarray}
x_0=0,x_1=\frac{2\pi}{n},x_2=\frac{4\pi}{n},x_j=\frac{2j\pi}{n},...,x_{n-1}=\frac{2(n-1) \pi}{n}
\end{eqnarray}
Notice how $x_n=2\pi$ is omitted;  periodicity implies that the value of the function at $2\pi$ is the same as the value of the function at $0$, so it need not be included. We will introduce the DFT using the language of linear algebra. Much of this formalism carries over to continuous functions that are being approximated. It also makes it easier to understand the computer implementation of the algorithms. Many computer packages and programs are optimized to perform calculations through matrix operations, so the formalism is also useful when actually calculating transforms. We write the approximation to $f(x)$ at the sample points as a finite dimensional vector
\begin{eqnarray}
\bm{f}=(f_0,f_1,...,f_{n-1})^{T}=(f(x_0),f(x_1), ... ,f(x_{n-1}))
\end{eqnarray}
where
\begin{eqnarray}
 f_j=f(x_j)=f \left(\frac{2 j \pi}{n} \right).
\end{eqnarray}
The DFT decomposes the sampled function $f(x)$ into a linear combination of complex exponentials, $\exp(ikx)$ where $k$ is an index. Since
\begin{eqnarray}
\exp(ikx)=\cos(kx)+i\sin(kx),
\end{eqnarray}
we also obtain an expansion in trigonometric functions, which may be more familiar from courses in calculus and differential equations. Since the function is sampled at $n$ points, the highest frequency of oscillation that can be resolved will have $n$ oscillations. Any frequencies higher than $n$ in the original function are not adequately resolved and cause an \emph{aliasing} error (see, for example, Boyd~\cite{Boy01} or Uecker~\cite{Uec09} for more on this). This error can be reduced by sampling at a greater number of points so that the number of approximating exponentials functions can also be increased. There is a tradeoff between increasing the accuracy of the simulation and the time required for the simulation to complete. For many cases of scientific and practical interest, simulations with up to thousands of grid points can be computed relatively quickly. Below we explain how a function $f(x)$ can be approximated by an interpolating trigonometric polynomial $p(x)$ so that
\begin{eqnarray}
f(x) \approx p(x)=c_0+c_1e^{2ix}+c_2e^{2ix}+...+c_{n-1} e^{(n-1)ix}=\sum\limits_{k=0}^{n-1} c_k e^{ikx}
\end{eqnarray}
The $\approx$ symbol means that $f(x)$ and $p(x)$ agree on each sample point, i.e., $f(x_j)=p(x_j)$ for each $j=0,1,...n-1$, but the interpolated polynomial $p(x)$ is only an approximation of the true solution $f(x)$ away from the sample points.. The $c_n$ are called discrete \emph{Fourier coefficients} and are what we will be looking to solve for. $p(x)$ represents the values of interpolating trigonometric polynomial of degree $\leq n-1$, so if we have the values of these coefficients then we have a function we can use as an approximation of $f(x)$. Since we are working in a finite-dimensional vector space, a useful approach is to rewrite the discrete Fourier series as a vector. We let
\begin{eqnarray}
\bm{\omega_k}&=&(e^{ikx_0},e^{ikx_1},e^{ikx_2},...,e^{ikx_n})^T \\
&=& (1,e^{2k \pi i/n},e^{4k\pi i/n},...,e^{2(n-1)k\pi i/n})^T,
\end{eqnarray}
where $k=0,1,...,n-1$. The interpolation conditions, $f(x_j)=p(x_j)$, can also be rewritten in vectorial form
\begin{eqnarray}\label{eq:FourExp}
\bm{f}=c_0\bm{\omega_0}+c_1\bm{\omega_1}+...+c_{n-1} \bm{\omega_{n-1}}.
\end{eqnarray}
Here $\bm f$ is a vector evaluated at the sample points, which is decomposed into vectors $\bm{\omega}_k$, much as a vector in three dimensional space can be decomposed into the components in the $x$, $y$ and $z$ directions. The DFT allows us to compute the coefficients $c_i$ given the value of the function at the sample points. This may at first seem unmotivated, but in many applications, such as solving differential equations, it is easier to manipulate a linear combination of trigonometric polynomials,  $\bm{\omega_0},...,\bm{\omega_{n-1}}$, than it is to work with the original function. In order to solve for $c_k$, we use the orthonormality of the basis elements $\bm{\omega_0},...,\bm{\omega_{n-1}}$. We now explain how this is done \footnote{For a more detailed explanation see Olver and Shakiban~\cite{OlvSha06}.}.

Define $\xi_n=e^{2\pi i/n}$. We observe that
\begin{equation}
\left(\xi_n\right)^n=\exp\left(\frac{2\pi i n}{n}\right)=\cos(2\pi ) +i\sin(2\pi )=1
\end{equation}
For this reason $\xi_n$ is known as the primitive $n^{\text{th}}$ root of unity. Note also that for $0\leq k < n$, we have that $(\xi_n^k)^n=1$, so all other roots of unity when taken to the power $n$ can be obtained from the primitive $n^{\text{th}}$ root of unity. We will use this to perform the DFT algorithm to calculate the coefficients $c_0,...,c_{k-1}$ in eq.\ \eqref{eq:FourExp}. The main idea behind the DFT algorithm is to use orthogonality of the vectors $\bm \omega_k$. To show the orthogonality between the vectors $\bm \omega_k$ and $\bm \omega_l$, we let $\bm \omega_l^*$ denote the complex conjugate of $\bm \omega_l$, and then take the inner product of $\bm \omega_k$ and $\bm \omega_l$ and find that
\begin{align*}
\left<\bm \omega_k, \bm \omega_l \right>&{}=\frac{1}{n}\sum_{m=0}^{n-1}\exp\left(\frac{2\pi ik m}{n}\right)\left[\exp(\frac{2\pi il m}{n})\right]^*
\\&{}=\frac{1}{n}\sum_{m=0}^{n-1}\exp\left(\frac{2\pi i(k-l) m}{n}\right)
\\&{}=\frac{1}{n}\sum_{m=0}^{n-1}\cos\left(\frac{\pi(k-l)m}{n}\right)+i\sin\left(\frac{\pi(k-l)m}{n}\right)
\\&{}=\left\{{1\quad\textup{if } k=l}\atop{0\quad\textup{otherwise}}\right.
\end{align*}
To deduce the last part, if $k=l$ then $\exp(0)=1$, and if $k\ne l$, then we are sampling the sine and cosine functions at equally spaced points on over an integral number of wavelengths. Since these functions have equal magnitude positive and negative parts, they sum to zero, much as the integral of a sine or cosine over an integral number of wavelengths is zero. This implies that we can compute the Fourier coefficients in the discrete Fourier sum by taking inner products
\begin{eqnarray}
c_k=<\bm{f},\bm{\omega_k}>=\frac{1}{n} \sum\limits_{m=0}^{n-1} \xi_n^{-mk}f_j.
\end{eqnarray}
We note the close connection between the continuous and discrete settings, where an integral is replaced by a sum.

%%%%%%%%
%subsection
%%%%%%%%
\section{Fast Fourier Transform}
Computing the Fourier coefficients, $c_0,...,c_{n-1}$ using the DFT from the definition can be very slow for large values of $n$. Computing the Fourier coefficients $c_0,...c_{n-1}$ requires $n^2-n$ complex multiplications and $n^2-n$ complex additions. In 1960, Cooley and Tukey~\cite{CooTuk65} rediscovered a much more efficient way of computing DFT by developing an algorithm known as the Fast Fourier Transforms (FFT) -- the method was known to Gauss, but received little attention since he did not publish it~\cite{HeiJohBur84}. The FFT cuts the number of arithmetic operations down to $O(n\log n)$. For large values of $n$, this can make a huge difference in computation time compared to the standard DFT. The reason why the FFT is so important is that it is heavily used in spectral methods. The basic FFT algorithm used by Cooley and Tukey~\cite{CooTuk65} is well documented in many places, however, there are other implementations of the algorithm and the best version of the algorithm to use depends heavily on computer architecture. We therefore do not give further descriptions here.