% Copyright 2018 Melvin Eloy Irizarry-Gelpí
\chapter{Mechanic Energy}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Preliminary}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
There are many kinds of energy. You have \textbf{kinetic energy}, which is associated with motion. The amount of kinetic energy $K$ depends on the amount of mass $m$ and the amount of speed $v$:
\begin{equation}
    K = \frac{1}{2} m v^{2}
\end{equation}
Note that kinetic energy changes if the velocity of the object changes.

You have \textbf{gravitational energy} (a form of potential energy), which is associated to being in a region with a gravitational field. The amount of gravitational energy $U_{g}$ depends on the amount of mass $m$, the amount of gravitational acceleration $g$, and the amount of height $h$ measured from a reference point:
\begin{equation}
    U_{g} = m g h
\end{equation}
Note that the gravitational energy changes if the height of the object changes.

Finally, you have \textbf{mechanic energy}. The amount of mechanic energy $E$ is just the sum of kinetic energy $K$ and potential energy. In the case when the potential energy is only gravitational in nature, then the mechanic energy is
\begin{equation}
    E = K + U_{g}
\end{equation}
Energy cannot be created nor destroyed; it can only be transformed into other forms of energy. In a closed system, where energy is not added nor removed by an external source, mechanic energy $E$ is \textbf{constant in time}.

If the object only moves and feels gravity, then the mechanic energy $E$ is given by
\begin{equation}
    E = \frac{1}{2} m v^{2} + m g h
\end{equation}
Solving for $v^{2}$ you get
\begin{equation} \label{eq.07.vv}
    v^{2} = \left( - 2 g \right) h + \left( \frac{2 E}{m} \right)
\end{equation}
This relation predicts that $v^{2}$ is a \textbf{linear function} of $h$ with the slope being $-2g$ and the intercept being $2 E / m$. That is, the data in a $v^{2}$ versus $h$ graph should have a \textbf{linear shape}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In order to study a system where there is gravitational energy, and also that that gravitational changes with time, we used an object moving along an incline. With a motion sensor, we recorded the \textbf{position} $d$ and \textbf{velocity} $v$ along the incline as they changed with time. You also recorded some height measurements to determine the sine of the angle of inclination $\theta$ of the incline. Also, you recorded the mass $m$ of the cart.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Analysis}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We would like to test two predictions:
\begin{enumerate}
    \item Mechanic energy does not change with time.
    \item There is a linear relation between $v^{2}$ and $h$ given by equation (\ref{eq.07.vv}).
\end{enumerate}
To test the first prediction, we need to compute the mechanic energy with the data that we collected. To test the second prediction, we need to make a graph with $v^{2}$ in the vertical axis and $h$ in the horizontal axis.

Here are some steps to complete the analysis.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calculate Sine}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
You measured the heights of four positions along the incline:
\begin{equation}
    h_{1} \qquad h_{2} \qquad h_{3} \qquad h_{4}
\end{equation}
Assuming that the four positions are equally separated by a distance $x$ (I used 10 cm; some people used that also; others did 20 cm), then the sine of the angle should be given by the following three calculations:
\begin{equation}
    \sin{(\theta_{1})} = \frac{h_{2} - h_{1}}{x}, \qquad \sin{(\theta_{2})} = \frac{h_{3} - h_{2}}{x}, \qquad \sin{(\theta_{3})} = \frac{h_{4} - h_{3}}{x}
\end{equation}
You can think of each of these as an independent measurement, so take the average value of the three and use that as the value for the sine of the angle:
\begin{equation}
    \sin{(\theta)} = \frac{1}{3} \left[ \sin{(\theta_{1})} + \sin{(\theta_{2})} + \sin{(\theta_{3})} \right]
\end{equation}
You need this sine because the height $h$ at the incline is related to the position $d$ along the incline by the relation
\begin{equation}
    h = d \sin{(\theta)}
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Truncate the Data}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Before you can do anything with the data from the experiment, you need to isolated the part of the data where the motion along the incline is happening. In order to know where to cut the data, it is best to make a scatter plot with velocity in the vertical axis and time in the horizontal axis. There are three kinds of truncation:
\begin{enumerate}
    \item Upward motion only: find the time just after the velocity values begin the diagonal downward trend, and the time when the velocity is close to zero
    \item Downward motion only: find the time when the velocity is close to zero, and the time just before the velocity ends the diagonal downward trend
    \item Upward and downward motion: find the time just after the velocity values begin the diagonal downward trend, and the time just before the downward trend ends
\end{enumerate}
Once you know the initial time and the final time of the motion, just delete the data outside of this time interval.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calculate Height}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In a separate column, you should compute the value of the height on the incline using the values in the position column. If $d$ is a position value, then the corresponding height is given by
\begin{equation}
    h = d \sin{(\theta)}
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calculate Kinetic Energy}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In a separate column, you should compute the value of the kinetic energy using the values in the velocity column. If $v$ is a velocity value, then the corresponding kinetic energy is given by
\begin{equation}
    K = \frac{1}{2} m v^{2}
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calculate Gravitational Energy}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In a separate column, you should compute the value of the gravitational energy using the values in the height column. If $h$ is a height value, then the corresponding gravitational energy is given by
\begin{equation}
    U_{g} = m g h
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calculate Mechanic Energy}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In a separate column, you should compute the value of the mechanic energy using the values in the kinetic and gravitational energy columns. If $K$ is a value in the kinetic energy column, and $U_{g}$ is a value in the gravitational energy column, then the mechanic energy is given by
\begin{equation}
    E = K + U_{g}
\end{equation}
Make sure that you add values that are evaluated at the same time.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Graph Mechanic Energy vs Time}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
One of the goals of this experiment is to verify that mechanic energy is constant in time. To check this, you can make a scatter plot with time in the horizontal axis and mechanic energy in the vertical axis. In principle, you should see that the values of mechanic energy do not change with time. In practice, we see that in general the mechanic energy decreases as time passes. This is actually true, since in this experiment we could not completely remove friction and this force leads to a dissipation of energy. However, the amount of change in energy over time is very small, so effectively and approximately mechanic energy is constant.

In particular, note that the energy values in my run 6 (figure \ref{figure.07.run.6.e}) appear to be random. This strongly suggest that the variations are more related to noise than anything physical.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-1-energy.png}
    \caption{}
    \label{figure.07.run.1.e}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-2-energy.png}
    \caption{}
    \label{figure.07.run.2.e}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-5-energy.png}
    \caption{}
    \label{figure.07.run.5.e}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-6-energy.png}
    \caption{}
    \label{figure.07.run.6.e}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calculate Velocity Squared}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In a separate column, you should calculate the velocity squared using the values in the velocity column. If $v$ is a velocity values, then the velocity square is $v^2$ or equivalently $v \times v$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Graph $v^2$ vs $h$}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The other goal of this experiment was to verify the linear relation between $v^{2}$ and $h$ suggested by equation (\ref{eq.07.vv}) due to conservation of energy. Since the relation is a linear function, the data points should arrange themselves to form a linear shape. You can use Google Sheets or Excel to find the best linear fit. In Google Sheets, this is under ``Series'' in the chart editor. You need to check the ``Trend line'' box and make sure that it is ``Linear''. It is good practice to reduce the size of the data points (I use 2px) and to use a color for the trend line that is different from the color of the data points. In my case I have blue data points for the experimental data and red line for the best fit line.

If the data were perfect, then the slope of this linear fit should be given by
\begin{equation} \label{eq.07.slope}
    \text{slope } = -2g = 19.6 \text{ m/s}^{2}
\end{equation}
and the intercept would be related to the amount of mechanic energy via
\begin{equation}
    \text{intercept } = \frac{2 E}{m}
\end{equation}
The slope is negative, so we should see a diagonal line that is downward as height increases. From the intercept you can get a value for mechanic energy that should be close to what you found in the mechanic energy column:
\begin{equation} \label{eq.07.intercept}
    E = \frac{m \times \text{ intercept}}{2}
\end{equation}
You can also compare this value with the average of the mechanic energy column (use the \texttt{AVERAGE} function to compute this). Note that this is an average over time.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-1-fit.png}
    \caption{}
    \label{figure.07.run.1.fit}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-2-fit.png}
    \caption{}
    \label{figure.07.run.2.fit}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-5-fit.png}
    \caption{}
    \label{figure.07.run.5.fit}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
    \centering
    \includegraphics[scale=0.71]{image/07-mechanic/run-6-fit.png}
    \caption{}
    \label{figure.07.run.6.fit}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{My Data}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
I collected ten runs of data. In run 1, I used a truncation that only kept the upward motion. In run 2, I used a truncation that kept the upward and the downward motion. For run 5 and 6, I used a truncation that only kept the downward motion. As you can see, the linear fits are very appropriate, so the data does satisfy a linear relation. Furthermore, the values for the slope agree with the prediction: they are all very close to the predicted value in (\ref{eq.07.slope}). Also, the value of the intercept can be used in equation (\ref{eq.07.intercept}) to obtain an energy value that agrees with the time average mechanic energy.

Let us extract these values from a graph. Look at graph \ref{figure.07.run.6.fit} for run 6. The slope is -19.5 m/s$^{2}$. The intercept is 1.49 m$^{2}$/s$^{2}$. If the mass is 0.5 kg, then
\begin{equation}
    \frac{m \times \text{ intercept}}{2} = \frac{1}{2} (0.5 \text{ kg}) (1.49 \text{ m}^{2}\text{/s}^{2}) = 0.3717 \text{ J}
\end{equation}
This number is close to the time-average of the mechanic energy: 0.3728 J.

Here is a table summarizing some of my results.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}
	\centering
    \begin{tabular}{|r|r|r|r|r|}\hline
        Run & 1 & 2 & 5 & 6 \\ \hline
        Slope (m/s$^{2}$) & -21.1 & -19.9 & -19.3 & -19.5 \\
        Intercept (m$^{2}$/s$^{2}$) & 1.78 & 1.33 & 1.82 & 1.49 \\
        $E$ from intercept (J) & 0.4460 & 0.3326 & 0.4540 & 0.3717 \\
        Time-averaged $E$ (J) & 0.4210 & 0.3283 & 0.4587 & 0.3728 \\
        \hline
    \end{tabular}
    \caption{Summary of results}
    \label{table.07.results}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Your Data}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
You should have ten runs of data as well.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Your Lab Report}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
For your lab report, I want you to look at three of your ten runs. You are free to choose which three runs. For the first run you should use a truncation that leaves the upward motion only. For the second run, use a truncation with the downward motion only. For the third run, use a truncation with both upward and downward motion.

For each of the three runs you should
\begin{enumerate}
    \item produce a mechanic energy versus time graph
    \item produce a $v^{2}$ versus $h$ graph; include the best linear fit and extract the slope and the intercept
\end{enumerate}
You should also have a table like the one above summarizing the results (my table has four runs; your table should have three).


\section{Error Introduced by Linear Interpolation}
We obtain the initial conditions for the standing accretion shock problem with an external code. We then have to interpolate these data onto the grid for use in \texttt{thornado}. We start by proving that linear interpolation is a convex combination of the solution at the boundary points.

\subsection{Proof that Linear Interpolation is a Convex Combination of Boundary-Points}
Linear interpolation of a function, $f$, of a variable, $r$, bounded by two points $r_{L}$ and $r_{H}$, with $r_{H}>r_{L}$, can be written as:
\begin{equation}
    f\left(r\right)=f\left(r_{L}\right)+\f{f\left(r_{H}\right)-f\left(r_{L}\right)}{r_{H}-r_{L}}\left(r-r_{L}\right).
\end{equation}
By making a change of variables from $r$ to $\eta$, where:
\begin{equation}
    r\left(\eta\right)=\eta\,r_{H}+\left(1-\eta\right)r_{L}=r_{L}+\eta\,\Delta r\implies\eta=\f{r-r_{L}}{\Delta r},\hspace{1em}\eta\in\left[0,1\right],
\end{equation}
such that $r\left(\eta=0\right)=r_{L}$ and $r\left(\eta=1\right)=r_{H}$, we have that:
\begin{align}
    f\left(\eta\right)&=f\left(0\right)+\f{f\left(1\right)-f\left(0\right)}{1-0}\left(\eta-0\right)=f\left(0\right)+\left(f\left(1\right)-f\left(0\right)\right)\eta\\
    \implies f\left(\eta\right)&=\eta\,f\left(1\right)+\left(1-\eta\right)f\left(0\right).
\end{align}

So when we interpolate a fluid variable, say the mass-density $\rho$, we have:
\begin{equation}
    \rho\left(r\right)=\rho_{L}+\f{\Delta\rho}{\Delta r}\left(r-r_{L}\right)\longrightarrow\rho\left(\eta\right)=\eta\,\rho_{H}+\left(1-\eta\right)\rho_{L},
\end{equation}
where $\rho_{L}\equiv\rho\left(r_{L}\right)$ and $\rho_{H}=\rho\left(r_{H}\right)$.

\subsection{Example: $f\left(x\right)=x^{2}$}
We can show this directly for simple functions. We take as an example $f\left(x\right)=x^{2}$:
\begin{align}
    f\left(x\right)&=x^{2}=\left[x_{L}+\eta\left(x_{R}-x_{L}\right)\right]^{2}=x_{L}^{2}+\eta^{2}\left(x_{R}-x_{L}\right)^{2}+2\,x_{L}\,\eta\left(x_{R}-x_{L}\right)\\
    &=x_{L}^{2}+\eta^{2}\,x_{R}^{2}+\eta^{2}\,x_{L}^{2}-2\,\eta^{2}\,x_{L}\,x_{R}+2\,\eta\,x_{L}\,x_{R}-2\,\eta\,x_{L}^{2}.
\end{align}
Now we add and subtract the interpolated solution: $\eta\,x_{H}^{2}+\left(1-\eta\right)x_{L}^{2}$. This yields:
\begin{align}
    x^{2}&=x_{L}^{2}+\eta^{2}\,x_{R}^{2}+\eta^{2}\,x_{L}^{2}-2\,\eta^{2}\,x_{L}\,x_{R}+2\,\eta\,x_{L}\,x_{R}-2\,\eta\,x_{L}^{2}+\eta\,x_{H}^{2}+\left(1-\eta\right)x_{L}^{2}-\eta\,x_{H}^{2}-\left(1-\eta\right)x_{L}^{2}\\
    &=\left[\eta\,x_{H}^{2}+\left(1-\eta\right)x_{L}^{2}\right]+\left[-\eta\left(1-\eta\right)x_{L}^{2}-\eta\left(1-\eta\right)x_{H}^{2}+2\,\eta\,x_{L}\,x_{H}\left(1-\eta\right)\right]\\
    &=\left[\eta\,x_{H}^{2}+\left(1-\eta\right)x_{L}^{2}\right]-\eta\left(1-\eta\right)\left[x_{L}^{2}+x_{H}^{2}-2\,x_{L}\,x_{H}\right]\\
    &=\left[\eta\,x_{H}^{2}+\left(1-\eta\right)x_{L}^{2}\right]-\eta\left(1-\eta\right)\left(x_{H}-x_{L}\right)^{2}\\
    &=\left[\eta\,x_{H}^{2}+\left(1-\eta\right)x_{L}^{2}\right]-\eta\left(1-\eta\right)\left(\Delta x\right)^{2}.
\end{align}
The term in square brackets is the interpolated solution, and the second term is the remainder, i.e. the error in the interpolated solution. So we see that when using linear interpolation to interpolate the function $f\left(x\right)=x^{2}$ we introduce an error (of the order $\left(\Delta x\right)^{2}$) that goes to zero as the step-size goes to zero.

As a final step we replace $\eta$ with $x$, yielding:
\begin{align}
    f\left(x\right)=x^{2}&=x_{L}^{2}+\f{x_{H}^{2}-x_{L}^{2}}{x_{H}-x_{L}}\left(x-x_{L}\right)-\left(x-x_{L}\right)\left(x_{H}-x\right)\\
    &=f\left(x_{L}\right)+\f{\Delta f}{\Delta x}\left(x-x_{L}\right)-\left(x-x_{L}\right)\left(x_{H}-x\right).
\end{align}

We show the results of this in the following figure:
\begin{figure}[H]
\centering
\includegraphics[scale=0.75]{LinearInterpolationError}
\end{figure}

\subsection{Mass Constant}
The mass constant, $C_{D}$, from the 3+1 GR hydrodynamics equations is:
\begin{equation}
    C_{D}\left(r\right)=\psi\left(r\right)^{6}\,\alpha\left(r\right)\,r^{2}\,\rho\left(r\right)\,W\left[v\left(r\right)\right]\,v\left(r\right).
\end{equation}
When we map from the radial coordinates from the original data to the mesh in \texttt{thornado} we don't introduce any error, because it is a linear map. This means that all the error we introduce must be due to $\rho$, $W$, and $v$. So, when deriving the error we consider the different function:
\begin{equation}
    C'_{D}\left(r\right)\equiv\f{C_{D}\left(r\right)}{\psi\left(r\right)^{6}\,\alpha\left(r\right)\,r^{2}}=\rho\left(r\right)\,W\left[v\left(r\right)\right]\,v\left(r\right).
\end{equation}

From the data we have this exact value at the two endpoints:
\begin{align}
    C'_{D}\left(r_{L}\right)&\equiv C'^{L}_{D}=\rho_{L}\,W_{L}\,v_{L}\\
    C'_{D}\left(r_{H}\right)&\equiv C'^{H}_{D}=\rho_{H}\,W_{H}\,v_{H}.
\end{align}


\chapter{Probability Distribution}
\label{ch:prob_distr}

\section{List of supported probability distributions}
StatAnaly supports the following probability distributions, in alphabetical order:

\begin{enumerate}
    \item Cauchy distribution
    \item Chi distribution
    \item Chi Squared distribution
    \item Erlang distribution
    \item Exponential distribution
    \item Gamma distribution
    \item Irwin-Hall distribution
    \item Non-central Chi distribution
    \item Non-central Chi Squared distribution
    \item Normal distribution
    \item Rayleigh distribution
    \item Standard Normal distribution
    \item Uniform distribution (Continuous)
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Cauchy distribution}

\subsubsection*{PDF}
\[
    P(x) = \frac{1}{\pi} \frac{b}{(x-m)^2 + b^2}
\]
where $b$ is the scale parameter which specifies the half-width at half-maximum;
$m$ is the location parameter which specifies the location of the peak of the distribution.

\subsubsection*{CDF}
\[
    D(x) = \frac{1}{\pi} \arctan{(\frac{x-m}{b})} + \frac{1}{2}
\]

\subsubsection*{Mean}
\[
    \text{Undefined}
\]

\subsubsection*{Variance}
\[
    \text{Undefined}
\]

\subsubsection*{Skewness}
\[
    \text{Undefined}
\]

Reference:
\href{https://mathworld.wolfram.com/CauchyDistribution.html}{WolframMathWorld}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Chi distribution}

\subsubsection*{PDF}
\[
    P(x) = \frac{2^{1-k/2} x^{k-1} \exp{(-x^2/2)}}{\Gamma(k/2)}
\]
where $k$ is the degrees of freedom.

\subsubsection*{CDF}
\[
    D(x) = P(k/2, x^2/2)
\]
where $P(a,b)$ is the regularized gamma function.

\subsubsection*{Mean}
\[
    \mu = \sqrt{2} \frac{\Gamma((k+1)/2)}{\Gamma(k/2)}
\]

\subsubsection*{Variance}
\[
    \sigma^2 = k-\mu^2
\]

\subsubsection*{Skewness}
\[
    \frac{\mu}{\sigma^3} (1-2\sigma^2)
\]

Reference:
\href{https://en.wikipedia.org/wiki/Chi_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Chi Squared distribution}

A Chi Squared distribution with $k$ degrees of freedom is the
distribution of a sum of the squares of $k$ independent
standard normal random variables.

\subsubsection*{PDF}
\[
    P(x) = \frac{x^{k/2-1} \exp(-x/2)}{2^{k/2} \Gamma(k/2)}
\]
where $k$ is the degrees of freedom; $\Gamma(a)$ is the gamma function.

\subsubsection*{CDF}
\[
    D(x) = P(k/2, x/2)
\]
where $P(a,b)$ is the regularized gamma function.

\subsubsection*{Mean}
\[
    \mu = k
\]

\subsubsection*{Variance}
\[
    2k
\]

\subsubsection*{Skewness}
\[
    \sqrt{8/k}
\]

Reference:
\href{https://en.wikipedia.org/wiki/Chi-squared_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Erlang distribution}

\subsubsection*{PDF}
\[
    P(x) = \frac{\lambda^k x^{k-1} \exp{(\lambda x)}}{(k-1)!}
\]
where $k$ is the shape parameter; $\lambda$ is the rate parameter.

\subsubsection*{CDF}
\[
    D(x) = \frac{\gamma(k, \lambda x)}{(k-1)!}
\]
where $\gamma(a,b)$ is the lower gamma function.

\subsubsection*{Mean}
\[
    \frac{k}{\lambda}
\]

\subsubsection*{Variance}
\[
    \frac{k}{\lambda^2}
\]

\subsubsection*{Skewness}
\[
    \frac{2}{\sqrt{k}}
\]

Reference:
\href{https://en.wikipedia.org/wiki/Erlang_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Exponential distribution}

\subsubsection*{PDF}
\[
    P(x) = \begin{cases}
        \lambda \exp{(-\lambda x)} & x\geq 0,\\
        0 & x<0.
        \end{cases}
\]

\subsubsection*{CDF}
\[
    D(x) = \begin{cases}
        1-\exp{(-\lambda x)} & x \ge 0, \\
        0 & x < 0.
        \end{cases}
\]

\subsubsection*{Mean}
\[
    \frac{1}{\lambda}
\]

\subsubsection*{Variance}
\[
    \frac{1}{\lambda^2}
\]

\subsubsection*{Skewness}
\[
    2
\]

Reference:
\href{https://en.wikipedia.org/wiki/Exponential_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Gamma distribution}

\subsubsection*{PDF}
\[
    P(x)= \frac{1}{\Gamma(\alpha)\theta^{\alpha}}
            x^{\alpha-1} \exp{(-\frac{x}{\theta})}
\]
where $\alpha$ is the shape parameter; $\theta$ is the scale parameter.

\subsubsection*{CDF}
\[
    D(x)= \frac{1}{\Gamma(\alpha)} \gamma(\alpha, \frac{x}{\theta})
\]

\subsubsection*{Mean}
\[
    k \theta
\]

\subsubsection*{Variance}
\[
    k \theta^2
\]

\subsubsection*{Skewness}
\[
    \frac{2}{\sqrt(\alpha)}
\]

Reference:
\href{https://mathworld.wolfram.com/GammaDistribution.html}{WolframMathWorld}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Irwin-Hall distribution}

\subsubsection*{PDF}
\[
    P(x) = \frac{1}{(n-1)!} \sum_{{k=0}}^{{\lfloor x\rfloor }}
        (-1)^{k}{\binom{n}{k}}(x-k)^{{n-1}}
\]
where $n$ is number of IDD of uniform distributions.

\subsubsection*{CDF}
\[
    D(x) = \frac{1}{n!} \sum_{{k=0}}^{{\lfloor x\rfloor}}
        (-1)^{k}{\binom{n}{k}}(x-k)^{n}
\]

\subsubsection*{Mean}
\[
    \frac{n}{2}
\]

\subsubsection*{Variance}
\[
    \frac{n}{12}
\]

\subsubsection*{Skewness}
\[
    0
\]

Reference:
\href{https://en.wikipedia.org/wiki/Irwin%E2%80%93Hall_distribution}{WolframMathWorld}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Non-central Chi distribution}

\subsubsection*{PDF}
\[
    P(x) = {\frac{\exp{(-(x^{2}+\lambda^{2})/2)}x^{k}\lambda}
        {(\lambda x)^{k/2}}}
        I_{k/2-1}(\lambda x)
\]
where $k$ is the degrees of freedom; $\lambda$ is the distance parameter;
$I_M(a)$ is a modified cylindrical Bessel function of the first kind.

\subsubsection*{CDF}
\[
    D(x) = 1-Q_{\frac {k}{2}}\left(\lambda ,x\right)
\]
where $Q_M(a,b)$ is Marcum Q-function.

\subsubsection*{Mean}
\[
    \text{To be implemented.}
\]

\subsubsection*{Variance}
\[
    \text{To be implemented.}
\]

Reference:
\href{https://en.wikipedia.org/wiki/Noncentral_chi_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Non-central Chi Squared distribution}

\subsubsection*{PDF}
\[
    P(x) = {\frac {1}{2}}e^{-(x+\lambda )/2}\left({\frac {x}{\lambda }}\right)
    ^{k/4-1/2}I_{k/2-1}({\sqrt {\lambda x}})
\]
where $k$ is the degrees of freedom; $\lambda$ is the distance parameter;

\subsubsection*{CDF}
\[
    D(x) = 1-Q_{\frac {k}{2}}\left({\sqrt {\lambda }},{\sqrt {x}}\right)
\]
where $Q_M(a,b)$ is Marcum Q-function.

\subsubsection*{Mean}
\[
    k + \lambda
\]

\subsubsection*{Variance}
\[
    2(k+2\lambda)
\]

\subsubsection*{Skewness}
\[
    \frac{2^{3/2}(k+3\lambda)}{(k+2\lambda)^{3/2}}
\]

Reference:
\href{https://en.wikipedia.org/wiki/Noncentral_chi-squared_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Normal distribution}

\subsubsection*{PDF}
\[
    P(x) = {\frac{1}{\sigma{\sqrt {2\pi }}}}
        \exp{(-{\frac {1}{2}}
        \left({\frac {x-\mu }{\sigma }}\right)^{2})}
\]
where $\mu$ is the expection of the distribution;
$\sigma$ is the standard deviation.

\subsubsection*{CDF}
\[
    D(x) = {\frac{1}{2}} \left[1+\operatorname {erf}
        \left({\frac {x-\mu }{\sigma {\sqrt {2}}}}\right)\right]
\]
where $erf(a)$ is the error function.

\subsubsection*{Mean}
\[
    \mu
\]

\subsubsection*{Variance}
\[
    \sigma^2
\]

\subsubsection*{Skewness}
\[
    0
\]

Reference:
\href{https://en.wikipedia.org/wiki/Normal_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Rayleigh distribution}

\subsubsection*{PDF}
\[
    P(x) = {\frac {x}{\sigma ^{2}}}e^{-x^{2}/\left(2\sigma ^{2}\right)}
\]

\subsubsection*{CDF}
\[
    D(x) = 1 - e^{-x^{2}/\left(2\sigma ^{2}\right)}
\]

\subsubsection*{Mean}
\[
    \sigma \sqrt{\frac{\pi}{2}}
\]

\subsubsection*{Variance}
\[
    {\frac {4-\pi }{2}}\sigma ^{2}
\]

\subsubsection*{Skewness}
\[
    {\frac {2{\sqrt {\pi }}(\pi -3)}{(4-\pi )^{3/2}}}
\]

Reference:
\href{https://en.wikipedia.org/wiki/Rayleigh_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Standard Normal distribution}

\subsubsection*{PDF}
\[
    P(x) = \frac{1}{\sqrt {2\pi }}
        \exp{(-{\frac {1}{2}} x^{2})}
\]
where $\mu$ is the expection of the distribution;
$\sigma$ is the standard deviation.

\subsubsection*{CDF}
\[
    D(x) = {\frac{1}{2}} \left[1+\operatorname {erf}
        \left( {\frac{x}{{\sqrt {2}}}} \right) \right]
\]
where $\operatorname {erf}(a)$ is the error function.

\subsubsection*{Mean}
\[
    0
\]

\subsubsection*{Variance}
\[
    1
\]

\subsubsection*{Skewness}
\[
    0
\]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Uniform distribution (Continuous)}

\subsubsection*{PDF}
\[
    P(x) = \begin{cases}
        {\frac {1}{b-a}}&{\text{for }}x\in [a,b]\\
        0&{\text{otherwise}}
    \end{cases}
\]
where $a$ is the lower bound; $b$ is the upper bound.

\subsubsection*{CDF}
\[
    D(x) = \begin{cases}
        0&{\text{for }}x<a\\
        {\frac {x-a}{b-a}}&{\text{for }}x\in [a,b]\\
        1&{\text{for }}x>b
    \end{cases}
\]

\subsubsection*{Mean}
\[
    \frac{1}{2}(a+b)
\]

\subsubsection*{Variance}
\[
    \frac{1}{12}(b-a)^2
\]

\subsubsection*{Skewness}
\[
    0
\]

Reference:
\href{https://en.wikipedia.org/wiki/Continuous_uniform_distribution}{Wikipedia}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Mixture distribution}

A \textbf{mixture distribution} is the probability distribution of a random variable
that is derived from a collection of other random varibales.
The probability density function (and the cumulative distribution function)
can be expressed as the a convex combination of other distribution functions.
The individual distributions that are combined to form the mixture distribution are called
the \textbf{mixture components}. The weights associated with each component are called the \textbf{mixture weights}.
(\href{https://www.overleaf.com/learn/latex/Bold,_italics_and_underlining}{Wikipedia})

\subsubsection*{PDF}
Given the mixture components' PDF ${P_1(x), ..., P_n(x)}$ and weights ${w_1,...,w_n}$,
the mixture's PDF is a convex combination:
\[
    P_\text{mixture}(x) = \sum^n_{i=1} w_i P_i(x)
\]

\subsubsection*{CDF}
Given the mixture components' CDF ${D_1(x), ..., D_n(x)}$ and weights ${w_1,...,w_n}$,
the mixture's CDF is a convex combination:
\[
    D_\text{mixture}(x) = \sum^n_{i=1} w_i D_i(x)
\]

The moments of a mixture distribution are not as mathematically simple as PDF or CDF.
To acquire the mathematically expressions for Mean, Variance, and Skewness, we need to
clarify the three types of moments:
\begin{enumerate}
    \item $k^\text{th}$ non-central moment: $\mu^{(k)} = \mathbb{E}[x^k]$
    \item $k^\text{th}$ central moment: $\mu^{(k)}_c = \mathbb{E}[(x-\mu^{(1)})^k]$
    \item $k^\text{th}$ standardized moment: $\mu^{(k)}_s = \mathbb{E}[(\frac{x-\mu^{(1)}}{\sigma})^k]$
\end{enumerate}
, where $^{(k)}$ denotes the $k^\text{th}$ moment.

\subsubsection*{Mean}
Mean is also known as the first non-central moment.
The $k^\text{th}$ non-central moment of a random variable can be rewritten in terms of integrals as:
\begin{align}
    \mu^{(k)} &= \mathbb{E}[x^k] \\
              &= \int_{-\infty}^{\infty} x^k f(x) dx \\
              &= \int_{-\infty}^{\infty} x^k \sum^n_{i=1} w_i f_i(x) dx \\
              &= \sum^n_{i=1} w_i \int_{-\infty}^{\infty} x^k f_i(x) dx \\
              &= \sum^n_{i=1} w_i \mathbb{E}_{f_i}[x^k] \\
              &= \sum^n_{i=1} w_i \mu_i^{(k)}
\end{align}
, where $\mu_i^{(k)}$ is the $k^\text{th}$ non-central moment of distribution function $f_i$.

Given the mixture components' mean ${\mu_1, ..., \mu_n}$ and weights ${w_1,...,w_n}$,
it is easy to see the mixture's mean is just a convex combination:
\[
    \mu^{(1)}_\text{mixture} = \sum^n_{i=1} w_i \mu^{(1)}_i
\]

\subsubsection*{Variance}
Variance is also known as the second central moment. The $k^\text{th}$ central moment
follows a deriavation similar to the non-central moment:
\begin{align}
    \sigma^2 \coloneqq \mu^{(2)}_c &= \mathbb{E}[(x-\mu^{(1)})^{2}] \\
        &= \mathbb{E}[x^2] - (\mathbb{E}[x])^2 \\
        &= \mu^{(2)} - (\mu^{(1)})^2 \\
        &= \sum^n_{i=1} w_i \mu^{(2)}_i - \left(\sum^n_{i=1} w_i \mu^{(1)}_i \right)^2  \\
        &= \sum^n_{i=1} w_i \left( \sigma^2_i + (\mu^{(1)}_i)^2 \right) - \left(\sum^n_{i=1} w_i \mu^{(1)}_i \right)^2  \\
        &= \sum^n_{i=1} w_i \sigma^2_i + \sum^n_{i=1} w_i (\mu^{(1)}_i)^2 - \left(\sum^n_{i=1} w_i \mu^{(1)}_i \right)^2
\end{align}
, where $\sigma_i^2$ is $i^{th}$ component's variance, and $\mu^{(1)}_i$ is $i^{th}$ component's mean.

With simplified the notations, we get a familiar expression for the mixture's variance.
\[
    \sigma^2_\text{mixture} = \sum^n_{i=1} w_i (\sigma^2_i+\mu^2_i) - \mu^2_\text{mixture}
\]

\subsubsection*{Skewness}
Skewness is also known as the third standardized moment.
\begin{align}
    \mu^{(3)}_s &= \mathbb{E}[ \left( \frac{x-\mu^{(1)}}{\sigma} \right) ^3] \\
                &= \frac{\mathbb{E}[x^3] - 3 \mu^{(1)} \mathbb{E}[x^2] + 3 (\mu^{(1)})^2 \mathbb{E}[x] - (\mu^{(1)})^3}{\sigma^3} \\
                &= \frac{\mathbb{E}[x^3] - 3 \mu^{(1)} \sigma^2 - (\mu^{(1)})^3}{\sigma^3}
\end{align}

To rewrite it in terms of the skewness of each component $(\mu^{(3)}_s)_i$, let's start by rewriting the third non-central moment:
\[
    \mathbb{E}[x^3] = \sum^n_{i=1} w_i \mu^{(3)}_i = \sum^n_{i=1} w_i \left( \sigma^3_i (\mu^{(3)}_s)_i + 3 \mu^{(1)}_i \sigma^2_i + (\mu^{(1)}_i)^3 \right)
\]

Now the Skewness of the mixture can be rewritten to:
\[
    \mu^{(3)}_s = \frac{ \sum^n_{i=1} w_i \left( \sigma^3_i (\mu^{(3)}_s)_i + 3 \mu^{(1)}_i \sigma^2_i + (\mu^{(1)}_i)^3 \right)  - 3 \mu^{(1)} \sigma^2 - (\mu^{(1)})^3 }{\sigma^3}
\]
where $\mu^{(1)}$ is the mean of the mixture, and $\sigma^2$ is the variance of the mixture.


% !TEX encoding = UTF-8 Unicode
% !TEX spellcheck = en_US
% !TEX root = ../../../ICMA2020.tex

\subsection{Parameter Estimation}
\label{subsec:ParameterEstimation}

The base parameter form of the dynamics in (\ref{eq:model_regressor}) is used to estimate
the parameter vector $\boldsymbol{\theta} \in \mathbb{R}^{35}$.
%
Using a sufficient number of samples in the estimation process leads to an over-determined system of linear equations:

\begin{equation} \label{eq:ParameterEstimation_problem}
\begin{aligned}
    \boldsymbol{y} &= \boldsymbol{C} \boldsymbol{\theta} + \boldsymbol{\epsilon}, \\
    \begin{pmatrix}
        \boldsymbol{\tau}(t_1) \\
        \vdots \\
        \boldsymbol{\tau}(t_N) \\
    \end{pmatrix} &=
    \begin{pmatrix}
        \boldsymbol{X}(t_1) \\
        \vdots \\
        \boldsymbol{X}(t_N) \\
    \end{pmatrix} \boldsymbol{\theta} +
    \begin{pmatrix}
        \boldsymbol{e}(t_1) \\
        \vdots \\
        \boldsymbol{e}(t_N) \\
    \end{pmatrix}.
\end{aligned}
\end{equation}

Here $\boldsymbol{y}$ contains the $N$ samples of the torque vector $\boldsymbol{\tau}$, $\boldsymbol{C}$ denotes the design matrix and $\boldsymbol{\epsilon}$ is the vector of errors.

Equation \eqref{eq:ParameterEstimation_problem} can be solved for $\boldsymbol{\theta}$ using a least squares estimator. The weighted least squares (WLS)  \cite{Gautier.2013,M.Gautier.1995,V.Bargsten.2013,Swevers.1997} is used to account for different levels of noise in the measurements:
\begin{equation} \label{eq:ParameterEstimation_WLS}
    \begin{aligned}
	    \hat{\boldsymbol{\theta}} &= \arg \min_{{\boldsymbol{\theta}}} (\boldsymbol{y} - \boldsymbol{C} \boldsymbol{\theta})^\mathrm{T} \boldsymbol{W} (\boldsymbol{y} - \boldsymbol{C} \boldsymbol{\theta}) \\
	    &= (\boldsymbol{C}^\mathrm{T} \boldsymbol{W} \boldsymbol{C})^{-1} \boldsymbol{C}^\mathrm{T} \boldsymbol{W} \boldsymbol{y}.
	\end{aligned}
\end{equation}
Matrix $\boldsymbol{W}$ contains the weights of the individual samples, see also Sec.\,\ref{subsec:SignalProcessing}.


\section*{Ex.2.5}
\subsection*{what is the probability that there is a key $y \in S$ such that $h(y) = h(x)$ and $s(y) = s(x)$}
The set $(h(x)=h(y))\cap(s(x)=s(y))$ is the set where the hash function and the signature function gives the same value for two distinct keys $x,y$. Hence the probability we want to determine is the probability for that set over all possible $y$ values in $S$. So we look at
\begin{align*}
\sum_{y\in S}P[(h(x)=h(y))\cap(s(x)=s(y))]
             &= \sum_{y\in S}P(h(x)=h(y))P(s(x)=s(y))\\
             &= nP(h(x)=h(y))P(s(x)=s(y))\\
             &\leq n\frac{1}{n}\frac{1}{n^3}\\
             &= \frac{1}{n^3}
\end{align*}
The first equality holds because the hash functions are independent. The next equality uses that $\abs{S}=n$. The inequality uses that the $h$ and $s$ are universal hash functions. So the probability for the hash function and for the signature function to have a collision for two distinct keys are less than $1/n^3$


\section{Probability modeling}
Find $h(x)$ that min. pred. error:\\
$\highlight*{R(h) = \mathbb{E}_{x,y}[\ell(y;h(x))]} = \int\!P(x,y)\,\ell(y;h) \diff x \!\diff y$

% ===
\emph{Bayes optimal predictor
\normalfont\sffamily $\normalcolor\to \hat y = h^\ast(x)$}

$\deriv{\ell(\hat y)}{\hat y} = 0 \to \highlight*{\hat y = \mathbb{E} [Y\vert X\!\!=\!x] = \int\!y\!\cdot\!\hat P(y\vert X\!\!=\!x) \diff y}$

%\subsection*{For least squares regression}
%Best $h$: $h^*(x) = \mathbb{E}[Y|X=x]$ \\
%Pred.: $\hat{y} = \hat{\mathbb{E}}[Y|X=\hat{x}] = \int \hat{P}(y|X=\hat{x}) y \partial y$

% ===
\emph{MLE \enskip $-\log P(y\vert x,w)$}

{\small $\theta^* \!=\! \arg\!\max\limits_\theta \hat{P}(y_{1:n}\vert x_{1:n},\theta) \overset{\textrm{iid}}{=} \highlight*{\arg\!\min\limits_\theta \!-\!\sum_{i=1}^n \ln P(y_i\vert x_i,\theta)} $}

e.g. lin. Gauss: $y_i = w^\top \!x_i + \varepsilon_i$, \enskip $\varepsilon_i \!\sim\! \mathcal{N}(0, \sigma^2)$\\
i.e. $y_i \!\sim\! \mathcal{N}(w^\top \!x_i, \sigma^2)$
$\xrightarrow{\text{MLE and}\,\log\;}$ LS regression

% ===
\emph{Bias/Variance/Noise}

Prediction error = $\normalcolor \textrm{Bias}^2 + \textrm{Variance} + \textrm{Noise}$\\
- Noise: risk incurred by the optimal model,\\
\phantom{- Noise:} $P(y\vert x)$ loss/likelihood fct.\\
- Variance: est. model from limited data\\
- Bias: incurred by regularizer\\
higher bias implies much lower variance

% ===
\emph{MAP \enskip $-\log P(w)$}

\textbf{Prior:} bias on param's, e.g. $w_i \!\sim\! \mathcal{N}(0, \beta^2)$\\
$\xrightarrow{\text{\!Bay.}}$ $P(w|x_{1:n},y_{1:n}) = \frac{P(w|x) P(y|x,w)}{P(y|x)} = \frac{P(w) P(y|x,w)}{P(y|x)}$

% ===
\emph{Logistic regression (Classification)}

\textbf{Link fct.:} $\sigma(w^\top x) = \frac{1}{1+\exp(-w^\top x)}$ \enskip \textbf{(Sigmoid)}\\
$P(y\vert x,w) = Ber (y; \sigma(w^\top\!x)) = \sigma(yw^\top x)$

\textbf{MLE:}
$\hat w = \arg\!\min\limits_w \sum_i \log(1+\exp(-y_iw^\top\!x_i))$\vspace{-3pt}\\
\phantom{\textbf{MLE:}} with $\hat R(w) = \sum_{i=1}^n \ell\ped{logistic} (w; x_i, y_i)$

\textbf{Grad.:} $\nabla\!_w \ell(w) = P(Y\!\!\neq\!y \vert x) \; (-yx)$
\hfill {\color{gray}\small $\to (Y\!\!=\!-\!y)$}

\textbf{MAP:} Gauss. prior $\to \norm{w}_2^2$, \enskip Lap. $\to \norm{w}_1^1$


% -*- root: Main.tex -*-
\section{Notes}
\textbullet Proof that eigenvectors of symmetric matrix are orthogonal: $\lambda \langle x,y \rangle = \langle \lambda x,y \rangle = \langle Ax,y \rangle = \langle x,A^\top y \rangle = \langle x,A y \rangle = \langle x,\mu y \rangle = \mu \langle x,y \rangle \Rightarrow (\lambda-\mu)\langle x,y\rangle=0$ where $\lambda \neq \mu$.\\
\textbullet K-means vs GMM: k-means has hard assignments, cheaper to train (less params). GMM has soft assginments, more expressive because cluster is described by a MVN i.e shape is defined by an arbitrary covariance martix and not restricted to spherical clusters. GMM is generative model i.e we can do outlier detection, generate data points, uncertainty estimation. \\
\textbullet SVD and PCA: if $A=UDV^\top$. Columns of $U$ are eigenvectors of $AA^\top$. Columns of $V$ are eigenvectors of $A^\top A$. Eigenvalues of $A^\top A$ and $AA^\top$ are singular values$^2$ of $A$. \\
\textbullet orthogonal Haar Basis for $4$-dim signals: \\
$U = \frac{1}{2}
\begin{bmatrix}
    1 & 1 & \sqrt{2} & 0 \\
    1 & 1 & -\sqrt{2} & 0 \\
    1 & -1 & 0 & \sqrt{2} \\
    1 & -1 & 0 & -\sqrt{2}
\end{bmatrix}
$


\lab{Algorithm}{Pseudorandom Number Generators}{Pseudorandom Number Generators}
\label{lab:PRNG}

\objective{Learn about the strengths and weaknesses of a few a pseudorandom number generators}

\section*{Random Numbers}
Lotteries, most board games, and statistics need random numbers.
In real life, we roll dice, take balls out of a bag, or spin a wheel.
Computers are, by nature, deterministic, meaning that they do exactly what they are told.
Because of this, random number generation on a computer can be difficult.
We can have a device measure a random process and use the data to generate random numbers,
But such sampling is often too slow and too expensive for practical use.
Pseudorandom number generators (PRNGs) are a common solution to this problem.
The numbers are not truly random, but they are based on a complex formula that makes them look ``random."
For convenient use, these generateors must also run quickly.
The goal is to have something that is fast and looks random.

There are many different algorithms for developing pseudorandom numbers.
Robert R. Coveyou titled an article ``The generation of random numbers is too important to be left to chance."
There has been much study about different PRNGs.
This lab will cover two of them: Linear Congruiental Generators and The Mersenne Twister.

\section*{Linear Congruential Genterators}
Linear Congruential Generators (LCG) are one of the oldest ways of generating random numbers.
The Generator is defined by the recurrence relation:
$X_{n+1}=(a*X_n + c)$ mod $m$ where

\begin{itemize}
\item $X$ is a sequence of pseudorandom values, and
\item $m$, $0<m$ is the modulus
\item $a$, $0<a<m$ is the multiplier
\item $c$, $0\leq c<m$ is the increment
\item $X_0$, $0\leq X_0 <m$ is the seed
\end{itemize}

each of these are integer constants.

\begin{problem}
Write a LCG that produces an array of pseudorandom numbers between $0.0$ and $1.0$.
Let the arguments be size of the array and let a, c, mod, and seed be optional arguments.
I recommend $a=1103515245$, $c=12345$, $m=2^{31}-1$, and $seed=4329$ as the default values.
\end{problem}

\begin{problem}
Write a LCG that produces an array of pseudorandom numbers of integers between two input arguments.
Do it by calling your algorithm from problem one and multiplying it by the values and casting the array as an int using the .astype() function.
Let the arguments be size of array and the two integers.
Let a, c, mod, and seed continue to be optional arguments.
\end{problem}

One easy way to ``see" if your generator is random is to look at a bitmap of the output.
In python, use the plt.imshow() function to see a bitmap of the array produced by your LCG.
Resize your output to be $512 \times 512$.

\begin{figure}
\includegraphics[width=.4\textwidth]{PRNG1.png}
\caption{
The bitmap with $a=3$, $c=2$, $m=2^{16}$.
There is a clear pattern in the random numbers.}
\end{figure}

\begin{problem}
For what values of $a$, $c$, and $m$ does your LCG have a visible pattern.
\end{problem}

This method is not rigorous, but there are several other ways to test the randomness of your output.

The length over which your random number generator repeats is called the period.
The period is at most m, but it may be shorter based on the values of a and c.

\begin{comment}
According to the Hull-Dobell Theorem (TO DO: find a source), a LCG will have a full period if and only if,
1. $c$ and $m$ are relatively prime,
2. $a-1$ is divisible by all prime factors of $m$,
3. $a-1$ is a multiple of 4 if $m$ is a multiple of 4

\begin{problem}
Test values of $a$,$c$, and $m$ that fit these requirements.
\end{problem}
\end{comment}

This algorithm is used as the default random number generator in Java, and C++ and is still used in a wide variety of situations.

\section*{Mersenne Twister}
\begin{comment}
(TO DO: decide how much of  this we want to keep) All numbers can be represented in bits as a base two number.
Computers are optimized to work with numbers in that manner.
The operators XOR, OR, and AND work on the bit representation of two numbers.

AND - if both numbers have a 1 in the ith place then the ith place is 1.
Otherwise the ith place is 0.

OR - if one or both numbers have a 1 in the ith place then the ith place is 1.
Otherwise the ith place is 0.

XOR - if only one of the two numbers has a 1 in the ith place then the ith place is 1.
If both or neither of the numbers has a 1 in the ith place, the ith place is 0.

In addition you can shift the bitwise number over a number of values.
For example, shifting 10100 to the right by one yields 1010 and shifting it to the left by one yields 101000.
This is really just division and multiplication by 2.
This can be done by $\ll$ and $\gg$ in python.
\end{comment}

The Mersenne twister PRNG does a series of bitwise operations to generate random numbers.
The Random class in python uses the Mersenne twister algorithm.

\begin{problem}
Look at the bitmap of output of \li{np.random.rand(512,512)}.
Can you see any patterns?
\end{problem}

\section*{Randomness Tests}
There have been statistical tests devised for measuring the quality of a random number generator. One is these is the overlapping permutations test. Analyze sequences of five consecutive random numbers. The 120 possible orderings should occur with statistically equal probability.

\begin{problem}
Use the overlappng permutations test to see how random python's random number generator is compared to the LCG you wrote in problem 1.
\end{problem}

\section*{Blackjack}
\begin{figure}
\includegraphics[width=\textwidth]{Blackjack_game_1.jpg}
\caption{Initial Round of a BlackJack game.}
\end{figure}

Black Jack is a card game that involves the use of randomness.
The game is simple.
The dealer deals the player and himself each two cards.
He flips over his first card so that the player can see it.
The player has to choose to take another card ("hit") or not ("stand").
If the player hits he gets another card and again has the choice to hit or stand.

The goal is to get your hand to be at or as close to 21 without going over.
Face cards are worth 10 points.
Aces can count either as 11 or 1.
The value of all other cards are equal to number on the card.

Once the player has decided to stand the dealer flips over his second card and deals himself cards until his hand value is 17 or greater.

If the player value goes above 21 he automatically loses.
If his value is 21 and below and dealer has above 21 then the player wins.
If they both have 21 or under than the player with the hand of highest value wins.
If both hands have the same value, the game is a tie.

\section*{Shuffling Algorithms}
One use of Pseudorandom Number Generators (PRNGs) is to shuffle cards.
The main goal of these algorithms is that the card order be random--so that no single player has an advantage based on order.
Often, as strange as it may seem, online gambling sites will post their shuffling algorithms online.
The only things they do not post are their seed values.
Often the time in milliseconds from midnight is used as the seed value.

John von Neumann said ``Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin."
As seen in the last lab, weak PRNGs are periodic and are predictable once a few outputs are known.
This lab will have you break blackjack based on a weak PRNG.

\section*{Cracking Blackjack}
For these next problems you will need three files that are provided with this lab: Black.py, BlackEasy.py, and bjHelp.py.
Black.py and BlackEasy.py are are programs that run games of Blackjack that use a Linear Congruentail Generator (LCG) to shuffle the cards.
They generate 52 random numbers and then the argsort of those numbers is the order of the cards.
The parameters for BlackEasy.py are a$=2521$, c$=13$, mod$=2^{16}$.
For Black.py they are a$=25214903917$, c$=11$, mod$=2^{48}$.
In order to play them type \li{python <<filename>> <<numberofgames>>} in your command line.
They are both seeded initially by the time.

bjHelp.py contains two functions that will help you "predict" the cards:
SuffleHack(n,a,c,mod,seed) gives the first $n$ card shuffles given the parameters for a LCG.
The shuffles are represented by numbers.
Hacker(Stats,['card','card','card']) Stats is the output of SuffleHack and takes a list of 3 cards (see below).
It prints all shuffles as a list of cards in Stats that have the same first three cards as the inputted list.

The trick to being able to "predict" the cards is to find the initial seed value.

Cards- A, 2-10, J, Q, or K combined with heart, diamond, club, or spade in single quotes.
Examples: '6diamond', 'Kclub'.

\begin{warn}
Both BlackEasy.py and Black.py use functions that are incompatible with ipython. They need to run \li{python Black.py} in command line.
\end{warn}

\begin{problem}
Play 10 games of BlackEasy.py and by the 5th game be able to predict the cards.
You can write your own functions or use the ones in bjHelp.py.
You will want to open two command prompts, one to play the game and one to predict the cards.
\end{problem}

Not too hard.
That is because there is only $2^{16}$ seed values.
This next one you will have to look at more hands until you can find out the initial seed value.

\begin{problem}
Play 20 games of Black.py and by the 15th game be able to predict the cards.
\end{problem}


\section{Deterministic finite automata}

Transition diagrams are useful graphical representations of instances
of the mathematical concept of \emph{deterministic finite automaton}
(\emph{DFA}). Formally, a DFA \(\mathcal{D}\) is a 5-tuple
\(\mathcal{D} = (Q, \Sigma, \delta, q_0, F)\) where
\begin{enumerate*}

  \item a finite set of \emph{states}, often noted \(Q\);

  \item an \emph{initial state} \(q_0 \in Q\);

  \item a set of \emph{final (\emph{or} accepting)
  states} \(F \subseteq Q\);

  \item a finite set of \emph{input symbols}, often noted \(\Sigma\);

  \item a \emph{transition function} \(\delta\) that takes a state and
    an input symbol and returns a state: if~\(q\) is a state with an
    edge labelled \(a\), the edge leads to the state \(\delta(q, a)\).

\end{enumerate*}

\subsection*{Recognised words}

Independently of the interpretation of the states, we can define how a
given word is accepted (or recognised) or rejected by a given DFA. For
example, the word \(a_1 a_2 \cdots a_n\), with \(a_i \in \Sigma\), is
recognised by the DFA \(\mathcal{D} = (Q, \Sigma, \delta, q_0, F)\)
if, for all \(0 \leqslant i \leqslant n-1\), there is a sequence of
states \(q_i \in Q\) such that \(\delta (q_i, a_{i+1}) = q_{i+1}\) and
\(q_n \in F\). The language recognised by \(\mathcal{D}\), noted
\(L(\mathcal{D})\) is the set of words recognised
by~\(\mathcal{D}\). For example, consider the DFA in
\fig~\vref{fig:trie_then}.
\begin{figure}[b]
\centering
\includegraphics[bb=48 644 218 730,scale=0.93]{trie_then}
\caption{A trie recognising \textsf{they}, \textsf{then},
  \textsf{this} and \textsf{thus} \label{fig:trie_then}}
\end{figure}
The word \textsf{then} is recognised because there is a sequence of
states \((q_0, q_1, q_2, q_4, q_5)\) connected by edges which
satisfies \(\delta (q_0, \textsf{t}) = q_1, \delta (q_1, \textsf{h}) =
q_2, \delta (q_2, \textsf{e}) = q_4 \,\text{and}\, \delta (q_4,
\textsf{n}) = q_5\), with \(q_5 \in F\), that is, \(q_5\)~is a final
state.

\subsection*{Recognised language}

It is easy to define formally \(L(\mathcal{D})\). Let \(\mathcal{D} =
(Q, \Sigma, \delta, q_0, F)\). First, let us extend~\(\delta\) to
words and let us call this extension~\(\hat{\delta}\):
\begin{itemize*}

  \item for all state \(q \in Q\), let \(\hat{\delta} (q, \varepsilon)
    = q\), where \(\varepsilon\) is the empty string;

  \item for all state \(q \in Q\), all word \(w \in \Sigma^{*}\), all
    input \(a \in \Sigma\), let us define \(\hat{\delta} (q, wa) =
    \delta (\hat{\delta}(q,w),a)\).

\end{itemize*}
Then the word \(w\) is recognised by \(\mathcal{D}\) if
\(\hat{\delta}(q_0, w) \in F\). The language \(L(\mathcal{D})\)
recognised by~\(\mathcal{D}\) is defined as \(L(\mathcal{D}) = \{w \in
\Sigma^{*} \; \lvert \; \hat{\delta}(q_0, w) \in F\}\). For example,
in our last example:
\begin{equation*}
\begin{aligned}
   \hat{\delta}(q_0, \epsilon)
&= q_0,\\
   \hat{\delta}(q_0, \textsf{t})
&= \delta (\hat{\delta}(q_0, \epsilon), \textsf{t})
= \delta (q_0, \textsf{t})
= q_1,\\
   \hat{\delta}(q_0, \textsf{th})
&= \delta (\hat{\delta}(q_0, \textsf{t}), \textsf{h})
= \delta (q_1, \textsf{h})
= q_2,\\
   \hat{\delta}(q_0, \textsf{the})
&= \delta (\hat{\delta}(q_0, \textsf{th}), \textsf{e})
= \delta (q_2, \textsf{e})
= q_4,\\
   \hat{\delta}(q_0, \textsf{then})
&= \delta (\hat{\delta}(q_0, \textsf{the}), \textsf{n})
= \delta (q_4, \textsf{n})
= q_5 \in F.
\end{aligned}
\end{equation*}

\subsection*{Transition diagrams}

We can also redefine transition diagrams in terms of the concept of
DFA. A transition diagram for a DFA \(\mathcal{D} = (Q, \Sigma,
\delta, q_0, F)\) is a graph defined as follows:
\begin{enumerate*}

  \item for each state \(q\) in \(Q\) there is a \emph{node}, \emph{i.e.,} a
    single circle with \(q\) inside;

  \item for each state \(q \in Q\) and each input symbol \(a \in
    \Sigma\), if \(\delta (q, a)\) exists, then there is an
    \emph{edge}, \emph{i.e.,} an arrow, from the node denoting~\(q\) to the
    node denoting \(\delta (q, a)\) labelled by \(a\); multiple edges
    can be merged into one and the labels are then separated by
    commas;

  \item there is an edge coming to the node denoting~\(q_0\) without
    origin;

  \item nodes corresponding to final states are doubly circled.

\end{enumerate*}
Here is a transition diagram for the language over alphabet \(\{0,
1\}\), called \emph{binary alphabet}, which contains the
string~\(01\):
\begin{center}
\includegraphics[bb=48 710 185 760]{dfa_01}
\end{center}

\subsection*{Transition table}

There is a compact textual way to represent the transition function of
a DFA: a \emph{transition table}. The rows of the table correspond to
the states and the columns correspond to the inputs (symbols). In
other words, the entry for the row corresponding to state~\(q\) and
the column corresponding to input~\(a\) is the state~\(\delta (q,
a)\), as seen in \fig~\ref{fig:gen_table}.
\begin{figure}
\centering
\subfloat[General table\label{fig:gen_table}]{
\(\begin{array}{c||c|c|c}
\delta & \ldots & a & \ldots\\
\hhline{=::===}
\vdots & & &\\
\hline
q & & \delta (q, a)\\
\hline
\vdots & & &
\end{array}
\)}
\qquad
\subfloat[Example\label{fig:example_table}]{
\(\begin{array}{r@{}l||c|c}
\multicolumn{2}{c||}{\mathcal{D}} & 0 & 1\\
\hhline{==::==}
\rightarrow & q_0 & q_1 & q_0\\
            & q_1 & q_1 & q_2\\
         \# & q_2 & q_2 & q_2\\
            &     &     &
\end{array}\)
}
\caption{Transition tables}
\end{figure}
For instance, the transition table corresponding to the
function~\(\delta\) of our last example is found in
\fig~\vref{fig:example_table}. Actually, we added some extra
information in the table: the initial state is marked with
\(\rightarrow\) and the final states are marked
with~\(\#\). Therefore, it is not only~\(\delta\) which is defined by
means of the transition table here, but the whole DFA~\(\mathcal{D}\).

Let us consider another example. We want to define formally a DFA
which recognises the language \(L\) whose words contain an even number
of 0's and an even number of 1's (the alphabet is binary). We should
understand that the role of the states here is to \emph{not} count the
exact number of 0's and 1's that have been recognised before, but,
instead, this number \emph{modulo~2}. Therefore, there are four states
because there are four cases:
\begin{enumerate*}

  \item there has been an even number of 0's and 1's (state \(q_0\));

  \item there has been an even number of 0's and an odd number of 1's
    (state \(q_1\));

  \item there has been an odd number of 0's and an even number of 1's
    (state \(q_2\));

  \item there has been an odd number of 0's and 1's (state \(q_3\)).

\end{enumerate*}
What about the initial and final states?
\begin{itemize*}

  \item State~\(q_0\) is the initial state because before considering
  any input, the number of 0's and 1's is zero and zero is even;

  \item state~\(q_0\) is the lone final state because its definition
  matches exactly the characteristic of~\(L\) and no other state
  matches.

\end{itemize*}
We almost know now how to specify the DFA for the language~\(L\). It
is
\begin{equation*}
\mathcal{D} = (\{q_0,q_1,q_2,q_3\}, \{0,1\}, \delta,q_0, \{q_0\}),
\end{equation*}
where the transition function~\(\delta\) is described by the
transition diagram in \fig~\vref{fig:dfa_even01}.
\begin{figure}
\centering
\subfloat[The DFA\label{fig:dfa_even01}]{
\includegraphics[bb=48 653 138 737]{dfa_even01}
}
\qquad
\subfloat[The table\label{fig:table_even01}]{
\includegraphics{table_even01}
}
\caption{A deterministic finite automaton and its table}
\end{figure}
Notice how each input~0 causes the state to cross the horizontal
line. Thus, after seeing an even number of 0's we are always above the
horizontal line, in state~\(q_0\) or~\(q_1\), and after seeing an odd
number of 0's we are always below this line, in state~\(q_2\)
or~\(q_3\). There is a vertically symmetric situation for transitions
on~1. We can also represent this DFA by the transition table in
\fig~\vref{fig:table_even01}. We can use that table to illustrate the
construction of~\(\hat{\delta}\) from~\(\delta\). Suppose the input is
\(110101\). Since this string has even numbers of 0's and 1's, it
belongs to~\(L\), that is, we expect \(\hat{\delta}(q_0,110101) =
q_0\), since \(q_0\)~is the sole final state. We can check this by
computing step by step \(\hat{\delta}(q_0,\verb+110101+)\), from the
shortest prefix to the longest, which is the word \verb+110101+
itself:
\begin{align*}
  \hat{\delta} (q_0, \varepsilon)
&= q_0,\\
   \hat{\delta} (q_0, \texttt{1})
&= \delta (\hat{\delta} (q_0, \varepsilon), \texttt{1})
= \delta (q_0, \texttt{1})
= q_1,\\
   \hat{\delta} (q_0, \texttt{11})
&= \delta (\hat{\delta} (q_0, \texttt{1}), \texttt{1})
= \delta (q_1, \texttt{1})
= q_0,\\
   \hat{\delta} (q_0, \texttt{110})
&= \delta (\hat{\delta} (q_0, \texttt{11}), \texttt{0})
= \delta (q_0, \texttt{0})
= q_2,\\
   \hat{\delta} (q_0, \texttt{1101})
&= \delta (\hat{\delta} (q_0, \texttt{110}), \texttt{1})
= \delta (q_2, \texttt{1})
= q_3,\\
   \hat{\delta} (q_0, \texttt{11010})
&= \delta (\hat{\delta} (q_0, \texttt{1101}), \texttt{0})
= \delta (q_3, \texttt{0})
= q_1,\\
   \hat{\delta} (q_0, \texttt{110101})
&= \delta (\hat{\delta} (q_0, \texttt{11010}), \texttt{1})
= \delta (q_1, \texttt{1})
= q_0 \in F.
\end{align*}


\subsection{Hyperbolic Paraboloids}
\noindent
Hyperbolic paraboloids have the form
\begin{equation*}
	z = x^2 - y^2.
\end{equation*}
They are not radially symmetric and look like a saddle or Pringle's chip.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.33\textwidth]{./Images/differentialMultivariableCalculus/hyperbolic_paraboloid.png}
	\caption{A hyperbolic paraboloid}
\end{figure}


\chapter{Cardinalities of Sets}
\label{chapter:cardinality}
\Cref{part:combinatorics} discusses sizes of sets; however, this notion was
never defined formally. This part is going to close this gap and study
properties of sizes of sets.

\section{Definition}
One may notice that if we have a bijection $f$ from $\range{n}$ to a set $S$ we
enumerate all the elements of $S$: $f(1)$, \dots, $f(n)$.
This observation allows us to define the cardinality of a set.
\begin{definition}
  Let $S$ be a set, we say that cardinality of $S$ is equal to $n$ (we write
  that $\cardinality{S} = n$) iff there is a bijection from $\range{n}$ to $S$.

  We also say that a set $T$ is finite if there is an integer $n$ such that
  $\cardinality{T} = n$.
\end{definition}

Note that this definition does not guarantee that cardinality is unique so
we need the following theorem.
\begin{theorem}
\label{theorem:correctness-of-cardinality}
    For any set $S$, if there are bijections $f : \range{n} \to S$ and
    $g : \range{m} \to S$, then $n = m$.
\end{theorem}
\begin{proof}[Proof of Theorem~\ref{theorem:correctness-of-cardinality}]
    Let us consider the inverse $g^{-1}$ of $g$ (it exists by
    Theorem~\ref{theorem:inverse-of-bijections} since $g$ is a
    bijection). Note that $h = g^{-1} \circ f$
    is a bijection from $\range{n}$ to $\range{m}$.

    We prove using induction by $n$ that for any $n, m \in \N$,
    if there is a bijection $h'$ from $\range{n}$ to $\range{m}$, then $n = m$.
    The base case is for $n = 1$; if $m \ge 2$,
    then there are $x, y \in [1]$ such that $h'(x) = 1$ and $h'(y) = 2$, but
    $x \neq y$ and we have only one element in $[1]$.

    The induction step is also simple. Assume that there is a bijection $h'$ from
    $[n + 1]$ to $\range{m}$. We define a function
    $h'' : \range{n} \to \range{m - 1}$ as follows:
    \[
        h''(i) =
        \begin{cases}
            h'(i) & \text{if } h'(i) < h'(n + 1) \\
            h'(i) - 1 & \text{otherwise}
        \end{cases}.
    \]
    We prove that $h''$ is a bijection.
    \begin{itemize}
        \item Let $i_1 \neq i_2 \in \range{n}$. If
            $h'(i_1), h'(i_2) < h'(n + 1)$ or $h'(i_1), h'(i_2) \ge h'(n + 1)$,
            then $h''(i_1) \neq h''(i_2)$ since $h'(i_1) \neq h'(i_2)$.
            Otherwise, without loss of generality we may assume that
            $h'(i_1) < h'(n + 1) < h'(i_2)$ but it implies that
            $h''(i_1) = h'(i_1) < h'(n + 1) \le h'(i_2) - 1 = h''(i_2)$.
        \item Let $j \in [m - 1]$. We need to consider two cases.
            \begin{enumerate}
                \item Let $j < h(n + 1)$. There is $i \in [n + 1]$ such that
                    $h'(i) = j$ since $h'$ is a bijection
                    (note that $i \neq n + 1$). Thus $h''(i) = j$.
                \item Otherwise, there is $i \in [n + 1]$ such that
                    $h'(i) = j + 1$ since $h'$ is a bijection
                    (note that $i \neq n + 1$). Thus $h''(i) = j$.
            \end{enumerate}
    \end{itemize}
    Since $h''$ is a bijection, the induction hypothesis implies that
    $n = m - 1$. As a result, $n + 1 = m$.
\end{proof}

Also, using this definition we may finally prove
\Cref{theorem:bijection-to-equality}.
\begin{proof}[Proof of \Cref{theorem:bijection-to-equality}]
    Let $\cardinality{X} = n$, and $g : \range{n} \to X$ be a bijection.
    Note that $f \circ g : \range{n} \to Y$ is a bijection, hence
    $\cardinality{Y} = n$.
\end{proof}

\section{Proofs of Combinatorial Principles}
We start from proving the additive principle. Recall that it says that if two
finite sets $X = \set{x_1, \dots, x_n}$ and $Y = \set{y_1, \dots, y_m}$ are
disjoint, then their union contains $n + m$ elements. Informally this statement
is very easy since we can simply enumerate elements of the set $X$ first and
after that enumerate elements of $Y$:
\[
  \begin{matrix}
    x_1 & x_2 & \dots & x_n & y_1    & y_2   & \dots & y_m \\
    1   & 2   & \dots & n   & n + 1  & n + 2 & \dots & n + m.
  \end{matrix}
\]
Using the definition of cardinality, this informal proof can be formalized as
follows.
\begin{proof}[Proof of \Cref{theorem:additive-principle}]
  Let $\cardinality{X} = n$, $\cardinality{Y} = m$ and $g : \range{n} \to X$ and $h : \range{m} \to Y$ be
  bijections. In order to prove it we just construct a bijection
  $f : [n + m] \to (X \cup Y)$.

  \[
    f(i) =
    \begin{cases}
        g(i) & i < n \\
        h(i  - n) & i > n
    \end{cases}.
  \]

  It's easy to see that $f$ is an injection. Let us start by assuming the opposite i.e.
  that $i_0 \neq i_1 \in X \cup Y$ such that $f(i_0) = f(i_1)$.
  There are three cases.
  \begin{itemize}
    \item The first is when $i_0, i_1 \in \range{n}$. In this case $g(i_0) = g(i_1)$
      which contradicts the assumption that $g$ is a bijection.
    \item The second is when $i_0, i_1 \in \set{n + 1, n + 2, \dots, m}$.
      In this case $h(i_0 - n) = h(i_1 - n)$ which contradicts the assumption
      that $h$ is a bijection.
    \item Finally, the last case is when $i_0 \in \range{n}$ and
      $i_1 \in \set{n + 1, n + 2, \dots, m}$. It is easy to see that this
      implies that $g(i_0) = h(i_1 - n)$. However, it
      means that $g(i_0) = h(i_1 - n) \in (X \cap Y)$, which contradicts the
      assumption that $X \cap Y = \emptyset$.
  \end{itemize}

  To finish the proof we need to show that $f$ is a surjection. Let
  $w \in (X \cup Y)$. Consider the following two cases.
  \begin{itemize}
    \item Let $w \in X$. There is $i \in \range{n}$ such that $f(i) = g(i) = w$ since
      $g$ is a bijection.
    \item Otherwise, $w \in Y$. In this case, there is $i \in \range{m}$ such that
      $f(i + n) = h(i) = w$ since $h$ is a bijection.
  \end{itemize}
\end{proof}

Using the additive principle it is pretty easy to prove the multiplicative
principle. Recall that the multiplicative principle says that the Cartesian
product of two finite sets $X = \set{x_1, \dots, x_n}$ and
$Y = \set{y_1, \dots, y_m}$ has size $nm$. It is easy to justify this statement
informally; indeed, note that $X \times Y$ has the following elements:
\[
  \begin{matrix}
    (x_1, y_1) & (x_1, y_2) & \dots & (x_1, y_m) \\
    (x_2, y_1) & (x_2, y_2) & \dots & (x_2, y_m) \\
    \dots \\
    (x_n, y_1) & (x_n, y_2) & \dots & (x_n, y_m);
  \end{matrix}
\]
each line has $m$ elements and there are $n$ lines, so there are $nm$ elements
altogether that can be enumerated as follows:
\[
  \begin{matrix}
    (x_1, y_1) & (x_1, y_2) & \dots & (x_1, y_m) \\
    1          & 2          & \dots & m \\
    (x_2, y_1) & (x_2, y_2) & \dots & (x_2, y_m) \\
    m + 1      & m + 2      & \dots & m + m \\
    \dots \\
    (x_n, y_1) & (x_n, y_2) & \dots & (x_n, y_m) \\
    (n - 1) m + 1 & (n - 1) m + 2 & \dots & (n - 1) m + m
  \end{matrix}
\]
The definition of cardinality gives us a simple formalization of this argument.
\begin{proof}[Proof of \Cref{theorem:multiplicative-principle}]
  If one of the sets $X$ and $Y$ is empty, then $X \times Y$ is empty as well
  and the statement is as follows.

  Assume that none of the sets are empty. Let $\cardinality{X} = n$,
  $\cardinality{Y} = m$, and
  $f : \range{n} \to X$ and $g : \range{m} \to Y$ be bijections.
  Note that
  \[
    \bigcup_{i = 1}^n (\set{f(i)} \times Y) = X \times Y.
  \]
  Additionally, note that $(\set{f(i)} \times Y) \cap (\set{f(j)} \times Y) =
  \emptyset$ for $i \neq j$. Finally, it is easy to see that $g_i : \range{m} \to
  (\set{f(i)} \times Y)$ such that $g_i(j) = (f(i), g(j))$ is a bijection.
  Hence,
  $\cardinality{X \times Y} = \sum_{i = 1}^n \cardinality{\set{f(i)} \times Y} = n \cdot m$.
\end{proof}

\section{Generalized Commutative Operations}
\label{section:generalized-sum}
Using the notion of cardinality we can formalize the summation operation over
sets:
\[
    \sum_{i \in S ~:~ P(i)} f(i) = \sum_{j = 1}^k f(i_j),
\]
where $\set[P(i)]{i \in S} = \set{i_1, \dots, i_k}$. More formally,
\[
    \sum_{i \in S ~:~ P(i)} f(i) = \sum_{j = 1}^k f(g(j)),
\]
where $k = \cardinality{\set[P(i)]{i \in S}}$ and $g : \set[P(i)]{i \in S} \to \range{k}$ is
a bijection.

\begin{theorem}
\label{theorem:sum-correctness}
    The definition of $\sum_{i \in S : P(i)} f(i)$ does not depend on the choice
    of $g$;
    i.e. $\sum_{i = 1}^k f(g_1(i)) = \sum_{i = 1}^k f(g_2(i))$
    for any two bijections $g_1, g_2 : \set[P(i)]{i \in S} \to \range{k}$.
\end{theorem}
Before we prove this statement we need to give a couple of definitions.
We say that a function $h : \range{n} \to \range{n}$ is a \emph{permutation} of
$\range{n}$ iff $h$ is a bijection. We also say that
$i, j \in \range{k}$ form the inversion in $h$ iff $h(i) > h(j)$ and $i < j$.
We denote by $I(h)$ the number of inversions in $h$; i.e. $I(h) =
\cardinality{\set[i, j \text{ form an inversion in } h]{(i, j)}}$.
\nomenclature[C]{$I(h)$}{denotes the number of inversions in $h$}

Important examples of permutations are transposition: for any
$i, j \in \range{n}$, $\tau_{i, j} : \range{n} \to \range{n}$ such that
\[
    \tau_{i, j}(x) =
    \begin{cases}
        j & \text{if } x = i \\
        i & \text{if } x = j \\
        x & \text{otherwise}
    \end{cases}.
\]
is called a transposition of $i$ and $j$.
\nomenclature[F]{$\tau_{i, j}$}{denotes the transposition of $i$ and $j$}

It is easy to see that $I(h) = 0$ iff $h(i) = i$ for any $i \in \range{k}$.
It is also clear that if $i, j$ form an inversion in $h$, then $I(h) > I(h')$,
where $h' = h \circ \tau_{i, j}$, i.e.
\[
    h'(x) =
    \begin{cases}
        h(j) & \text{if } x = i \\
        h(i) & \text{if } x = j \\
        h(x) & \text{otherwise}
    \end{cases}.
\]

\begin{proof}[Proof of Theorem~\ref{theorem:sum-correctness}]
    Proof of this theorem consists of two parts.
    First, we prove that
    \begin{equation}
        \label{equation:summation-with-respect-to_bijection}
        \sum_{i = 1}^k f(g(i)) = \sum_{i = 1}^k f(g(h(i)))
    \end{equation}
    for any bijections $g : \set[P(i)]{i \in S} \to \range{k}$ and
    $h : \range{k} \to \range{k}$.

    We prove Equation~\ref{equation:summation-with-respect-to_bijection} using the
    induction by $I(h)$.
    \begin{description}
        \item[(the base case)] If $I(h) = 0$, then $h$ is the identity function and
            $g(i) = g(h(i))$. Hence,
            Equation~\ref{equation:summation-with-respect-to_bijection} is true.
        \item[(the induction step)] By the induction hypothesis, for any permutation
            $h' : \range{k} \to \range{k}$,
            if $I(h') < \ell$, then
            \[
                \sum_{i = 1}^k f(g(i)) = \sum_{i = 1}^k f(g(h'(i))).
            \]
            Let us consider a permutation $h : \range{k} \to \range{k}$ such that
            $I(h) = \ell$. Let $i$ and $j$ form an inversion in $h$ (such $i$ and $j$
            exist since $I(h) \neq 0$). Let $h' = h \circ \tau_{i, j}$.
            Note that by the induction hypothesis,
            \[
                \sum_{i = 1}^k f(g(i)) = \sum_{i = 1}^k f(g(h'(i)))
            \]
            since $I(h') < I(h) = \ell$ and it is clear that
            \[
                \sum_{i = 1}^k f(g(h'(i))) = \sum_{i = 1}^k f(g(h(i))).
            \]
            As a result,
            Equation~\ref{equation:summation-with-respect-to_bijection} is true.
    \end{description}

    Now we are ready to finish proof of the theorem.
    Consider $g_1, g_2 : \set[P(i)]{i \in S} \to \range{k}$ and define
    $h = g_1^{-1} \circ g_2$. Note that $h : \range{k} \to \range{k}$ is a
    permutation and $g_1(h(i)) = g_2(i)$. Thus we proved that
    \[
        \sum_{i = 1}^k f(g_1(i)) = \sum_{i = 1}^k f(g(h(i))) =
        \sum_{i = 1}^k f(g_2(i)).
    \]
\end{proof}

Similarly one may define a generalized union and intersection of sets.
Let $\Omega$ and $S$ be some sets, $X : S \to 2^\Omega$ and $P(i)$ be a
predicate. Then
\begin{gather*}
    \bigcup_{i \in S ~:~ P(i)} X(i) = \bigcup_{i = 1}^k X(g(i)) \\
    \text{and}\\
    \bigcap_{i \in S ~:~ P(i)} X(i) = \bigcap_{i = 1}^k X(g(i)),
\end{gather*}
where $k = \cardinality{\set[P(i)]{i \in S}}$ and $g : \set[P(i)]{i \in S} \to \range{k}$ is
a bijection.
\nomenclature[S]{$\bigcup_{i \in S ~:~ P(i)} A_i$}{denotes $A_{i_1} \cup \dots
\cup A_{i_k}$, where $\set[P(i)]{i \in S} = \set{i_1, \dots, i_k}$}
\nomenclature[S]{$\bigcap_{i \in S ~:~ P(i)} A_i$}{denotes $A_{i_1} \cap \dots
\cap A_{i_k}$, where $\set[P(i)]{i \in S} = \set{i_1, \dots, i_k}$}

\begin{exercise}
    Show that the definitions of $\bigcup_{i \in S : P(i)} X(i)$ and
    $\bigcap_{i \in S : P(i)} X(i)$ are correct,
    i.e. that they do not depend on the choice of $g$.
\end{exercise}

\begin{chapterendexercises}
    \exercise Prove Theorem~\ref{theorem:injections-surjections-inequalities}.
\end{chapterendexercises}


\section{Exponents and roots}
\begin{center}
    \begin{longtable}{lcr}
        $a^n \cdot a^m = a^{(n + m)}$
        &
        $\frac{a^n}{a^m} = a^{(n - m)}$
        &
        $a^{-n} = \frac{1}{a^n}$
        \\
        $\left(a^n\right)^m = a^{(n \cdot m)}$
        &
        $(a \cdot b)^n = a^n \cdot b^n$
        &
        $\left(\frac{a}{b}\right)^n = \frac{a^n}{b^n}$
        \\
        $\sqrt[q]{a} = a^\frac{1}{q}$
        &
        $\sqrt[q]{a^p} = a^\frac{p}{q}$
        &
        $\sqrt[q]{a \cdot b} = \sqrt[q]{a} \cdot \sqrt[q]{b}$
        \\
        $\sqrt[q]{\frac{a}{b}} = \frac{\sqrt[q]{a}}{\sqrt[q]{b}}$
        &
        $a^0 = 1$
        &
        $a^1 = a$
        \\
        $(a \pm b)^2 = a^2 + b^2 \pm 2ab$
        &
        $(-1)^n = \begin{cases}
            1 & \text{if } n \text{ is even}
            \\
            -1 & \text{if } n \text{ is odd}
        \end{cases} $
        &
        $\frac{1}{\sqrt{a}} = \frac{\sqrt{a}}{a}$
        \\
        $(a + b)(a - b) = a^2 - b^2$
        &
        $a \cdot \sqrt{a} = a^\frac{3}{2}$
    \end{longtable}
\end{center}


\section{Estimators}

A \textbf{statistic} is any measurable function calculated with the data ($\bar{X_n}, max(X_i),$ etc).\\

An \textbf{estimator} $\hat{\theta }_ n$ of $\theta$ is any statistic which does not depend on $\theta$.\\

Estimators are random variables if they depend on the data (= realizations of random variables).\\

An estimator $\hat{\theta }_ n$ is \textbf{weakly consistent} if: $\displaystyle \lim _{n \to \infty } \hat{\theta }_ n = \theta$ or $ \hat{\theta}_n \xrightarrow[n \rightarrow \infty]{P} \mathbb{E}[g(X)]$. If the convergence is almost surely it is \textbf{strongly consistent}.\\
\textbf{Asymptotic normality of an estimator:}
\[\sqrt(n) (\hat{\theta}_n-\theta) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\sigma^2)\]
$\sigma^2$ is called the \textbf{Asymptotic Variance} of the estimator $\hat{\theta}_n$. In the case of the sample mean it is the same variance as as the single $X_i$.\\
If the estimator is a function of the sample mean the \textbf{Delta Method} is needed to compute the asymptotic variance.\textbf{Asymptotic Variance} $\neq$ Variance of an estimator.\\
\textbf{Bias of an estimator:}
\[Bias(\hat{\theta}_n) = \mathbb{E}[\hat{\theta_n}] - \theta\]
\textbf{Quadratic risk of an estimator}
\begin{align*}
R(\hat{\theta}_n) & = \mathbb{E}[(\hat{\theta}_n-\theta)^2]\\
& = Bias^2 + Variance
\end{align*}


\subsubsection{Underdamped ($\Delta < 0$)}
This is probably the most complicated case. Here, both roots are complex. Specifically,
\begin{equation*}
	r = \frac{-b}{2m} \pm i\frac{\sqrt{\abs{\Delta}}}{2m}
\end{equation*}
Letting the coefficient of the imaginary part be $\omega$,
\begin{equation*}
	r = \frac{-b}{2m} \pm i\omega
\end{equation*}
So, our solution becomes
\begin{equation*}
	y = e^{\frac{-b}{2m} t}\left(C_1\cos{(\omega t)} + C_2\sin{(\omega t)}\right)
\end{equation*}
Rewriting in terms of $\cos$ and a phase shift,
\begin{equation*}
	y = Ae^{\frac{-b}{2m} t}\cos{(\omega t - \phi)} \text{ where }
	A = \sqrt{A^2 + B^2} \text{, } \phi = \begin{cases}
		\arctan{\left(\frac{B}{A}\right)} + \pi & A \leq 0 \\
		\arctan{\left(\frac{B}{A}\right)} & A > 0
	\end{cases}
\end{equation*}
\begin{center}
	\includegraphics[width=0.5\textwidth]{./higherOrder/freeVibrs/underdamped.png}
\end{center}
Here, the exponential term dominates the limit, so
\begin{equation*}
	\lim\limits_{t \to 0}{Ae^{\frac{-b}{2m}t}\cos{(\omega t - \phi)}} = 0
\end{equation*}
meaning the mass's oscillation decays over time, bounded by the exponential curves.


%% 8
\section{ Y Combinator. }

% https://mvanier.livejournal.com/2897.html

% See also http://r6.ca/blog/20060919T084800Z.html

$$
Y = λf.(λx.f(xx))(λx.f(xx))
$$

\begin{align*}
Y\ g &= (λx.g(xx))(λx.g(xx)) \\
     &= g ((λx.g(xx)) (λx.g(xx))) \\
     &= g (Y\ g) \\
     &= g (g (g (g (g \ldots))))
\end{align*}
and it never ends if we use it like that. Still, if we take (using Haskell notation)
\begin{minted}{haskell}
g h x | x == 1    = 1
      | otherwise = x * (h $ x - 1)
\end{minted}
then for example
\begin{align*}
Y\ g\ 4 &= g\ (Y\ g)\ 4                \\
        &= 4 * ((Y\ g)\ 3)             \\
        &= 4 * (g\ (Y\ g)\ 3)          \\
        &= 4 * 3 * ((Y\ g)\ 2)         \\
        &= 4 * 3 * (g\ (Y\ g)\ 2)      \\
        &= 4 * 3 * 2 * ((Y\ g)\ 1)     \\
        &= 4 * 3 * 2 * (g\ (Y\ g)\ 1)  \\
        &= 4 * 3 * 2 * 1               \\
        &= 24,\text{ also known as } 4!
\end{align*}

Here, we have a definition of a recursive function (factorial) in a language that does
not directly allow recursion. Of course, that requires lazy evaluation ---
and the definitions for naturals and multiplication.


\subsection{Calculus of sine and cosine}

\subsubsection{Unity}

Note that with imaginary numbers we can reverse all \(i\)s. So:

\(e^{i\theta }=\cos (\theta )+i\sin (\theta )\)

\(e^{-i\theta }=\cos (\theta )-i\sin (\theta )\)

\(e^{i\theta }e^{-i\theta }=(\cos (\theta )+i\sin (\theta ))(\cos (\theta )-i\sin (\theta ))\)

\(e^{i\theta }e^{-i\theta }=\cos (\theta )^2+\sin (\theta )^2\)

\(e^{i\theta }e^{-i\theta }=e^{i\theta -i\theta }=e^0=1\)

So:

\(\cos (\theta )^2+\sin (\theta )^2=1\)

Note that if \(\cos (\theta )^2=0\), then \(\sin (\theta )^2=\pm 1\)

That is, if the real part of \(e^{i\theta }\) is \(0\), the imaginary part is \(\pm 1\). And visa versa.

Similarly if the derivative of the real part of \(e^{i\theta }\) is \(0\), the imaginary part is \(\pm 1\). And visa versa.

\subsubsection{Sine and cosine are linked by their derivatives}

Note that these functions are linked in their derivatives.

\(\dfrac{\delta }{\delta \theta }\cos (\theta )=\sum_{j=0}^\infty \dfrac{(\theta )^{(4j+3)}}{(4j+3)!}-\sum_{j=0}^\infty \dfrac{(\theta )^{4j+1}}{(4j+1)!}\)

\(\dfrac{\delta }{\delta \theta }\cos (\theta )=-\sin (\theta )\)

Similarly:

\(\dfrac{\delta }{\delta \theta }\sin (\theta )=cos(\theta )\)

\subsubsection{Both sine and cosine oscillate}

\(\dfrac{\delta^2 }{\delta \theta^2}\sin (\theta )=-\sin (\theta )\)

\(\dfrac{\delta^2 }{\delta \theta^2}\cos (\theta )=-\cos (\theta )\)

So for either of:

\(y=\cos (\theta )\)

\(y=\sin (\theta )\)

We know that

\(\dfrac{\delta^2 }{\delta \theta^2}y(\theta )=-y(\theta )\)

Consider \(\theta =0\).

\(e^{i.0}=\cos (0)+i\sin (0)\)

\(1=\cos (0)+i\sin (0)\)

\(\sin (0)=0\)

\(\cos (0)=1\)

Similarly we know that the derivative:

\(\sin'(0)=\cos(0)=1\)

\(\cos'(0)=-\sin(0)=0\)

Consider \(\cos(\theta )\).

As \(\cos (0)\) is static at \(\theta =0\), and is positive, it will fall until \(\cos (\theta )=0\).

While this is happening, \(\sin (\theta )\) is increasing. As:

\(\cos (\theta )^2+\sin (\theta )^2=1\)

\(\sin (\theta )\) will equal \(1\) where \(\cos (\theta )=0\).

Due to symmetry this will repeat \(4\) times.

Let's call the length of this period \(\tau \).

Where \(\theta =\tau *0\)

\begin{itemize}
\item \(\cos (\theta )=1\)
\item \(\sin (\theta )=0\)
\end{itemize}

Where \(\theta =\tau *\dfrac{1}{4}\)

\begin{itemize}
\item \(\cos (\theta )=0\)
\item \(\sin (\theta )=1\)
\end{itemize}

Where \(\theta =\tau *\dfrac{2}{4}\)

\begin{itemize}
\item \(\cos (\theta )=-1\)
\item \(\sin (\theta )=0\)
\end{itemize}

Where \(\theta =\tau *\dfrac{3}{4}\)

\begin{itemize}
\item \(\cos (\theta )=0\)
\item \(\sin (\theta )=-1\)
\end{itemize}

\subsubsection{Relationship between\( \cos (\theta )\) and \(\sin(\theta )\)}

Note that \(\sin(\theta + \dfrac{\tau }{4})=\cos(\theta )\)

Note that \(\sin (\theta )=\cos (\theta )\) at

\begin{itemize}
\item \(\tau *\dfrac{1}{8}\)
\item \(\tau *\dfrac{5}{8}\)
\end{itemize}

And that all these answers loop. That is, add any integer multiple of \(\tau \) to \(\theta \) and the results hold.

\(e^{i\theta } = e^{i\theta +n\tau }\)

\(n \in \mathbb{N}\)

\(e^{i\theta } = \cos(\theta )+i\sin(\theta )\)

\(e^{i\theta } = \cos(\theta +n\tau )+i\sin(\theta +n\tau ) \)

\(e^{i\theta } = e^{i(\theta +n\tau )}\)

\subsubsection{Calculus of trig}

Relationship between cos and sine

\(\sin(x+\dfrac{\pi }{2})=\cos(x)\)

\(\cos(x+\dfrac{\pi }{2})=-\sin(x)\)

\(\sin(x+\pi )=-\sin(x)\)

\(\cos(x+\pi )=-\cos(x)\)

\(\sin(x+\tau )=\sin(x)\)

\(\cos(x+\tau )=\cos(x)\)


\sec{Some integral functions}
Here we study some \emph{integral}\footnote{pun intended} functions that will help us later.\\
The reader may skip this section for now and return to this when referenced later.\\
We skip the formal discussion about convergence of improper Riemann integrals. (These are different from the integrals done in MA 105.) A discussion can be found in Professor GKS' notes mentioned.
\subsection{The Gamma Function}
\begin{defn}[The Gamma Function]
	The Gamma function is defined for $a > 0$ as follows:
	\begin{equation*}
		\Gamma(a) := \int_{0}^{\infty} t^{a-1}e^{-t} dt.
	\end{equation*}
\end{defn}
\begin{mdframed}[style=boxstyle, frametitle={Some results}]
	\begin{enumerate}[leftmargin=*]
		\item $\Gamma(n) = (n - 1)!$ for all $n \in \mathbb{Z}^+ = \{1, 2, \ldots\}.$
		\item $\Gamma(x+1) = x\Gamma(x)$ for all $x > 0.$
		\item $\Gamma\left(\dfrac{1}{2}\right) = \sqrt{\pi}.$
		\item The last two relations now let us calculate $\Gamma$ for all half-integers.
		\item And that's as far as we can get. Other values of Gamma are not known in such elementary terms.
		\item Note that we can still calculate ratios such as
		\begin{equation*}
			\Gamma\left(\dfrac{5}{4}\right)\bigg/\Gamma\left(\dfrac{1}{4}\right).
		\end{equation*}
	\end{enumerate}
\end{mdframed}

\subsection{The Beta Function}
\begin{defn}[The Beta Function]
	This is a function of two variables defined for $a > 0, b > 0$ as
	\begin{equation*}
		B(a, b) := \int_{0}^{1} x^{a-1}(1 - x)^{b-1} dx
	\end{equation*}
\end{defn}
\begin{mdframed}[style=boxstyle, frametitle={Some identities}]
	\begin{enumerate}[leftmargin=*]
		\item $B(a, b) = B(b, a).$
		\item $B(a + 1, b) = \dfrac{a}{b}B(a, b+1).$
		\item $B(a + 1, b) + B(a, b+1) = B(a, b).$
		\item $B\left(\dfrac{1}{2}, \dfrac{1}{2}\right) = \pi.$
	\end{enumerate}
\end{mdframed}
\begin{thm}[The Beta-Gamma Relation]
	\begin{equation*}
		\Gamma(a)\Gamma(b) = \Gamma(a + b)B(a, b) \quad \forall\; a, b > 0.
	\end{equation*}
\end{thm}
\begin{thm}[Euler's Reflection Formula]
	\begin{equation*}
		\Gamma(a)\Gamma(1 - a) = \dfrac{\pi}{\sin(\pi a)} \quad \forall\;0 < a < 1
	\end{equation*}
\end{thm}

\begin{mdframed}[style=boxstyle2, frametitle={An integral computation}]
	\begin{thm} \label{thm:randomintegral}
		\begin{equation*}
			\int_{0}^{\infty} \dfrac{x^{a-1}}{1 + x} dx = \dfrac{\pi}{\sin \pi a} \quad \text{ for } 0 < a < 1
		\end{equation*}
	\end{thm}
	\begin{proof}
		Put $t = \dfrac{1}{1 + x}.$ The integral transforms as
		\begin{align*}
			& \int_{0}^{1} (1 - t)^{a - 1}t^{-a} dt\\
			& = B(a, 1 - a)\\
			& = \Gamma(a)\Gamma(1 - a)\\
			& = \dfrac{\pi}{\sin \pi a}.
		\end{align*}
	\end{proof}
\end{mdframed}


\begin{solution}
In this problem we are going to see how inadequate it is to use equispaced nodes instead of, for instance, Chebishev nodes. In the interval $[-1,1]$ we are going to use polynomial approximation with equispaced nodes to apporximate the function
\begin{align*}
f(x)=\frac{1}{1+25x^2},
\end{align*}
and compute the $L_{\infty}$ error. In the following figure we ca see how the error grows with the number of nodes used. This clearly telling us that the nodes are not well distributed since using more nodes should give us more precision (at least for not too exotic problems). To understand this, as we saw in class, let's look at the function and the polynomial of degree 100 (highest degree used) together in the figure. It is obvious that the polynomial does not approximate the function at all when we get close to the boundries. This is causing the increasing error with $N$.

\begin{figure}[H]
\centering     %%% not \center
\hspace*{\fill}
\subfigure[Error dependency on $N$.]{\includegraphics[scale=0.5]{IMAGES/problem5_1.eps}}
\hfill
\subfigure[Function and polynomial comparison.]{\includegraphics[scale=0.5]{IMAGES/problem5_2.eps}}
\hspace*{\fill}
\caption{Polynomial interpolation using equispaced nodes for the function $f(x)=\frac{1}{1+25x^2}$.}
\end{figure}
\subsection*{Matlab code for this problem}
\begin{verbatim}
ff = @(x) 1./(1+25*x.^2);
N = 1:100;
err = 0*N;
xx = linspace(-1,1,1000)';
for k = 1:length(N)
    x = linspace(-1,1,N(k))';
    W = baryWeights(x);
    y = ff(x);
    yy = bary(xx,y,x,W);
    err(k) = norm(ff(xx)-yy,inf);
end
figure
semilogy(N,err,'*',N,N.^-1,'--',N,N.^-3,'--')
set(gca,'fontsize',14)
grid on
xlabel('$N$','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
saveas(gcf,'IMAGES/problem5_1','epsc')

figure
plot(xx,ff(xx))
hold on
plot(xx,yy,'r--')
xlabel('$x$','fontsize',20,'interpreter','latex')
ylabel('$f(x),p(x)$','fontsize',20,'interpreter','latex')
legend({'$f(x)$','$p(x)$'},'Interpreter','latex','Location','north','fontsize',16)
axis([-1 1 -2 2])
grid on
saveas(gcf,'IMAGES/problem5_2','epsc')
\end{verbatim}
\end{solution}


\section{Unilinear Grammar}

A grammar is unilinear iff all rules are either all left-linear or right-linear. It can also be required that it has:
\begin{itemize}
    \item Strictly unilinear rules: $A \rarr aB$ with $a\in \Sigma \cup \epsilon$ and $B \in V \cup \epsilon$
    \item All terminal rules are empty: $B \rarr b$ replaced with $B \rarr bB'$, $B' \rarr \epsilon$
\end{itemize}

\subsection{From Unilinear to Regular Expression}

\textbf{Assumptions}: strictly right unilinear, all terminal rules empty.

For every nonterminal $A$ defined by $A \rarr a_1A_1|\ldots|a_kA_k|\epsilon$:
$L_A = a_1L_{A_1} \cup | \ldots | \cup a_kL_{A_k} \cup \epsilon$

\textbf{Arden Identity}: $X = KX \cup L$ (with $K$ nonempty) has exactly one solution $X = K^*L$.

\textbf{Example} $S \rarr sS | eA$, $A \rarr sS|\epsilon$
\begin{alignat*}{2}
&
    \begin{cases}
        L_S = sL_S \cup eL_A \\
        L_A = sL_S \cup \epsilon
    \end{cases}
&&
    \begin{cases}
        L_S = sL_S \cup e(sL_S \cup \epsilon) \\
        L_A = sL_S \cup \epsilon
    \end{cases}
\\
&
    \begin{cases}
        L_S = (s \cup es) L_S \cup e \\
        L_A = sL_S \cup \epsilon
    \end{cases}
&&
    \begin{cases}
        L_S = (s \cup es)^*e \\
        L_A = sL_S \cup \epsilon = s(s \cup es)^*e \cup \epsilon
    \end{cases}
\end{alignat*}


\section{Interpolation and the Lagrange Polynomial}

\begin{theo}[Weierstrass Approximation Theorem]
$f\in C[a,b]$, $\forall$ polynomial $P(x)$, s.t. $\left|f(x)-P(x)\right|<\epsilon$ for all $x\in [a,b]$.
\end{theo}

\begin{theo}[$n$th Lagrange Interpolating Polynomial]
$x_0,\ldots,x_n$ are $n+1$ distinct numbers, then a unique polynomial $P(x)$ of degree at most $n$ exists with
\begin{align*}
  f(x_k)=P(x_k)\quad for\space k=0:n \\
  \begin{cases}
    P(x)=\sum_{k=0}^{n}f(x_k)L_{n,k}(x) \\
    L_{n,k}=\prod_{\substack{i=0\\i\neq k}}^{k}\frac{(x-x_i)}{x_k-x_i} \\
    L_{n,k}(x_j)=\delta_{k,j}
  \end{cases}
\end{align*}
If $f\in C^{n+1}[a,b]$, $\exists\xi(x)\in (a,b)$, s.t.
\[
f(x) = P(x) + \frac{f^{(n+1)}\left(\xi\left(x\right)\right)}{(n+1)!}\prod_{i=0}^{n}(x-x_i)
\]
\end{theo}

\begin{proof}
let \[g(t)=f(t)-P(t)-\left[f(x)-g(x)\right]\prod_{i=0}^{n}\frac{(t-x_i)}{x-x_i},\]
$g(t)$ satisfies
\begin{align*}
  \begin{cases}
  g(x_k)=0 & k=0:n\\
  g(x)=0 & \\
  g\in C^{n+1}[a,b] \\
  \end{cases}
\end{align*}
By \emph{Generalized Rolle's Theorem}, $\exists\xi\in (a,b)$, s.t. $g^{(n+1)}(\xi)=0$, then we have
\begin{align*}
& 0 = g^{(n+1)}(\xi)=f^{(n+1)}(\xi)-P^{(n+1)}(\xi)-\left[f(x)-P(x)\right]\frac{\D^{n+1}}{\D t^{n+1}}\left[\prod_{i=0}^{n}\frac{(t-x_i)}{x-x_i}\right]_{t=\xi} \\
\Rightarrow & 0 = f^{(n+1)}(\xi)-\left[f(x)-P(x)\right]\frac{(n+1)!}{\prod_{i=0}^{n}(x-x_i)} \\
\Rightarrow & f(x) = P(x) + \frac{f^{(n+1)}\left(\xi\left(x\right)\right)}{(n+1)!}\prod_{i=0}^{n}(x-x_i).
\end{align*}
\end{proof}


\documentclass[11pt,letterpaper]{article}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}

\usepackage[charter]{mathdesign}
\usepackage{fullpage}
\pagestyle{empty}

\input{../fncextra}

\begin{document}

\begin{center}
  \bf
  Project: Deblur, denoise, delight!
\end{center}

Recall that an image may be represented as an $m\times n$ matrix $\mX$ of pixel intensities. A fair way to simulate blurring the image is by
\begin{equation}
\label{blurmat}
\mZ =  (\mB_m)^p \mX \, (\mB_n^T)^p,
\end{equation}
where $p$ is a positive integer and each $\m{B}_k$ is a $k\times k$ symmetric tridiagonal matrix with $1/2$ all along the main diagonal, and $1/4$ all along both the sub- and superdiagonals. As can easily be verified, the mapping from $\mX$ to $\mZ$ is linear, so if we represent the images as vectors (say, by stacking columns), then $\bfz=\mG \bfx$. The matrix $\m{G}$ must be $mn\times mn$, and can be found explicitly, but we consider it to be unavailable.

Suppose that the image $\mZ$ is derived from both blurring of $\mX$ and some added noise. We can write this as
\begin{equation}
\label{forward}
\bfz = \mG \bfx + \bfy, \quad \text{for some $\bfy$ with $\|\bfy\|^2=\delta^2$},
\end{equation}
where $\delta$ is a small(?) number. Our goal is the reconstruction problem: for known $\bfz$, $\mG$, and $\delta$, return the best (most likely) $\bfx$. An optimization is essential, since there are many possible solutions of~\eqref{forward}.

Truly random noise will look like static, with lots of value jumps from pixel to pixel. Real images, on the other hand, usually have large patches of slowly-varying values. Consider the $k\times k$ matrix
\begin{equation}
\mR_k =
\begin{bmatrix}
  -2 & 1      &        &        &    \\
  1  & -2     & 1      &        &    \\
     & \ddots & \ddots & \ddots &    \\
     &        & 1      & -2     & 1  \\
     &        &        & 1      & -2 \\
\end{bmatrix}.
\end{equation}
$\mR_k$ is a scaled finite-difference approximation of the second derivative, so $\|\mR_k\bfu\|$ is a measure of the ``roughness'' of the vector $\bfu$. The 2D equivalent of this process on an image matrix $\mX$ is to apply roughening from both the left and the right, in the form
\begin{equation}
  \label{roughmat}
   \mR_m\mX + \mX\mR_n^T.
\end{equation}
This too is a linear transformation of the image $\mX$, which we denote by $\m{L}\bfx$ for an unspecified matrix $\mL$. Minimizing roughness, as measured by $\|\mL\bfx\|$, seems like a reasonable optimization criterion.

We now have a complete problem to state:
\begin{equation}
   \label{consopt}
   \min \| \mL \bfx\|^2 \; \text{over all $\bfx$ satisfying $\|\mG\bfx-\bfz\| = \delta$.}
\end{equation}
This is a multidimensional minimization subject to a scalar constraint function, a problem nicely changed to an unconstrained form by the method of Lagrange multipliers. Introducing the unknown scalar $\lambda$, we obtain the simultaneous equations
\begin{align}
    (\mL^T\mL + \lambda \mG^T\mG)\bfx &= \lambda \mG\bfz \label{lineq}\\
    \|\mG\bfx - \bfz\| - \delta & = 0. \label{nonlineq}
\end{align}

Equation~\eqref{lineq} is an $mn\times mn$ linear system that defines $\bfx$ when $\lambda$ is given. We could pose~\eqref{nonlineq} as a rootfinding problem for $\lambda$. However, we rarely know a good value for $\delta$ in advance, so it makes more sense to regard the solution $\widehat{\bfx}$ of~\eqref{lineq} as a function of a tunable parameter $\lambda$. As $\lambda\to0$, greater emphasis is placed on minimizing roughness (increased smoothing), and as $\lambda\to\infty$, more emphasis is on deblurring (increased sharpness).

The matrices $\mL$ and $\mG$ are never needed for these computations. It's possible to show that $\mL$ and $\mG$ are both symmetric, so $\mL^T\mL\bfx=\mL(\mL\bfx)$ and $\mG^T\mG\bfx=\mG(\mG\bfx)$. For implementation, the multiplication $\mL\bfx$ or $\mG\bfx$ is achieved by reshaping $\bfx$ to the image matrix $\mX$,  applying~\eqref{roughmat} or~\eqref{blurmat}, respectively, and then reshaping back into a vector. Hence, given $\lambda$, the operator on the left side of~\eqref{lineq} can be implemented by writing a function that can be passed to \texttt{gmres} in lieu of a matrix.

A final note: Adding a constant to all the entries of $\mZ$ can improve the reconstruction. A typical choice is to make the mean value of the pixels equal to zero. This doesn't affect the plot using \texttt{imagesc}, which maps min and max pixel values to black and white.

\subsection*{Project assignment}
\label{sec:project-assignment}

Submit a zip archive with files as directed here. Don't forget to convert an imported image to a 2D matrix of class \texttt{double}. \textbf{You should use \texttt{imagesc} for plotting all images, and after each image plot use}
\begin{verbatim}
    colormap(gray(256))), axis equal
\end{verbatim}

\begin{description}
\item[\textbf{Objective 1.}] Derive~\eqref{lineq}--\eqref{nonlineq} from~\eqref{consopt}. You will need to consult a calculus book, and to take the gradients of expressions like $\mC^T \mC \bfu$ and $\bfu^T \bfc$ with respect to $\bfu$.

For this objective, submit the following:
\begin{itemize}
\item A PDF, \texttt{objective1.pdf}, typeset nicely or scanned from a neatly written document and showing the derivation in convincing detail.
\end{itemize}

\item[\textbf{Objective 2.}]  Write two functions:
\begin{verbatim}
    function Z = blur(X,p)
    function Z = roughen(X)
\end{verbatim}
They implement the equivalent of $\mG\bfx$ and $\mL\bfx$, respectively, but by operating on $m\times n$ images via~\eqref{blurmat} and~\eqref{roughmat}. In practice, both $m$ and $n$ are small enough that it's not important to use sparse matrices.
You can test your functions on the image \texttt{checkeredflag} available on the website.

For this objective, submit the following:
\begin{itemize}
\item Files \texttt{blur.m} and \texttt{rough.m}. Each should be self-contained.
\item Two PNG images. One shows the original and blurred (with $p=2$) checkered flag images side by side, and the other shows the original and roughened images side by side.
\end{itemize}

\item[\textbf{Objective 3.}] Write a function
\begin{verbatim}
    function y = linop(x,lambda,p,m,n)
\end{verbatim}
The inputs are a vectorized $mn\times 1$ image $\bfx$, a positive scalar value for $\lambda$, and the size $m\times n$ of the original image. The output is $\bfy=(\mL^T\mL + \lambda \mG^T\mG)\bfx$, computed using calls to \texttt{blur} and \texttt{roughen}.

For this objective, submit the following:
\begin{itemize}
\item The file \texttt{linop.m}. It should depend only on native MATLAB commands and \texttt{blur} and \texttt{roughen}.
\item A PNG file showing the result of applying \texttt{linop} to the checkered flag image with $\lambda=8$, $p=4$.
\end{itemize}

\item[{Objective 4.}] Download \texttt{fonzie.png} from the website and import it as the degraded image $\mZ$. For $p=3$ and $\lambda=0.05,0.2,1,5$, apply \texttt{gmres} using \texttt{linop} to reconstruct the image $\widehat{\mX}$ (the matrix form of the solution of~\eqref{lineq}).

For this objective, submit the following:
\begin{itemize}
\item Four PNG images, each showing one of your reconstructions with the value of $\lambda$ in the title.
\end{itemize}

\item[Objective 5.] Download \texttt{plate.png} from the website and import it as the degraded image $\mZ$. It's a severely degraded image of a license plate. Experiment with $p$ and $\lambda$ and do your best to identify all of the characters on the plate. You might want to use different parameter values for different characters.

For this objective, submit the following:
\begin{itemize}
\item One or more PNG images, showing your best reconstructions for the characters on the plate. The title of each image should show the values of $p$ and $\lambda$ used to generate it.
\end{itemize}

\end{description}

\end{document}