\section{Linear Support Vector Classifier} \label{section:svc}

Given $n$ training points, where each input $x_i$ has $m$ attributes, i.e., is of dimensionality $m$, and is in one of two classes $y_i=\pm1$, i.e., our training data is of the form:

\begin{equation}
	\{(x_i,y_i), x_i\in\Re^m, y_i=\pm1, i=1, \dots, n\} \label{eq:svc_data}
\end{equation}

For simplicity we first assume that data are (not fully) linearly separable in the input space $x$, meaning that we can draw a line separating the two classes when $m=2$, a plane for $m=3$ and, more in general, a hyperplane for an arbitrary $m$.

Support vectors are the examples closest to the separating hyperplane and the aim of support vector machines is to orientate this hyperplane in such a way as to be as far as possible from the closest members of both classes, i.e., we need to maximize this margin.

This hyperplane is represented by the equation $w^T x + b=0$. So, we need to find $w$ and $b$ so that our training data can be described by:

\begin{equation} \label{eq:svc_consts}
	\begin{aligned}
		& w^T x_i + b \geq +1 - \xi_i, \forall y_i=+1 \\
    	& w^T x_i + b \leq -1 + \xi_i, \forall y_i=-1 \\
    	& \xi_i \geq 0 \ \forall_i
	\end{aligned}
\end{equation}

where the positive slack variables $\xi_i$ are introduced to allow misclassified points. In this way data points on the incorrect side of the margin boundary will have a penalty that increases with the distance from it.

These two equations can be combined into:

\begin{equation} \label{eq:svc_const}
	\begin{aligned}
    	& y_i (w^T x_i + b) \geq 1 - \xi_i \ \forall_i \\
    	& \xi_i\geq 0 \ \forall_i
    \end{aligned}
\end{equation}

The margin is equal to $\displaystyle \frac{1}{\| w \|}$ and maximizing it subject to the constraint in~\eqref{eq:svc_const} while as we are trying to reduce the number of misclassifications is equivalent to finding:

\begin{equation} \label{eq:svc_obj}
    \begin{aligned}
        \min_{w,b,\xi} \quad & \| w \| + C \sum_{i=1}^n \xi_i \\
            \text{subject to} \quad & y_i (w^T x_i + b) \geq 1 - \xi_i \ \forall_i \\ & \xi_i \geq 0 \ \forall_i
    \end{aligned}
\end{equation}

Minimizing $\| w \|$ is equivalent to minimizing $\displaystyle \frac{1}{2} \| w \|^2$, but in this form we will deal with a 1-strongly convex regularization term that has more desirable convergence properties. So we need to find:

\begin{equation} \label{eq:quad_svc_obj}
    \begin{aligned}
        \min_{w,b,\xi} \quad & \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \xi_i \\
            \text{subject to} \quad & y_i (w^T x_i + b) \geq 1 - \xi_i \ \forall_i \\ & \xi_i \geq 0 \ \forall_i
    \end{aligned}
\end{equation}

where the parameter $C$ controls the trade-off between the slack variable penalty and the size of the margin.

\begin{figure}[h!]
	\centering
	\includegraphics[scale=0.6]{img/linear_dual_l1_svc_hyperplane}
	\caption{Linear SVC hyperplane}
	\label{fig:linear_dual_l1_svc_hyperplane}
\end{figure}

\pagebreak

\subsection{Hinge loss}

The \emph{hinge} loss is defined as:

\begin{equation} \label{eq:hinge_loss1}
	\mathcal{L}_1 = \max(0, 1 - y (w^T x + b))
\end{equation}

or, equivalently:

\begin{equation} \label{eq:hinge_loss2}
	\mathcal{L}_1 =
	\begin{cases}
		0 & \text{if} \ y (w^T x + b) \geq 1 \\
		1 - y (w^T x + b) & \text{otherwise} \\
	\end{cases}
\end{equation}

and it is a nondifferentiable convex function due to its nonsmoothness in 1, but has a subgradient that is given by:

\begin{equation} \label{eq:hinge_loss_der}
    \partial_w \mathcal{L}_1=
        \begin{cases}
            -y x & \text{if} \ y (w^T x + b) < 1 \\
            0 & \text{otherwise} \\
        \end{cases}
\end{equation}

\subsubsection{Primal formulation}

The general primal unconstrained formulation takes the form:

\begin{equation} \label{eq:primal_svm}
    \min_{w,b} \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \mathcal{L}(w,b;x_i,y_i)
\end{equation}

where $\displaystyle \frac{1}{2} \| w \|^2$ is the \emph{regularization term} and $\mathcal{L}(w,b;x_i,y_i)$ is the \emph{loss function} associated with the observation $(x_i,y_i)$~\cite{piccialli2018nonlinear}.

The quadratic optimization problem~\eqref{eq:quad_svc_obj} can be equivalently formulated as:

\begin{equation} \label{eq:primal_l1_svc}
    \min_{w,b} \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \max(0, 1 - y_i (w^T x_i + b))
\end{equation}

where we make use of the \emph{hinge} loss~\eqref{eq:hinge_loss1} or~\eqref{eq:hinge_loss2}.

The above formulation penalizes slacks $\xi$ linearly and is called $\mathcal{L}_1$-SVC.

\begin{figure}[h!]
	\centering
  	\includegraphics[scale=0.4]{img/l1_svc_loss}
  	\caption{Hinge loss with different optimization steps}
  	\label{fig:l1_svc_loss}
\end{figure}

To simplify the notation and so also the design of the algorithms, the simplest approach to learn the bias term $b$ is that of including that into the \emph{regularization term}; so we can rewrite~\eqref{eq:primal_svm} as follows:

\begin{equation} \label{eq:reg_bias_primal_svm1}
    \min_{w,b} \frac{1}{2} (\| w \|^2 + b^2) + C \sum_{i=1}^n \mathcal{L}(w,b;x_i,y_i)
\end{equation}

or, equivalently, by augmenting the weight vector $w$ with the bias term $b$ and each instance $x_i$ with an additional dimension, i.e., with constant value equal to 1:

\begin{equation} \label{eq:reg_bias_primal_svm2}
    \begin{aligned}
        \min_{w} \quad & \frac{1}{2} \| \hat{w} \|^2 + C \sum_{i=1}^n \mathcal{L}(\hat{w};\hat{x}_i,y_i) \\
            \text{where} \quad & \hat{w}^T = [w^T, b] \\ & \hat{x}_i^T = [x_i^T, 1]
    \end{aligned}
\end{equation}

with the advantages of having convex properties of the objective function useful for convergence analysis and the possibility to directly apply algorithms designed for models without the bias term.

In the specific case of the $\mathcal{L}_1$-SVC the objective~\eqref{eq:primal_l1_svc} become:

\begin{equation} \label{eq:reg_bias_primal_l1_svc}
    \min_{w,b} \frac{1}{2} (\| w \|^2 + b^2) + C \sum_{i=1}^n \max(0, 1 - y_i (w^T x_i + b))
\end{equation}

Note that in terms of numerical optimization the formulation~\eqref{eq:primal_l1_svc} is not equivalent to~\eqref{eq:reg_bias_primal_l1_svc} since in the first one the bias term $b$ does not contribute to the \emph{regularization term}, so the SVM formulation is based on an unregularized bias term $b$, as highlighted by the \emph{statistical learning theory}. But, in machine learning sense, numerical experiments in~\cite{hsu2002simple} show that the accuracy does not vary much when the bias term $b$ is embedded into the weight vector $w$.

\subsubsection{Wolfe dual formulation}

To reformulate the~\eqref{eq:quad_svc_obj} as a \emph{Wolfe dual}, we need to allocate the Lagrange multipliers $\alpha_i, \mu_i \geq 0 \ \forall_i$:

\begin{equation} \label{eq:svc_wolfe_dual}
    \max_{\alpha,\mu} \min_{w,b,\xi} \mathcal{W}(w,b,\xi,\alpha,\mu) = \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \xi_i-\sum_{i=1}^n \alpha_i(y_i(w^T x_i + b)-1+\xi_i)-\sum_{i=1}^n\mu_i\xi_i
\end{equation}

We wish to find the $w$, $b$ and $\xi_i$ which minimizes, and the $\alpha$ and $\mu$ which maximizes $\mathcal{W}$, provided $\alpha_i\geq 0, \mu_i \geq 0 \ \forall_i$. We can do this by differentiating $\mathcal{W}$ wrt $w$ and $b$ and setting the derivatives to 0:

\begin{equation} \label{eq:svc_wolfe_der_w}
	\frac{\partial \mathcal{W}}{\partial w}=w-\sum_{i=1}^n \alpha_i y_i x_i \Rightarrow w=\sum_{i=1}^n \alpha_i y_i x_i
\end{equation}

\begin{equation} \label{eq:svc_wolfe_der_b}
	\frac{\partial \mathcal{W}}{\partial b}=-\sum_{i=1}^n \alpha_i y_i\Rightarrow\sum_{i=1}^n \alpha_i y_i=0
\end{equation}

\begin{equation} \label{eq:svc_wolfe_der_xi}
	\frac{\partial \mathcal{W}}{\partial\xi_i}=0\Rightarrow C=\alpha_i+\mu_i
\end{equation}

Substituting~\eqref{eq:svc_wolfe_der_w} and~\eqref{eq:svc_wolfe_der_b} into~\eqref{eq:svc_wolfe_dual} together with $\mu_i\geq 0 \ \forall_i$, which implies that $\alpha\leq C$, gives a new formulation being dependent on $\alpha$. We therefore need to find:

\begin{equation} \label{eq:svc_max_wolfe_dual}
	\begin{aligned}
    	\max_{\alpha} \mathcal{W}(\alpha) &= \sum_{i=1}^n \alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i\alpha_j y_i y_j \langle x_i, x_j \rangle \\
    	&= \sum_{i=1}^n \alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i Q_{ij}\alpha_j \ \text{where} \ Q_{ij} = y_i y_j \langle x_i, x_j \rangle \\
    	&= \sum_{i=1}^n \alpha_i - \frac{1}{2}\alpha^T Q\alpha \ \text{subject to} \ 0\leq\alpha_i\leq C \ \forall_i, \sum_{i=1}^n \alpha_i y_i=0
	\end{aligned}
\end{equation}

or, equivalently:

\begin{equation} \label{eq:svc_min_wolfe_dual}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2}\alpha^T Q\alpha+q^T\alpha \\
            \text{subject to} \quad & 0\leq\alpha_i\leq C \ \forall_i \\ & y^T\alpha=0
    \end{aligned}
\end{equation}

where $q^T = [1, \dots, 1]$.

By solving~\eqref{eq:svc_min_wolfe_dual} we will know $\alpha$ and, from~\eqref{eq:svc_wolfe_der_w}, we will get $w$, so we need to calculate $b$.

We know that any data point satisfying~\eqref{eq:svc_wolfe_der_b} which is a support vector $x_s$ will have the form:

\begin{equation} \label{eq:svc_sv_const1}
	y_s(w^T x_s + b)=1
\end{equation}

and, by substituting in~\eqref{eq:svc_wolfe_der_w}, we get:

\begin{equation} \label{eq:svc_sv_const2}
	y_s\big(\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle +b\big)=1
\end{equation}

where $s$ denotes the set of indices of the support vectors and is determined by finding the indices $i$ where $\alpha_i>0$, i.e., nonzero Lagrange multipliers.

Multiplying through by $y_s$ and then using $y_s^2=1$ from~\eqref{eq:svc_consts}:

\begin{equation} \label{eq:svc_sv_squared_const2}
	y_s^2\big(\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle +b\big)=y_s
\end{equation}

\begin{equation} \label{eq:svc_b}
	b=y_s-\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle
\end{equation}

Instead of using an arbitrary support vector $x_s$, it is better to take an average over all of the support vectors in $S$:

\begin{equation} \label{eq:svc_b_avg}
	b=\frac{1}{N_s}\sum_{s\in S} y_s-\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle
\end{equation}

We now have the variables $w$ and $b$ that define our separating hyperplane's optimal orientation and hence our support vector machine. Each new point $x'$ is classified by evaluating:

\begin{equation} \label{eq:svc_pred}
    y'=\operatorname{sign}\big(\sum_{i=1}^n\alpha_i y_i\langle x_i, x' \rangle+b\big)
\end{equation}

From~\eqref{eq:svc_min_wolfe_dual} we can notice that the equality constraint $y^T \alpha = 0$ arises form the stationarity condition $\partial_{{b}} \mathcal{W}=0$. So, again, for simplicity, we can again consider the bias term $b$ embedded into the weight vector. We report below the box-constrained dual formulation~\cite{hsu2002simple} that arises from the primal~\eqref{eq:reg_bias_primal_svm1} or~\eqref{eq:reg_bias_primal_svm2} where the bias term $b$ is embedded into the weight vector $w$:

\begin{equation} \label{eq:svc_min_bcqp_wolf_dual}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2} \alpha^T (Q + yy^T)\alpha+q^T\alpha \\
            \text{subject to} \quad & 0\leq\alpha_i\leq C \ \forall_i
    \end{aligned}
\end{equation}

\subsubsection{Lagrangian dual formulation}

In order to relax the constraints in the \emph{Wolfe dual} formulation~\eqref{eq:svc_min_wolfe_dual} we define the problem as a \emph{Lagrangian dual} relaxation by embedding them into objective function, so we need to allocate the Lagrange multipliers $\mu$ and $\lambda_+, \lambda_- \geq 0$:

\begin{equation} \label{eq:l1_svc_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda_+,\lambda_-} \min_{\alpha} \mathcal{L}(\alpha,\mu,\lambda_+,\lambda_-) &= \frac{1}{2} \alpha^T Q\alpha+q^T\alpha + \mu^T (y^T \alpha) + \lambda_+^T (ub - \alpha) - \lambda_-^T \alpha \\
    &= \frac{1}{2} \alpha^T Q\alpha + (q + \mu y^T + \lambda_+ - \lambda_-)^T \alpha + \lambda_+^T ub \\
    \text{subject to} \quad & \,\, \lambda_+, \lambda_- \geq 0
	\end{aligned}
\end{equation}

where the upper bound $ub^T = [C, \dots, C]$.

Taking the derivative of the Lagrangian $\mathcal{L}$ wrt $\alpha$ and settings it to 0 gives:

\begin{equation} \label{eq:svc_lagrangian_der_a}
	\frac{\partial \mathcal{L}}{\partial \alpha}=0\Rightarrow Q \alpha + (q + \mu y^T + \lambda_+ - \lambda_-) = 0
\end{equation}

With $\alpha$ optimal solution of the linear system:

\begin{equation} \label{eq:l1_svc_lagrangian_sol}
    Q \alpha = - (q + \mu y^T + \lambda_+ - \lambda_-)
\end{equation}

the gradients wrt $\mu$, $\lambda_+$ and $\lambda_-$ are:

\begin{equation} \label{eq:svc_lagrangian_der_mu}
	\frac{\partial \mathcal{L}}{\partial \mu}=-y \alpha
\end{equation}

\begin{equation} \label{eq:svc_lagrangian_der_lp}
	\frac{\partial \mathcal{L}}{\partial \lambda_+}=\alpha - ub
\end{equation}

\begin{equation} \label{eq:svc_lagrangian_der_lm}
    \frac{\partial \mathcal{L}}{\partial \lambda_-}=-\alpha
\end{equation}

From~\eqref{eq:svc_min_wolfe_dual} we can notice that the equality constraint $y^T \alpha = 0$ arises form the stationarity condition $\partial_{{b}} \mathcal{W}=0$. So, again, for simplicity, we can again consider the bias term $b$ embedded into the weight vector. In this way the dimensionality of~\eqref{eq:l1_svc_lagrangian_dual} is reduced by removing the multipliers $\mu$ which was allocated to control the equality constraint $y^T \alpha=0$, so we will end up solving exactly the problem~\eqref{eq:svc_min_bcqp_wolf_dual}.

\begin{equation} \label{eq:l1_svc_bcqp_lagrangian_dual}
	\begin{aligned}
    	\max_{\lambda_+,\lambda_-} \min_{\alpha} \mathcal{L}(\alpha,\lambda_+,\lambda_-) &= \frac{1}{2} \alpha^T (Q + yy^T)\alpha+q^T\alpha + \lambda_+^T (ub - \alpha) - \lambda_-^T \alpha \\
    &= \frac{1}{2} \alpha^T (Q + yy^T)\alpha + (q + \lambda_+ - \lambda_-)^T \alpha + \lambda_+^T ub \\
    \text{subject to} \quad & \,\, \lambda_+, \lambda_- \geq 0
	\end{aligned}
\end{equation}

where, again, the upper bound $ub^T = [C, \dots, C]$.

Now, taking the derivative of the Lagrangian $\mathcal{L}$ wrt $\alpha$ and settings it to 0 gives:

\begin{equation} \label{eq:l1_svc_bcqp_lagrangian_der_a}
	\frac{\partial \mathcal{L}}{\partial \alpha}=0\Rightarrow (Q + yy^T) \alpha + (q + \lambda_+ - \lambda_-) = 0
\end{equation}

With $\alpha$ optimal solution of the linear system:

\begin{equation} \label{eq:l1_svc_bcqp_lagrangian_sol}
    (Q + yy^T) \alpha = - (q + \lambda_+ - \lambda_-)
\end{equation}

the gradients wrt $\lambda_+$ and $\lambda_-$ are:

\begin{equation} \label{eq:l1_svc_bcqp_lagrangian_der_lp}
	\frac{\partial \mathcal{L}}{\partial \lambda_+}=\alpha - ub
\end{equation}

\begin{equation} \label{eq:l1_svc_bcqp_lagrangian_der_lm}
    \frac{\partial \mathcal{L}}{\partial \lambda_-}=-\alpha
\end{equation}

\bigskip

Note that since the Hessian matrix $Q$ of the $\mathcal{L}_1$-SVC is not strictly positive definite, i.e., the Lagrangian function is not strictly convex since it will be linear along the eigenvectors correspondent to the null eigenvalues and so it will be unbounded below, the Lagrangian dual relaxation, i.e.,~\ref{eq:l1_svc_lagrangian_sol} and~\ref{eq:l1_svc_bcqp_lagrangian_sol}, will be nondifferentiable, so it will have infinite solutions and for each of them it will have a different subgradient. In order to compute an approximation of the gradient, we will choose $\alpha$ in such a way as the one that minimizes the 2-norm since it is good almost like the gradient:

\begin{equation} \label{eq:svc_lagrangian_krylov_sol}
	\min_{\alpha_n \in K_n(Q, b)} \| Q \alpha_n - b \|
\end{equation}

Since we are dealing with a symmetric system we will choose a well-known Krylov method that performs the Lanczos iterate, i.e., symmetric Arnoldi iterate, called \emph{minres}, i.e., symmetric \emph{gmres}, to compute the vector $\alpha_n$ that minimizes the norm of the residual $r_n = Q \alpha_n - b$ among all vectors in $K_n(Q, b) = span(b, Qb, Q^2b, \dots, Q^{n-1}b)$.

\bigskip

Since the linear algebra methods in the ML context are crucial and also in order to deal with a per-iteration cost equals to the other algorithms described later to provide a coherent comparison of all at the end, we will solve it with a primal-dual optimization method and we modify its definition by adding a strictly convex augmentation term, i.e., a penalty term, in order to improve the actual convergence of the algorithms. So, if we consider a general quadratic optimization problem subject to linear constraints, i.e., equality and inequality constraints, defined as:

\begin{equation}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2} \alpha^T Q \alpha + q^T \alpha \\
            \textrm{subject to} \quad & A \alpha = b \\ & G \alpha \leq h \\ & lb \leq \alpha \leq ub
    \end{aligned}
\end{equation}

or, equivalently:

\begin{equation}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2} \alpha^T Q \alpha + q^T \alpha \\
            \textrm{subject to} \quad & A \alpha = b \\ & \hat{G} \alpha \leq \hat{h}
    \end{aligned}
\end{equation}

where $\hat{G} =
\begin{bmatrix}
 G \\
-I \\
 I
\end{bmatrix}$ and $\hat{h} =
\begin{bmatrix}
h & -lb & ub
\end{bmatrix}$; we give the following \emph{augmented Lagrangian dual}:

\begin{equation} \label{eq:svc_gen_aug_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda} \min_{\alpha} \quad & \frac{1}{2} \alpha^T Q \alpha + q^T \alpha + \mu^T (A \alpha - b) + \lambda^T (\hat{G} \alpha - \hat{h}) + \frac{\rho}{2} \| A \alpha - b \|^2 + \frac{\rho}{2} \| \hat{G} \alpha - \hat{h} \|^2 \\
    \text{subject to} \quad & \lambda \geq 0
	\end{aligned}
\end{equation}

with $\rho > 0$.

\bigskip

According to this definition, we change the formulation~\ref{eq:l1_svc_lagrangian_dual} as:

\begin{equation} \label{eq:l1_svc_aug_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda} \min_{\alpha} \mathcal{L}(\alpha,\mu,\lambda) &= \frac{1}{2} \alpha^T Q\alpha+q^T\alpha + \mu^T (y^T \alpha) + \lambda^T (\hat{G} \alpha - \hat{h}) + \frac{\rho}{2} \| y^T \alpha \|^2 + \frac{\rho}{2} \| \hat{G} \alpha - \hat{h} \|^2 \\
    \text{subject to} \quad & \,\, \lambda \geq 0
	\end{aligned}
\end{equation}

and the formulation~\ref{eq:l1_svc_bcqp_lagrangian_dual} as:

\begin{equation} \label{eq:l1_svc_bcqp_aug_lagrangian_dual}
	\begin{aligned}
    	\max_{\lambda} \min_{\alpha} \mathcal{L}(\alpha,\lambda) &= \frac{1}{2} \alpha^T (Q + yy^T) \alpha + q^T \alpha + \lambda^T (\hat{G} \alpha - \hat{h}) + \frac{\rho}{2} \| \hat{G} \alpha - \hat{h} \|^2 \\
    \text{subject to} \quad & \,\, \lambda \geq 0
	\end{aligned}
\end{equation}

where $\hat{G} =
\begin{bmatrix}
-I \\
 I
\end{bmatrix}$ and $\hat{h} =
\begin{bmatrix}
-lb & ub
\end{bmatrix}$ with $lb^T = [0, \dots, 0]$, $ub^T = [C, \dots, C]$ and $\rho > 0$.

\pagebreak

\subsection{Squared hinge loss}

The \emph{squared hinge} loss is defined as:

\begin{equation} \label{eq:squared_hinge_loss2}
	\mathcal{L}_2 = \max(0, 1 - y (w^T x + b))^2
\end{equation}

or, equivalently:

\begin{equation} \label{eq:squared_hinge_loss1}
	\mathcal{L}_2 =
	\begin{cases}
		0 & \text{if} \ y (w^T x + b) \geq 1 \\
		(1 - y (w^T x + b))^2 & \text{otherwise} \\
	\end{cases}
\end{equation}

It is a strictly convex function and its gradient is given by:

\begin{equation} \label{eq:squared_hinge_loss_der}
    \nabla_w \mathcal{L}_2=
        \begin{cases}
            - 2 \max(0, 1 - y (w^T x + b)) y x & \text{if} \ y (w^T x + b) < 1 \\
            0 & \text{otherwise} \\
        \end{cases}
\end{equation}

\subsubsection{Primal formulation}

Since smoothed versions of objective functions may be preferred for optimization, we can reformulate~\eqref{eq:primal_l1_svc} as:

\begin{equation} \label{eq:primal_l2_svc}
    \min_{w,b} \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \max(0, 1 - y_i (w^T x_i + b))^2
\end{equation}

where we make use of the \emph{squared hinge} loss that quadratically penalized slacks $\xi$ and is called $\mathcal{L}_2$-SVC.

The $\mathcal{L}_2$-SVC objective~\eqref{eq:primal_l2_svc} can be rewritten in form~\eqref{eq:reg_bias_primal_svm1} or~\eqref{eq:reg_bias_primal_svm2} as:

\begin{equation} \label{eq:reg_bias_primal_l2_svc}
    \min_{w,b} \frac{1}{2} (\| w \|^2 + b^2) + C \sum_{i=1}^n \max(0, 1 - y_i (w^T x_i + b))^2
\end{equation}

\begin{figure}[h!]
	\centering
  	\includegraphics[scale=0.4]{img/l2_svc_loss}
  	\caption{Squared hinge loss with different optimization steps}
  	\label{fig:l2_svc_loss}
\end{figure}

\subsubsection{Wolfe dual formulation}

As done for the $\mathcal{L}_1$-SVC we can derive the \emph{Wolfe dual} formulation of the $\mathcal{L}_2$-SVC by obtaining:

\begin{equation} \label{eq:wolfe_dual_l2_svc}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2}\alpha^T (Q + D)\alpha+q^T\alpha \\
            \text{subject to} \quad & \alpha_i\geq 0 \ \forall_i \\ & y^T\alpha=0
    \end{aligned}
\end{equation}

or, alternatively, with the regularized bias term by obtaining:

\begin{equation} \label{eq:reg_bias_wolfe_dual_l2_svc}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2}\alpha^T (Q + yy^T + D) \alpha + q^T \alpha \\
            \text{subject to} \quad & \alpha_i \geq 0 \ \forall_i
    \end{aligned}
\end{equation}

where the diagonal matrix $\displaystyle D_{ii} = \frac{1}{2C} \ \forall_i$.

\subsubsection{Lagrangian dual formulation}

In order to relax the constraints in the $\mathcal{L}_2$-SVC \emph{Wolfe dual} formulation~\eqref{eq:wolfe_dual_l2_svc} we define the problem as a \emph{Lagrangian dual} relaxation by embedding them into objective function, so we need to allocate the Lagrange multipliers $\mu$ and $\lambda \geq 0$:

\begin{equation} \label{eq:l2_svc_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda} \min_{\alpha} \mathcal{L}(\alpha,\mu,\lambda) &= \frac{1}{2} \alpha^T (Q+D)\alpha+q^T\alpha + \mu^T (y^T \alpha) - \lambda^T \alpha \\
    &= \frac{1}{2} \alpha^T (Q+D)\alpha + (q + \mu y^T - \lambda)^T \alpha \\
    \text{subject to} \quad & \,\, \lambda \geq 0
	\end{aligned}
\end{equation}

Taking the derivative of the Lagrangian $\mathcal{L}$ wrt $\alpha$ and settings it to 0 gives:

\begin{equation} \label{eq:l2_svc_lagrangian_der_a}
	\frac{\partial \mathcal{L}}{\partial \alpha}=0\Rightarrow (Q+D) \alpha + (q + \mu y^T - \lambda) = 0
\end{equation}

With $\alpha$ optimal solution of the linear system:

\begin{equation} \label{eq:l2_svc_lagrangian_sol}
    (Q+D) \alpha = - (q + \mu y^T - \lambda)
\end{equation}

the gradients wrt $\mu$ and $\lambda$ are:

\begin{equation} \label{eq:l2_svc_lagrangian_der_mu}
	\frac{\partial \mathcal{L}}{\partial \mu}=-y \alpha
\end{equation}

\begin{equation} \label{eq:l2_svc_lagrangian_der_lambda}
    \frac{\partial \mathcal{L}}{\partial \lambda}=-\alpha
\end{equation}

From~\eqref{eq:svc_min_wolfe_dual} we can notice that the equality constraint $y^T \alpha = 0$ arises form the stationarity condition $\partial_{{b}} \mathcal{W}=0$. So, again, for simplicity, we can again consider the bias term $b$ embedded into the weight vector. In this way the dimensionality of~\eqref{eq:l2_svc_lagrangian_dual} is reduced by removing the multipliers $\mu$ which was allocated to control the equality constraint $y^T \alpha=0$, so we will end up solving exactly the problem~\eqref{eq:reg_bias_wolfe_dual_l2_svc}.

\begin{equation} \label{eq:l2_svc_lb_lagrangian_dual}
	\begin{aligned}
    	\max_{\lambda} \min_{\alpha} \mathcal{L}(\alpha,\lambda) &= \frac{1}{2} \alpha^T (Q + yy^T + D) \alpha+q^T\alpha - \lambda^T \alpha \\
    &= \frac{1}{2} \alpha^T (Q + yy^T + D) \alpha + (q - \lambda)^T \alpha \\
    \text{subject to} \quad & \,\, \lambda \geq 0
	\end{aligned}
\end{equation}

Now, taking the derivative of the Lagrangian $\mathcal{L}$ wrt $\alpha$ and settings it to 0 gives:

\begin{equation} \label{eq:l2_svc_lb_lagrangian_der_a}
	\frac{\partial \mathcal{L}}{\partial \alpha}=0\Rightarrow (Q + yy^T + D) \alpha + (q - \lambda) = 0
\end{equation}

With $\alpha$ optimal solution of the linear system:

\begin{equation} \label{eq:l2_svc_lb_lagrangian_sol}
    (Q + yy^T + D) \alpha = - (q - \lambda)
\end{equation}

the gradient wrt $\lambda$ is:

\begin{equation} \label{eq:l2_svc_lb_lagrangian_der_l}
    \frac{\partial \mathcal{L}}{\partial \lambda}=-\alpha
\end{equation}

\bigskip

Note that since the Hessian matrix $Q$ of the $\mathcal{L}_2$-SVC is symmetric and strictly positive definite, we can find the unique solution of the Lagrangian dual relaxation, i.e.,~\ref{eq:l2_svc_lagrangian_sol} and~\ref{eq:l2_svc_lb_lagrangian_sol}, by solving the system with the Cholesky factorization.

\bigskip

Since the linear algebra methods in the ML context are crucial and also in order to deal with a per-iteration cost equals to the other algorithms described later to provide a coherent comparison of all at the end, we will solve it with a primal-dual optimization method and we modify its definition by adding a strictly convex augmentation term, i.e., a penalty term, in order to improve the actual convergence of the algorithms. So, if we consider a general quadratic optimization problem subject to linear constraints, i.e., equality and inequality constraints, defined as:

\begin{equation}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2} \alpha^T Q \alpha + q^T \alpha \\
            \textrm{subject to} \quad & A \alpha = b \\ & G \alpha \leq h \\ & lb \leq \alpha \leq ub
    \end{aligned}
\end{equation}

or, equivalently:

\begin{equation}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2} \alpha^T Q \alpha + q^T \alpha \\
            \textrm{subject to} \quad & A \alpha = b \\ & \hat{G} \alpha \leq \hat{h}
    \end{aligned}
\end{equation}

where $\hat{G} =
\begin{bmatrix}
 G \\
-I \\
 I
\end{bmatrix}$ and $\hat{h} =
\begin{bmatrix}
h & -lb & ub
\end{bmatrix}$; we give the following \emph{augmented Lagrangian dual}:

\begin{equation} \label{eq:l2_svc_gen_aug_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda} \min_{\alpha} \quad & \frac{1}{2} \alpha^T Q \alpha + q^T \alpha + \mu^T (A \alpha - b) + \lambda^T (\hat{G} \alpha - \hat{h}) + \frac{\rho}{2} \| A \alpha - b \|^2 + \frac{\rho}{2} \| \hat{G} \alpha - \hat{h} \|^2 \\
    \text{subject to} \quad & \lambda \geq 0
	\end{aligned}
\end{equation}

with $\rho > 0$.

\bigskip

According to this definition, we change the formulation~\ref{eq:l2_svc_lagrangian_dual} as:

\begin{equation} \label{eq:l2_svc_aug_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda} \min_{\alpha} \mathcal{L}(\alpha,\mu,\lambda) &= \frac{1}{2} \alpha^T (Q + D) \alpha+q^T\alpha + \mu^T (y^T \alpha) + \lambda^T (\hat{G} \alpha - \hat{h}) + \frac{\rho}{2} \| y^T \alpha \|^2 + \frac{\rho}{2} \| \hat{G} \alpha - \hat{h} \|^2 \\
    \text{subject to} \quad & \,\, \lambda \geq 0
	\end{aligned}
\end{equation}

and the formulation~\ref{eq:l2_svc_lb_lagrangian_dual} as:

\begin{equation} \label{eq:l2_svc_lb_aug_lagrangian_dual}
	\begin{aligned}
    	\max_{\lambda} \min_{\alpha} \mathcal{L}(\alpha,\lambda) &= \frac{1}{2} \alpha^T (Q + yy^T + D) \alpha + q^T \alpha + \lambda^T (\hat{G} \alpha - \hat{h}) + \frac{\rho}{2} \| \hat{G} \alpha - \hat{h} \|^2 \\
    \text{subject to} \quad & \,\, \lambda \geq 0
	\end{aligned}
\end{equation}

where $\hat{G} =
\begin{bmatrix}
-I
\end{bmatrix}$ and $\hat{h} =
\begin{bmatrix}
-lb
\end{bmatrix}$ with $lb^T = [0, \dots, 0]$ and $\rho > 0$.


\subsection{Introduction}
In this section, we will be dealing with differential equations of the form $P\p{x, y} + Q\p{x, y}y' = 0$, which we will write as $P\p{x, y}\,\diff{x} + Q\p{x, y}\,\diff{y} = 0$.

Our goal is to find a real-valued function $F\p{x, y}$ such that $F\p{x, y} = C$ implicitly defines a solution to the differential equation. To illustrate this, consider the following example.

\textit{Example:} \\
Let $P\p{x, y} = x$, $Q\p{x, y} = y$. We claim that the equation
\begin{equation}
	F\p{x, y} = x^2 + y^2 = C
\end{equation}
implicitly defines a solution to the exact differential equation
\[
	x\,\diff{x} + y\,\diff{y} = 0.
\]
Taking the derivative with respect to $x$ on both sides of (1) yields
\begin{align*}
	2x + 2y\frac{\diff{y}}{\diff{x}} &= 0 \\
	2x\,\diff{x} + 2y\,\diff{y} &= 0 \\
	x\,\diff{x} + y\,\diff{y} &= 0
\end{align*}
as we wanted to show.

So, $x^2 + y^2 = C$ are solutions to the problem, which are circles of radius $\sqrt{C}$ centered at the origin.

\begin{definition}
	Suppose that the solutions for the differential equation $P\p{x, y} + Q\p{x, y}y' = 0$ are implicitly given by $F\p{x, y} = C$. Then the set $\left\{\p{x, y} \in \mathbb{R}^2 \mid F\p{x, y} = C \right\}$ are called the \textbf{integral curves} of the differential equation.
\end{definition}


\subsection{Binomial expansion}

How can we expand

\((a+b)^n, n\in \mathbb{N}\)

We know that:

\((a+b)^n=(a+b)(a+b)^{n-1}\)

\((a+b)^n=a(a+b)^{n-1}+b(a+b)^{n-1}\)

Each time this is done, the terms split, and each terms is multiplied by either \(a\) or \(b\). That means at the end there are \(n\) total multiplications.

This can be shown as:

\((a+b)^n=\sum_{i=1}^n a^i b^{n-i} c_i\)

So we want to identify \(c_i\).

Each term can be shown as a series of \(n\) \(a\)s and \(b\)s. For example:

\begin{itemize}
\item \(aaba\)
\item \(baaa\)
\end{itemize}

For any of these, there are \(n!\) ways or arranging the sequence, but this includes duplicates. If we were given \(n\) unique terms to multiply there would indeed by \(n!\) different ways this could have arisen, but we can swap \(a\)s and \(b\)s, as they were only generated once. So let's count duplicates.

There are duplicates in the \(a\)s. If there are  \(i\) \(a\)s, then there are \(i!\) ways of rearranging this. Similarly, if there are \(n-i\) \(b\)s, then there are \((n-i)!\) ways or arranging this.

As a result the number of actual observed instances, \(c_i\), is:

\(c_i=\dfrac{n!}{i!(n-i)!}\)

And so:

\((a+b)^n=\sum^n_{i=0} a^i b^{n-i} \dfrac{n!}{i!(n-i)!}\)

We can also write this last term as:

\(\begin{pmatrix}n\\i\end{pmatrix}\)


% Project for Integration Workshop. Dept. Mathematics. UArizona
% A little linearalgebra with non square matrices.

\section{The Fredholm Alternative \& Least Squares Solutions}
Sometimes we focus too much on solving the matrix equation $A \bm{x} = \bm{b}$ for situations where $A$ is a square matrix,  and we ignore situations where $A$ is not square. In many practical applications,  $A$ is an $m \times n$ matrix with $m \neq n$.
In this project, you are going to explore what we mean by \textit{solutions to the matrix equation for non-square matrices}.

\subsubsection*{The Fredholm Alternative}
The Fredholm alternative states that for the matrix equation $Ax=b$, exactly one of the following statements is true:
\begin{itemize}
    \item (Either) There exists an $x$ that solves the matrix equation $Ax=b$.
    \item (Or) There exists a $y$ that solves $A^\top y = 0$ such that $y^\top b \neq 0$.
\end{itemize}
Let $A_m$ be the $3 \times m$ matrix (below), and let $b'$ and $b''$ be the $3 \times 1$ vectors (below).
\begin{equation*}
A_m = \begin{bmatrix} 1 & 1 & 1 & \cdots & 1\\ 1 & 1 & 1 & \cdots & 1 \\ 1 & 2 & 3 &  \cdots & m \end{bmatrix}, \quad  \quad b' = \begin{bmatrix} -1 \\ -1 \\ +1 \end{bmatrix}, \quad b'' = \begin{bmatrix*}[r] -1 \\ 0 \\ +1 \end{bmatrix*}
 % A_n = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 2 \\ \vdots & \vdots & \vdots \\ 1 & 1 & n \end{bmatrix}, \quad  \quad b_1 = \begin{bmatrix} 0 & 0 & -1 \end{bmatrix}, \quad b_2 = \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}
\end{equation*}

\begin{enumerate}[(a)]
    \item Plot the vectors that make the columns of $A_m$ for $m = 5$, and use your figure to describe the column space of $A_m$.
    \item Use pencil-and-paper to verify the Fredholm Alternative for $b'$ and for $b''$.
    \item Plot the vectors $b'$ and $b''$ on the same figure from part (a), and use your figure to provide an intuitive explanation for the Fredholm Alternative.
\end{enumerate}

\subsubsection*{Pseudo-inverses \& Least Squares Solutions}
When there is no solution to the matrix equation $Ax = b$, we may have to look for the `next best thing'. Your co-worker suggests the following matrix algebra to find the `next best thing'.
\begin{equation*}
Ax = b \quad \Rightarrow \quad A^\top A x = A^\top b \quad \Rightarrow \quad x = (A^\top A)^{-1} A^\top b
\end{equation*}
\begin{enumerate}[(a), resume]
    \item What are the dimensions of the matrix $(A^\top A)$? What are requirements on $A$ for the matrix $(A^\top A)$ to be invertable?
    \item For $m = 2$, find the matrix $A^\dagger:=(A_m^\top A_m) A_m^\top$ and compute the vectors $x': = A^\dagger b'$ and $x'':=A^\dagger b''$.
    \item Does $Ax' = b'$? What about $Ax'' = b''$? Can you describe what we mean when we say that $A^\dagger$ gives the `next best thing'?
    \item[($\ast$)] \textit{Bonus:} Use some calculus to verify your answer in part (f).
\end{enumerate}


\chapter{Math}

\section{Functions}
\rih{Equals.} Requirements for equals
\begin{enumerate}
\item Reflexive
\item Symmetric
\item Transitive
\item Non-null
\end{enumerate}
\rih{Compare.} Requirements for compares (total order):
\begin{enumerate}
\item Antisymmetry
\item Transitivity
\item Totality
\end{enumerate}
\section{Divisor}
\runinhead{gcd.} Greatest common divisor.

\begin{python}
def gcd(a, b):
    while b:
        a, b = b, a%b

    return a
\end{python}

Proof. Euclidean Algorithm. Proove the following recursive form:
$$
gcd(a,b) = gcd(b, r)
$$
\section{Prime Numbers}
\subsection{Sieve of Eratosthenes}
\subsubsection{Basics}
To find all the prime numbers less than or equal to a given integer n by Eratosthenes' method:
\begin{enumerate}
\item Create a   list of consecutive integers from 2 through n: (2, 3, 4, ..., n).
\item Initially, let $p$ equal 2, the first prime number.
\item Starting from $p$, enumerate its multiples by counting to n in increments of  $p$, and mark them in the list (these will be $2p$, $3p$, $4p$, ... ; the $p$ itself should not be marked).
\item Find the first number greater than $p$ in the list that is not marked. If there was no such number, stop. Otherwise, let $p$ now equal this new number (which is the next prime), and repeat from step 3.
\end{enumerate}

When the algorithm terminates, the numbers remaining not marked in the list are all the primes below $n$.

\subsubsection{Refinements}
The main idea here is that every value for $p$ is prime, because we have already marked all the multiples of the numbers less than $p$. Note that some of the numbers being marked may have already been marked earlier (e.g., 15 will be marked both for 3 and 5).

As a refinement, it is sufficient to mark the numbers in step 3 starting from $p^2$, because all the smaller multiples of $p$ will have already been marked at that point by the previous smaller prime factor other than $p$. From $p^2$, $p$ becomes the smaller prime factor of a composite number. This means that the algorithm is allowed to terminate in step 4 when $p^2$ is greater than n.

Another refinement is to initialize list odd numbers only, (3, 5, ..., n), and count in increments of $2p$ in step 3, thus marking only odd multiples of $p$. This actually appears in the original algorithm. This can be generalized with wheel factorization, forming the initial list only from numbers coprime with the first few primes and not just from odds (i.e., numbers coprime with 2), and counting in the correspondingly adjusted increments so that only such multiples of $p$ are generated that are coprime with those small primes, in the first place.

To summarized, the refinements include:
\begin{enumerate}
\item Starting from $p^2$; thus $p$ is the smaller prime factor.
\item Preprocessing even numbers and then only process odd numbers; thus the increment becomes $2p$.
\end{enumerate}}

\subsubsection{code}
\begin{python}
def countPrimes(n):
    """
    Find prime using Sieve's algorithm
    :type n: int
    :rtype: int
    """
    if n < 3: return 0
    is_prime = [False if i%2 == 0 else True
                for i in xrange(n)]
    is_prime[0], is_prime[1] = False, False
    for i in xrange(3, int(math.sqrt(n))+1, 2):
        if is_prime[i]:
            for j in xrange(i*i, n, 2*i):
                is_prime[j] = False

    return is_prime.count(True)
\end{python}

\subsection{Factorization}
Backtracking: Section-\ref{factorization}.

\section{Median}
\subsection{Basic DualHeap}
DualHeap to keep track the median when a method to find median is called multiple times.

Here we use the negation of the value as a trick to convert min-heap to max-heap.
\begin{python}
import heapq

class DualHeap(object):
  def __init__(self):
    self.min_h = []
    self.max_h = []  # need to negate the value

  def insert(self, num):
    if not self.min_h or num > self.min_h[0]:
      heapq.heappush(self.min_h, num)
    else:
      heapq.heappush(self.max_h, -num)
    self.balance()

  def balance(self):
    l1 = len(self.min_h)
    l2 = len(self.max_h)
    if l1-l2 > 1:
      heapq.heappush(self.max_h,
                     -heapq.heappop(self.min_h))
      self.balance()
    elif l2-l1 > 1:
      heapq.heappush(self.min_h,
                     -heapq.heappop(self.max_h))
      self.balance()
    return

  def get_median(self):
    """Straightforward"""
\end{python}

\subsection{DualHeap with Lazy Deletion}\label{dh_lazy_del}
Clues:
\begin{enumerate}
\item Wrap the value and wrap the heap
\item When delete a value, mark it with tombstone.
\item When negate the value, only change the value, not the reference.
\item When heap pop, clean the op first.
\end{enumerate}
\begin{python}
import heapq
from collections import defaultdict

class Value(object):
    def __init__(self, val):
        self.val = val
        self.deleted = False

    def __neg__(self):
        """negate without creating new instance"""
        self.val = -self.val
        return self

    def __cmp__(self, other):
        assert isinstance(other, Value)
        return self.val - other.val

    def __repr__(self):
        return repr(self.val)

class Heap(object):
    def __init__(self):
        self.h = []
        self.len = 0

    def push(self, item):
        heapq.heappush(self.h, item)
        self.len += 1

    def pop(self):
        self._clean_top()
        self.len -= 1
        return heapq.heappop(self.h)

    def remove(self, item):
        """lazy delete"""
        item.deleted = True
        self.len -= 1

    def __len__(self):
        return self.len

    def _clean_top(self):
        while self.h and self.h[0].deleted:
            heapq.heappop(self.h)

    def peek(self):
        self._clean_top()
        return self.h[0]

class DualHeap(object):
    def __init__(self):
        self.min_h = Heap()  # represent right side
        self.max_h = Heap()  # represent left side
    # others similar as the previous section's above DualHeap
\end{python}

\section{Modular}
\subsection{Power of 4}
To check whether a number of the power of 4, we can check whether it mod 3 equals 1.
\begin{align*}
4^a &\equiv 1^a\mod 3 \\
&\equiv 1 \mod 3
\end{align*}

Alternatively, we can use bit manipulation based on the power of 4 in the binary form of \pyinline{repeat n 1 << 2}.

\section{Ord}
\runinhead{Number in lexical order.} Given an integer n, return 1 - n in lexicographical order. For example, given 13, return: [1,10,11,12,13,2,3,4,5,6,7,8,9].

\begin{python}
def gen():
    i = 1
    for _ in xrange(n):
        yield i
        if i * 10 <= n:
            i *= 10  # * 10
        elif i % 10 != 9 and i + 1 <= n:
            i += 1  # for current digit
        else:
            while i % 10 == 9 or i + 1 > n:
                i /= 10
            i += 1
\end{python}


\section*{Things To Remember}
$ln(x) \leq x - 1, x>0$; $||x||_2 = \sqrt{x^T x}$; $\nabla_x ||x||_2^2 = 2 x$%; $||x||_p = (\sum_{i=1}^n|x_i|^p)^{\frac{1}{p}}$, $1 \leq p < \infty$

$f(x) = x^T A x$; $\nabla_x f(x) = (A + A^T) x$

%$D_{KL} = \mathbb{E}_p[log(\frac{p(x)}{q(x)})]$; $D_{KL} (P||Q) = \sum_{x \in X}P(x) \cdot log \frac{P(x)}{Q(x)} =  \int_{-\infty}^{+\infty} p(x) log \frac{p(x)}{q(x)} \, dx $ always nonneg

Standard Gaussian: CDF: $\Phi(x) = \int_{-\infty}^{x} \phi(t) \partial t$;\\ %CDF: cumulative distribution function; PDF: standard normal probability density function, $\mu = 0$, $\sigma = 1$
PDF: $\phi(x) = \frac{1}{\sqrt{2\pi}} e^{-(1/2)x^2}$; $\int \phi(x) \partial x = \Phi(x) + c$;\\
$\int x \phi(x) = -\phi(x) + c$; $\int x^2 \phi(x) \partial x = \Phi(x) -x \phi(x) + c$


\subsection{Useful to know}

\subsubsection{Min of iid exponential r.v}
Let  $X_1$,...,$X_nn$  be i.i.d. $Exp(\lambda)$ random variables.

Distribution of $min_i(Xi)$

\begin{align*}
   \mathbf{P}(\min _ i (X_ i)\leq t) &=\\
&=1-\mathbf{P}(\min _ i (X_ i)\geq t)\\
&=1-(\mathbf{P}(X_1\geq t))(\mathbf{P}(X_2\geq t))\ldots (\mathbf{P}(X_ n\geq t))\\
&=1-(1-F_ X(t))^ n \, =\, 1-e^{-n\lambda x}
\end{align*}

Differentiate w.r.t $x$ to get the pdf of $min_i(Xi)$:

\begin{align*}
f_{\text {min}}(x)= (n\lambda ) e^{-(n\lambda ) x}
\end{align*}

\subsubsection{Counting Committees}

Out of  $2n$  people, we want to choose a committee of $n$ people, one of whom will be its chair. In how many different ways can this be done?"

$$n\binom {2n}{n}=2n\binom {2n-1}{n-1}.$$

“In a group of  2n  people, consisting of  n  boys and  n  girls, we want to select a committee of  n  people. In how many ways can this be done?"

$$\binom {2n}{n}=\sum _{i=0}^ n \binom {n}{i}\binom {n}{n-i}$$

“How many subsets does a set with  2n  elements have?"

$$2^{2n}=\sum _{i=0}^{2n}\binom {2n}{i}$$

“Out of  $n$  people, we want to form a committee consisting of a chair and other members. We allow the committee size to be any integer in the range $1$,$2$,...,$n$ . How many choices do we have in selecting a committee-chair combination?"

$$n2^{n-1}=\sum _{i=0}^ n \binom {n}{i}i.$$

\subsection{Finding Joint PDFS}

$f_{X,Y}(x,y)=f_ X(x)f_{Y|X}(y\mid x)$


\textbf{Consider the wave propagation PDE solved in HW 3,}
\begin{align*}
u_t&=p_x,\\
p_t&=u_x,\quad x\in(0,\pi),t>0,
\end{align*}
\textbf{with boundary conditions $u(t,0)=u(t,\pi)=0$ and initial conditions $u(0,x)=e^{-30(x-\pi/2)^2}$ and $p(0,x)=0$.}
\vskip.00005in
\noindent\rule{\textwidth}{1pt}
\vspace{0.1in}
\begin{enumerate}
\item[a)] \textbf{Show that the eigenvalues of the differential operator $\mathcal{L}$ are integers in the imaginary axis.}

\proof By making use of the PDE,
\begin{align*}
u_{xx}=\lambda p_x=\lambda^2u.
\end{align*}
Let $\lambda^2=-k^2$. Then,
\begin{align*}
u=A\cos(kx)+iB\sin(kx).
\end{align*}
Applying the boundary conditions,
\begin{align*}
u(t,0)=A=0,
\end{align*}
and
\begin{align*}
u(t,\pi)=iB\sin(k\pi)=0,
\end{align*}
which implies either $B=0$ (trivial solution) or $\sin(k\pi)=0$. Hence we get
\begin{align*}
k=0,\pm 1,\pm 2,...~\Rightarrow~\lambda=0,\pm i,\pm 2i,...
\end{align*}
since $\lambda=\pm\sqrt{-k^2}=\pm k i$.

\item[b)] \textbf{Consider the finite difference discretization used in HW3 for the spatial derivatives. The eigenvectors of the discrete FD matrix are}
$$
V^k=\begin{bmatrix}i\sin(kx_1)\\i\sin(kx_2)\\\vdots\\i\sin(kx_{N-1})\\cos(kx_{1/2})\\cos(kx_{3/2})\\\vdots\\cos(kx_{N-1/2})\\\end{bmatrix},\quad k=-(N-1),\ldots, -1, 0, 1, \ldots, N-1.
$$
\textbf{Find the analytical expression for the eigenvalues and check it with \texttt{eig(M)} in \textsl{Matlab}.}

From the matrix $M$ given and the $k$-th eigenvector above, we can compute $MV^k$,
\begin{align*}
MV^k = \begin{cases}
-V_{j+N-1}^k + V_{j+n}^k & j \leq N-1 \\
V_1^k & j=N \\
-V_{j-N}^k + V_{j-N+1}^k & N+1 \leq j < 2N-1 \\
-V_{N-1}^k & j = 2N-1
\end{cases}~,
\end{align*}
where $V^k$ represents the $k$-th eigenvector, and $j$ is the column inside the vector. From here we can solve for the eigenvalues for the different values of $j$. Let $j \leq N-1$. Then,
\begin{align*}
MV^k &= \frac{1}{h}\left[-V_{j+N-1}^k + V_{j+n}^k\right] \\
&= -\cos(kx_{j-1/2}) + \cos(kx_{j+1/2}) \\
& = -\cos\left(k\left(j-\frac{1}{2}\right)h\right)+\cos\left(k\left(j+\frac{1}{2}\right)h\right) \\
& = -2\sin(kjh)\sin\left(k\frac{h}{2}\right),
\end{align*}
where we have used the identity $\cos(a+b) - \cos(a - b) = -2\sin(a)(b)$. Then, since we are looking for the eigenvalue that satisfies $MV^k=\lambda^kV^k$, we have
\begin{align*}
&-2\sin(kjh)\sin\left(k\frac{h}{2}\right) = ih\sin(kjh)\lambda_j \\ &\Rightarrow \lambda_j^k = \frac{2i}{h}\sin\left(k\frac{h}{2}\right)=\lambda^k.
\end{align*}
Note that the $k$-th eigenvalue does not depend on $j$. This is the desired result since the $k$-th eigenvalue is the same for the each entry of the $k$-th eigenvector.

Next let $j = N$. Then,
\begin{align*}
& MV^k = \frac{1}{h}V_1^k = \frac{i}{h}\sin(kh)~,
\end{align*}
and
\begin{align*}
& \lambda_N^kV^k = \lambda^k_N\cos(kx_{1/2}) = \lambda^k_N\cos\left(k\frac{h}{2}\right)~,
\end{align*}
which implies
\begin{align*}
& \frac{i}{h}\sin(kh) = \lambda^k_N\cos\left(k\frac{h}{2}\right) \\
& \Rightarrow i\sin(kh) = \lambda^k_Nh\cos\left(k\frac{h}{2}\right) \\
& \Rightarrow 2i\sin\left(k\frac{h}{2}\right)\cos\left(k\frac{h}{2}\right) = \lambda^k_Nh \cos\left(k\frac{h}{2}\right) \\
& \Rightarrow \lambda^k_N =\frac{2i}{h}\sin\left(k\frac{h}{2}\right)=\lambda^k.
\end{align*}
Now, let $j \in [N+1,2N-1)$, then
\begin{align*}
MV^k &= \frac{1}{h}\left[-V_{j-N}^k + V_{j-N+1}^k \right]\\
&= \frac{i}{h}\left[-\sin\left(kx_{j-N}\right)+\sin\left(kx_{j-N+1}\right)\right] \\
& = \frac{i}{h}\left[\sin(k(j-N+1)h) - \sin(k(j-N)h)\right]~,
\end{align*}
and
\begin{align*}
\lambda^k_j V_j^k = \lambda^k_j\cos(kx_{j-N+1/2}) = \lambda^k_j \cos(k(j-N+1/2)h)~.
\end{align*}
This gives us
\begin{align*}
& \frac{i}{h}\left[\sin(k(j-N+1)h) - \sin(k(j-N)h)\right] = \lambda^k_j \cos(k(j-N+1/2)h) \\
& \Rightarrow \frac{2i}{h}\cos(k(j-N+1/2)h)\sin\left(k\frac{h}{2}\right) = \lambda^k_j \cos(k(j-N+1/2)h) \\
& \Rightarrow \lambda^k_j = \frac{2i}{h}\sin\left(k\frac{h}{2}\right)=\lambda^k.
\end{align*}
Finally, for the last entry of the vector, let $j= 2N-1$, then
\begin{align*}
MV^k & = -\frac{1}{h}V_{N-1}^k = -\frac{i}{h}\sin(kx_{N-1}) = -\frac{i}{h}\sin(k(N-1)h) = -\frac{i}{h}\sin(k\pi - kh) \\
& = -\frac{i}{h}\sin(k\pi)\cos(kh) + \frac{i}{h}\sin(kh)\cos(k\pi) = \frac{i}{h}\sin(kh)\cos(k\pi)~,
\end{align*}
and
\begin{align*}
\lambda^k_{2N-1}V^k &= \lambda^k_{2N-1}\cos(k(N-1/2)h) = \lambda^k_{2N-1}\cos\left(knh - k\frac{h}{2}\right) = \lambda^k_{2N-1}\cos\left(k\pi - k\frac{h}{2}\right) \\
& = \lambda^k_{2N-1}\left[\cos(k\pi)\cos\left(k\frac{h}{2}\right) - \sin(k\pi)\sin\left(k\frac{h}{2}\right)\right] = \lambda^k_{2N-1}\cos(k\pi)\cos\left(k\frac{h}{2}\right)~.
\end{align*}
Joining the two results we get the following equation,
\begin{align*}
& \lambda^k_{2N-1}\cos\left(k\pi - k\frac{h}{2}\right) = \frac{i}{h}\sin(kh)\cos(k\pi) \\
& \Rightarrow \lambda^k_{2N-1} = \frac{2i}{h}\sin\left(k\frac{h}{2}\right)=\lambda^k.
\end{align*}
Thus,
\begin{align*}
\boxed{\lambda^k= \frac{2i}{h}\sin\left(k\frac{h}{2}\right)}~.
\end{align*}
\subsection*{Matlab code for this part}
\begin{verbatim}
%% Problem 2b
clear variables
close all

N=100;
h=pi/N;

Du = gallery('tridiag',N,-1,1,0)/h; % In sparse form.
Du(:,end)=[];
Dp = gallery('tridiag',N,0,-1,1)/h; % In sparse form.
Dp(end,:)=[];
Z1 = zeros(N-1,N-1);
Z2 = zeros(N,N);

M = [Z1 Dp ; Du Z2];

E=eig(full(M));
E=sort(E);
k=-(N-1):(N-1);
E_analytic=2*1i/h*sin(k*h/2)';
E_analytic=sort(E_analytic);

difference=norm(E-E_analytic,inf)
% The expression in part b is correct
\end{verbatim}
\item[c)] \textbf{Show that the eigenvalues of $M$ are $\mathcal{O}(h^2)$.}
\proof We can prove the state ment by simply Taylor expanding the previous expression for the eigenvalues :
\begin{align*}
\lambda_k(h) &= \lambda_k(0) + h\lambda'_k(0) + \frac{h^2}{2}\lambda''_k(0) + \frac{h^3}{6} \lambda'''_k(0) + \cdots \\
& = \frac{ik}{h}h - \frac{h^3}{6}i\frac{k^3}{4h} + \mathcal{O}(h^4) \\
& = ik\left[ 1 + \frac{k^2h^2}{6} + \mathcal{O}(h^4) \right].
\end{align*}
Note that we have dropped the $k$ for convenience. Thus,
\begin{align*}
\lambda_k(h)  = ik\left[1 + \mathcal{O}((kh)^2)\right].
\end{align*}

\item[d)] \textbf{Show that}
\begin{align*}
\lambda_{\pm(N-1)} = \pm \frac{2i}{h}\cos(h/2), ~~\text{and}~~|\lambda_{\pm(N-1)}| < \frac{2}{h} = \frac{2}{\pi}N~.
\end{align*}
By simply plugging $\pm(N-1)$ into the eigenvalue equation,
\begin{align*}
\lambda_{\pm (N-1)} &= \frac{2i}{h}\sin\left(\pm N\frac{h}{2} \mp \frac{h}{2} \right) = \frac{2i}{h} \sin \left( \pm \frac{\pi}{2} \mp \frac{h}{2}\right) \\
& = \frac{2i}{h} \left[\sin\left(\pm \frac{\pi}{2}\right)\cos\left(\mp\frac{h}{2}\right) + \cos\left(\pm \frac{\pi}{2}\right)\sin\left(\mp\frac{h}{2}\right) \right] \\
& = \pm \frac{2i}{h}\cos\left(\frac{h}{2}\right)~.
\end{align*}

Using this, we note that $0 \leq |\cos\left(\frac{\pi}{2N}\right)| < 1$ and $|i| = 1$, so,
\begin{align*}
|\lambda_{\pm (N-1)}| =\left|\pm \frac{2i}{h}\cos\left(\frac{h}{2}\right)\right| < \frac{2}{h} = \frac{2N}{\pi}~.
\end{align*}
\item[e)] \textbf{Use \texttt{RK4} to time-step this PDE. Plot the stability region of \texttt{RK4} and find a stable $\Delta t$. Find the growth function $g$ for this method and plot $|g|$ using contour.}

To obtain the growth function $g(z)$ let $z = \Delta t \lambda$ and $f(t,y) = \lambda y$. Then,
 \begin{align*}
 & k_1 = zy^n \\
& k_2 = \left(z + \frac{z^2}{2}\right)y^n \\
 & k_3 = \left(z + \frac{1}{2}\left(z^2 + \frac{z^3}{2}\right)\right)y^n \\
 & k_4 = \left(z + z\left(z + \frac{1}{2} \left(z^2+\frac{z^3}{2}\right)\right)\right)y^n ~,
 \end{align*}
and
\begin{align*}
y^{n+1} &= y^n + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)y^n\\
&=\left(1 + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)\right)y^n\\
&=\left[1 + \frac{1}{6}\left(6z+3z^2+z^3+\frac{z^4}{4}\right)\right]y^n.
\end{align*}
Hence,
\begin{align*}
g(z) = 1 + \frac{1}{6}\left(6z+3z^2+z^3+\frac{z^4}{4}\right).
\end{align*}

We can plot $g(z)$ in \textsl{Matlab} to obtain the stability region, shown in part f).

\item[f)] \textbf{Assuming that the boundary of the stability region crosses the imaginary axis approximately at $\pm 2.81i$, estimate the stability requirement on $\Delta t$ (use part $(4)$ to find $c$ so that $\Delta t < ch$ for stability). For $N = 100$, plot the eigenvalues of $M$ multiplied by $\Delta t$ together with the stability region and verify that they fall inside the stability region.}

For stability we know that
\begin{align*}
\left|\Delta t \lambda_k\right|<2.81,
\end{align*}
for all the eigenvalues $\lambda_k$. Since the largest eigenvalue is going to be the one from part d),
\begin{align*}
\frac{2.81}{|\lambda_k|} > \frac{2.81}{|\lambda_{\pm (N-1)}|} > \frac{2.81}{2}h > \Delta t.
\end{align*}
Thus,
\begin{align*}
\Delta t < 1.405h.
\end{align*}

The figure below shows that $1.405h \lambda_k$ falls within the stability region for all $\lambda_k$.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{P2_ef.eps}}
\caption{Stability region and eigenvalues.}
\end{figure}

\subsection*{Matlab code for this part}
\begin{verbatim}
a=-20;
b=20;
c=-20;
d=20;
dx=0.1;
dy=dx;

[zr,zi]=meshgrid(a:dx:b,c:dy:d);
z=zr+1i*zi;
g=1+1/6*(6*z+3*z.^2+z.^3+z.^4/4);
dt=1.405*h;
lambda=dt*E;

figure
contourf(zr,zi,abs(g),[-inf 1 inf], 'k'), colorbar
hold on
grid on
plot(real(lambda),imag(lambda),'r*')
xlabel('$x(t)$','interpreter','latex','fontsize',16)
ylabel('$y(t)$','interpreter','latex','fontsize',16)
axis('image', [-4 1 -3 3])
set(gca,'fontsize',12)
txt='Latex/FIGURES/P2_ef';
saveas(gcf,txt,'epsc')
\end{verbatim}

\item[g)] \textbf{With $N=100$, solve the PDE using \texttt{RK4} and plot your solution at times $t = \pi/2$, $\pi,3$, $\pi/2$, $2\pi$.}

We can see the solutions at those values of time in the next figure.
\begin{figure}[H]
\centering     %%% not \center
\hspace*{\fill}
\subfigure[Solutions at $t = \pi/2$.]{\includegraphics[scale=0.55]{P2_g1}}
\hfill
\subfigure[Solutions at $t = \pi$.]{\includegraphics[scale=0.55]{P2_g2}}
\hspace*{\fill}

\hspace*{\fill}
\subfigure[Solutions at $t = 3\pi/2$.]{\includegraphics[scale=0.55]{P2_g3}}
\hfill
\subfigure[Solutions at $t = 2\pi$.]{\includegraphics[scale=0.55]{P2_g4}}
\hspace*{\fill}
\caption{Solutions of the acoustic equation.}
\end{figure}
\subsection*{Matlab code for this part}
\begin{verbatim}
%% Problem 2g
clear variables
close all
format long
clc

linewidth=1.7;
legendfontsize=14;
axisfontsize=14;

N=100;
L=pi;
h=L/N;
xu=linspace(0,pi,N+1);
xp=linspace(0+h/2,pi-h/2,N);

Du = gallery('tridiag',N,-1,1,0); % In sparse form.
Du(:,end)=[];
Du=Du/h;
Dp = gallery('tridiag',N,0,-1,1); % In sparse form.
Dp(end,:)=[];
Dp=Dp/h;
Z1 = zeros(N-1,N-1);
Z2 = zeros(N,N);
M = [Z1 Dp ; Du Z2];

u0 = exp(-30*(xu-pi/2).^2)';
p0 = zeros(N,1);
V0 = [u0(2:end-1) ; p0];

t = 0:pi/200:2*pi;
dydt = @(t,V) M*V;
[t,V] = rk4(dydt,[0 2*pi],V0,length(t)-1);
% Plots
figure(1)
plot(xu,[0 V(101,1:N-1) 0],'linewidth',linewidth)
hold on
plot(xp,V(101,N:2*N-1),'linewidth',linewidth)
grid on
axis([0 pi -1 1])
xlabel('$x(t)$','interpreter','latex')
legend({' $u(x,t)$',' $p(x,t)$'},...
 'Interpreter','latex','fontsize',legendfontsize)
set(gca,'fontsize',axisfontsize)
txt='Latex/FIGURES/P2_g1';
saveas(gcf,txt,'epsc')

figure(2)
plot(xu,[0 V(201,1:N-1) 0],'linewidth',linewidth)
hold on
plot(xp,V(201,N:2*N-1),'linewidth',linewidth)
grid on
axis([0 pi -1 1])
xlabel('$x(t)$','interpreter','latex')
legend({' $u(x,t)$',' $p(x,t)$'},...
 'Interpreter','latex','fontsize',legendfontsize)
set(gca,'fontsize',axisfontsize)
txt='Latex/FIGURES/P2_g2';
saveas(gcf,txt,'epsc')

figure(3)
plot(xu,[0 V(301,1:N-1) 0],'linewidth',linewidth)
hold on
plot(xp,V(301,N:2*N-1),'linewidth',linewidth)
grid on
axis([0 pi -1 1])
xlabel('$x(t)$','interpreter','latex')
legend({' $u(x,t)$',' $p(x,t)$'},...
 'Interpreter','latex','fontsize',legendfontsize)
set(gca,'fontsize',axisfontsize)
txt='Latex/FIGURES/P2_g3';
saveas(gcf,txt,'epsc')

figure(4)
plot(xu,[0 V(401,1:N-1) 0],'linewidth',linewidth)
hold on
plot(xp,V(401,N:2*N-1),'linewidth',linewidth)
grid on
axis([0 pi -1 1])
xlabel('$x(t)$','interpreter','latex')
legend({' $u(x,t)$',' $p(x,t)$'},...
 'Interpreter','latex','fontsize',legendfontsize)
set(gca,'fontsize',axisfontsize)
txt='Latex/FIGURES/P2_g4';
saveas(gcf,txt,'epsc')
\end{verbatim}

\item[h)] \textbf{With $N=100$, run you code with $\Delta t$ that is slightly bigger than the optimal (stable) choice, is the computation stable?}

With a choice of $\Delta t=1.471 h$, the solution is unstable as we can see in the next figure.

\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{P2_h.eps}}
\caption{Unstable solution.}
\end{figure}

\subsection*{Matlab code for this part}
\begin{verbatim}
%% Problem 2h
t = 0:1.48*h:2*pi;
dydt = @(t,V) M*V;
[t,V] = rk4(dydt,[0 2*pi],V0,length(t)-1);
figure
plot(xu,[0 V(end,1:N-1) 0])
hold on
plot(xp,V(end,N:2*N-1))
grid on
% axis([0 pi -1 1])
xlim([0 pi])
xlabel('$x(t)$','interpreter','latex')
set(gca,'fontsize',axisfontsize)
txt='Latex/FIGURES/P2_h';
saveas(gcf,txt,'epsc')
\end{verbatim}

\item[i)] \textbf{Using the stability criteria for choosing $\Delta t$, plot the error at $t = 2\pi$ for several values of $N$ and verify that the error decays as $\mathcal{O}(h^2)$. Conclude that the error is dominated by the spatial discretization in this case.}

The spatial discretization dominates the error since the method used is second order in space and fourth order in time, which means the spatial discretization leads the error. However, we can see in the following figure that the observed order of convergence is not the theoretical one. We have obtained a observed order of convergence of $3.998$ for $u$ and $1.993$ for $p$ using the infinity norm for the errors of both. For the interpolation we have not used the first two data points.

\begin{figure}[H]
\centering     %%% not \center
\hspace*{\fill}
\subfigure[Error of $u$ with $N$.]{\includegraphics[scale=0.55]{P2_iu}}
\hfill
\subfigure[Error of $p$ with $N$.]{\includegraphics[scale=0.55]{P2_ip}}
\hspace*{\fill}
\caption{Order of convergence.}
\end{figure}
\subsection*{Matlab code for this part}
\begin{verbatim}
%% Problem 2i
clear variables
close all
format long
clc
%Calculate exact solution with a large number of N
axisfontsize=14;
L=pi;

% Check order of convergence
Nvector=[50 100 200 400 800 1600];
for i=1:length(Nvector)
    N=Nvector(i)
    h=L/N;
    xu=linspace(0,pi,N+1);
    xp=linspace(0+h/2,pi-h/2,N);

    Du = gallery('tridiag',N,-1,1,0); % In sparse form.
    Du(:,end)=[];
    Du=Du/h;
    Dp = gallery('tridiag',N,0,-1,1); % In sparse form.
    Dp(end,:)=[];
    Dp=Dp/h;
    Z1 = zeros(N-1,N-1);
    Z2 = zeros(N,N);
    M = [Z1 Dp ; Du Z2];

    u0 = exp(-30*(xu-pi/2).^2)';
    p0 = zeros(N,1);
    V0 = [u0(2:end-1) ; p0];
    dt=h;
    t = 0:dt:2*pi;
    dydt = @(t,V) M*V;
    [t,V] = rk4(dydt,[0 2*pi],V0,length(t)-1);

    difNormu(i)=norm(V(end,1:N-1)'-u0(2:end-1),inf);
    index=find(xp>=pi/2,1);
    difNormp(i)=norm(V(end,N:2*N-1)'-p0,inf);
end
% Order of convergence of u and p
hvector=pi/1600:1e-5:pi/50;
NvectorCont=20:500:7e3;

% Fitting curves for the error.
% Ignore first 2 points. No asymptotic regime.

cu=polyfit(log(Nvector(3:end)),log(difNormu(3:end)),1);
ufit=exp(cu(2))*NvectorCont.^cu(1);
cp=polyfit(log(Nvector(3:end)),log(difNormp(3:end)),1);
pfit=exp(cp(2))*NvectorCont.^cp(1);
cu(1)
cp(1)

figure
loglog(Nvector,difNormu,'r*')
hold on
loglog(NvectorCont,ufit,'linewidth',2)
grid on
xlim([30 2000])
set(gca,'fontsize',axisfontsize)
xlabel('$N$','interpreter','latex','fontsize',16)
ylabel('$e_u$','interpreter','latex','fontsize',16)
txt='Latex/FIGURES/P2_iu';
saveas(gcf,txt,'epsc')

figure
loglog(Nvector,difNormp,'r*')
hold on
loglog(NvectorCont,pfit,'linewidth',2)
grid on
xlim([30 2000])
set(gca,'fontsize',axisfontsize)
xlabel('$N$','interpreter','latex','fontsize',16)
ylabel('$e_p$','interpreter','latex','fontsize',16)
txt='Latex/FIGURES/P2_ip';
saveas(gcf,txt,'epsc')
\end{verbatim}

\item[j)] \textbf{Repeat part g), but use 3rd order Adams-Bashforth for time-stepping. Use \texttt{RK4} to compute the first 2 time levels. Use a $\Delta t$ that is close to the stability limit for \texttt{AB3}.}
First we calculate the stability region for 3rd order Adams-Bashforth, which has the following form,
\begin{align*}
y^{n+1} = y^n = \frac{\Delta t}{12}(23f^n - 16f^{n-1} + 5f^{n-2}).
\end{align*}
We begin with the substitution $f^n = \lambda y^n$:
\begin{align*}
y^{n+1} = y^n = \frac{\lambda \Delta t}{12}(23y^n - 16y^{n-1} + 5y^{n-2}).
\end{align*}
Note that $y^n = gy^{n-1}$ and let $z = \lambda \Delta t$,
\begin{align*}
gy^n = y^n + \frac{z}{12}\left(23y^n - 16\frac{y^n}{g} + 5\frac{y^n}{g^2}\right).
\end{align*}
Factoring out $y^n$ we get
\begin{align*}
g^3 = g^2 + \frac{z}{12}(23g^2 - 16g +5),
\end{align*}
and solving for $z$,
\begin{align*}
z = 12 \left( \frac{g^3-g^2}{23g^2-16g+5} \right).
\end{align*}

Using Matlab, we find the stability region crosses the imaginary axis just shy of $\pm 0.72$. Thus we calculate a stable time step to be
\begin{align*}
\Delta t\leq 0.36h.
\end{align*}
We will choose in the code $\Delta t = 0.36h$. In the next figure we can see how all the eigenvalues are inside the stability region for the time step chosen.

\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.55]{P2_jRegion.eps}}
\caption{Stability region and eigenvalues.}
\end{figure}

Finally, in the next figure we can see that we obtained the same solution as in part g).

\begin{figure}[H]
\centering     %%% not \center
\hspace*{\fill}
\subfigure[Solutions at $t = \pi/2$.]{\includegraphics[scale=0.55]{P2_j1}}
\hfill
\subfigure[Solutions at $t = \pi$.]{\includegraphics[scale=0.55]{P2_j2}}
\hspace*{\fill}

\hspace*{\fill}
\subfigure[Solutions at $t = 3\pi/2$.]{\includegraphics[scale=0.55]{P2_j3}}
\hfill
\subfigure[Solutions at $t = 2\pi$.]{\includegraphics[scale=0.55]{P2_j4}}
\hspace*{\fill}
\caption{Solutions of the acoustic equation.}
\end{figure}
\subsection*{Matlab code for this part}
\begin{verbatim}
%% Problem 2j
clear variables
close all
clc

legendfontsize=14;
axisfontsize=14;

L=pi;
N=100;
h=L/N;

theta=linspace(0,2*pi,N);
g=exp(1i*theta);
z=12*(g.^3-g.^2)./(23*g.^2-16*g+5);

xu=linspace(0,pi,N+1);
xp=linspace(0+h/2,pi-h/2,N);

Du = gallery('tridiag',N,-1,1,0); % In sparse form.
Du(:,end)=[];
Du=Du/h;
Dp = gallery('tridiag',N,0,-1,1); % In sparse form.
Dp(end,:)=[];
Dp=Dp/h;
Z1 = zeros(N-1,N-1);
Z2 = zeros(N,N);
M = [Z1 Dp ; Du Z2];

e=eig(full(M));
dt=0.36*h;
lambdadt=dt*e;

%Plot eigenvalues in stability region
figure
plot(z,'b','linewidth',2)
hold on
plot(real(lambdadt),imag(lambdadt),'r*')
grid on
xlabel('$\Re(z)$','interpreter','latex'...
    ,'fontsize',16)
ylabel('$\Im(z)$','interpreter','latex'...
    ,'fontsize',16)
txt='Latex/FIGURES/P2_jRegion';
saveas(gcf,txt,'epsc')

u0 = exp(-30*(xu-pi/2).^2)';
p0 = zeros(N,1);
V0 = [u0(2:end-1) ; p0];

F2=@(t,v) M*v;

[t,V]=rk4(F2,[0 2*dt],V0,2);

F1=@(v) M*v;
f0=F1(V(1,:)');
f1=F1(V(2,:)');
f=F1(V(3,:)');

V=V(3,:)'+dt/12*(23*f-16*f1+5*f0);

t=t(end);
outputTime=[pi/2 pi 3*pi/2 2*pi];
n=1;
while t<2*pi
    if (t < outputTime(n) && t+dt >= outputTime(n))
        dt=outputTime(n)-t;
    else
        dt=0.36*h;
    end
    f=F1(V);
    V=V+dt/12*(23*f-16*f1+5*f0);
    %Advance variables
    f0=f1;
    f1=f;
    t=t+dt;
    if (t==outputTime(n))
        figure
        plot(xu',[0; V(1:N-1); 0],'linewidth',2)
        hold on
        plot(xp',V(N:2*N-1),'linewidth',2)
        grid on
        legend({' $u(x,t)$',' $p(x,t)$'},...
        'Interpreter','latex','fontsize',legendfontsize)
        set(gca,'fontsize',axisfontsize)
        xlabel('$x(t)$','interpreter','latex','fontsize',16)
        axis([0 pi -1 1])
        txt=['Latex/FIGURES/P2_j' num2str(n)];
        saveas(gcf,txt,'epsc')
        n=n+1;
    end
end
\end{verbatim}
\end{enumerate}


\subsection{Fran\c cois Vi\`ete}
Fran\c cois Vi\`ete was the first to discover an exact formula for $\pi$.
Here is his formula.
\begin{displaymath}
{2\over\pi}={\sqrt2\over2}\times{\sqrt{2+\sqrt2}\over2}\times
{\sqrt{2+\sqrt{2+\sqrt2}}\over2}\times\cdots
\end{displaymath}
%We can flip it around and write the formula like this.
%\begin{displaymath}
%\pi=2\times{2\over\sqrt2}\times{2\over\sqrt{2+\sqrt2}}\times
%{2\over\sqrt{2+\sqrt{2+\sqrt2}}}\times\cdots
%\end{displaymath}
Let $a_0=0$ and $a_{n}=\sqrt{2+a_{n-1}}$.
Then we can write
\begin{displaymath}
{2\over\pi}={a_1\over2}\times{a_2\over2}\times
{a_3\over2}\times\cdots
\end{displaymath}
%
Solving for $\pi$ we have
\begin{displaymath}
\pi=2\times{2\over a_1}\times{2\over a_2}\times{2\over a_3}\times\cdots=2\prod_{k=1}^\infty
{2\over a_k}
\end{displaymath}
%
Let us now use Eigenmath to compute $\pi$ according to Vi\`ete's formula.
Of course, we cannot calculate all the way out to infinity, we have to stop somewhere.
It turns out that nine factors are just enough to get six digits of accuracy.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
a(n)=test(n=0,0,sqrt(2+a(n-1)))
float(2*product(k,1,9,2/a(k)))
\end{Verbatim}

$\displaystyle 3.14159$

The function $a(n)$ calls itself $n$ times so overall there are
54 calls to $a(n)$.
By using a different algorithm with temporary variables, we can get the
answer in just nine steps.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
a = 0
b = 2
for(k,1,9,a=sqrt(2+a),b=b*2/a)
float(b)
\end{Verbatim}

$\displaystyle 3.14159$


\subsection{Identifying upper and lower bounds of linear programming}

In min/max problem, any feasibly solution is an upper/lower bound.

can we get a bound at the other side?
yes, by doing linear combinations of inequalities
eg maximise
\(30x + 100y\)
subject to:
\(4x + 10y <= 40\)
\(x >=3 \)

We can identify a lower bound by inputting something which works, for example \(x=3\) and \(y=0\). This gives us a lower bound of \(90\).

To get an upper bound we can manipulate the constraints:
\(40x + 100y<=400\)
\(10x>=30\)
And then:
\(40x+100y<=370+30\)
\(40x+100y<=370+10x\)
\(30x+100y<=370\)

So we have an upper bound of \(370\).

This lower bound is a result of doing linear combinations of the inequalities. For different combinations, we could have a lower lower bound.

This is the dual problem. How do we choose the linear combination of inequalities such that the resulting lower bound is minimised?


\Appendix{Mathematical Preliminaries}\label{sec:intro-app}

In the analysis of distributed algorithms, we will encounter power towers and iterated logarithms.

\subsection{Power Tower}

We write power towers with the notation
\[
    {}^i 2 = 2^{2^{\cdot^{\cdot^2}}},
\]
where there are $i$ twos in the tower. Power towers grow very fast; for example,
\begin{align*}
    {}^1 2 &= 2,\\
    {}^2 2 &= 4,\\
    {}^3 2 &= 16,\\
    {}^4 2 &= 65536,\\
    {}^5 2 &= 2^{65536} > 10^{19728}.
\end{align*}

\subsection{Iterated Logarithm}

The iterated logarithm of $x$, in notation $\log^* x$ or $\log^*(x)$, is defined recursively as follows:
\[
    \log^*(x) = \begin{cases}
        0 & \text{ if $x \le 1$}, \\
        1 + \log^*(\log_2 x) & \text{ otherwise}.
    \end{cases}
\]
In essence, this is the inverse of the power tower function. For all positive integers $i$, we have
\[
    \log^*({}^i 2) = i.
\]
As power towers grow very fast, iterated logarithms grow very slowly; for example,
\begin{align*}
    \log^* 2 &= 1, &
    \log^* 16 &= 3, &
    \log^* 10^{10} &= 5, \\
    \log^* 3 &= 2, &
    \log^* 17 &= 4, &
    \log^* 10^{100} &= 5, \\
    \log^* 4 &= 2, &
    \log^* 65536 &= 4, &
    \log^* 10^{1000} &= 5, \\
    \log^* 5 &= 3, &
    \log^* 65537 &= 5, &
    \log^* 10^{10000} &= 5, \dotsc
\end{align*}


\chapter{Back-Propagation}\label{backprop}

One of the most common algorithms used for deep feedforward neural network learning is the \VUname{back-propagation} algorithm (\cite{rumelhart_learning_1986}), sometimes called \VUname{backprop}. This term is often misunderstood as meaning the whole learning algorithm, whereas in fact this name refers only to the algorithm used for computing the gradient of the network, which is then used by the actual learning algorithm, such as gradient descent (\cite{cauchy_methode_1847}), stochastic gradient descent or more advanced algorithms such as ADAM (\cite{kingma_adam:_2014}).

While back-propagation is commonly used for computing the gradient of a feedforward neural network, it is not limited to this task. Given a function \( f \left( \VUvec{x}, \VUvec{y} \right) \), the back-propagation algorithm can be used to compute the gradient of such a function with respect to \( \VUvec{x} \), i. e. \( \nabla_{\VUvec{x}} f \left( \VUvec{x}, \VUvec{y} \right) \). To use the back-propagation algorithm, it is useful to describe the function with a \VUname{computational graph}. A computational graph is a connected directed acyclic graph with nodes representing variables and edges representing operations, that is functions describing how to compute said variables. For example the equation \( y = f \left( x \right) \) is represented by two nodes, \( x \) and \( y \), and a directed edge from \( x \) to \( y \). The node representing \( y \) is labelled \( f \). This label is applied to the end node as the operation can in general have multiple input variables. Some of these labels may be omitted if the itermediary values are not of interest.

\section{Chain Rule of Calculus}
\begin{theorem}\label{chain_rule}
	Let \( x \) be a real number, \( f : \VUfield{R} \to \VUfield{R} \), \( g : \VUfield{R} \to \VUfield{R} \). Let \( y = f \left( x \right) \) and \( z = g \left( y \right) \). Then
	\[ \frac{\mathrm{d} z}{\mathrm{d} x} = \frac{\mathrm{d} z}{\mathrm{d} y} \frac{\mathrm{d} y}{\mathrm{d} x} \]
\end{theorem}

This theorem can be generalized for vectors:

\begin{theorem}\label{chain_rule_vector}
	Let \( \VUvec{x} \in \VUfield{R}^n \), \( f : \VUfield{R}^n \to \VUfield{R}^m \), \( g : \VUfield{R}^m \to \VUfield{R} \). Let \( \VUvec{y} = f \left( \VUvec{x} \right) \) and \( z = g \left( \VUvec{y} \right) \). Then
	\[ \frac{\partial z}{\partial x_i} = \sum_{j = 1}^{m} \frac{\partial z}{\partial y_j} \frac{\partial y_j}{\partial x_i} \]
	or, in vector notation
	\[ \nabla_{\VUvec{x}} z = \left( \frac{\partial \VUvec{y}}{\partial \VUvec{x}} \right)^T \nabla_{\VUvec{y}} z \]
	where \( \frac{\partial \VUvec{y}}{\partial \VUvec{x}} \) is the Jacobian matrix of \( f \).
\end{theorem}

And this theorem can in turn be similarly generalized for tensors:

\begin{theorem}\label{chain_rule_tensor}
	Let \( \VUmat{X} \) be a tensor, \( f \) and \( g \) functions. Let \( \VUmat{Y} = f \left( \VUmat{X} \right) \) and \( z = g \left( \VUmat{Y} \right) \). Then
	\[ \nabla_{\VUmat{X}} z = \sum_j \left( \nabla_{\VUmat{X}} Y_j \right) \frac{\partial z}{\partial Y_j} \]
	where \( j \) is a tuple of indices for \( \VUmat{Y} \).
\end{theorem}

\section{Applying the chain rule}

In a computational graph, if there exists a directed edge from \( x \) to \( y \), the influence changing \( x \) has on \( y \) is represented by \( \frac{\partial y}{\partial x} \). To compute such influence between two variables which are not directly connected by an edge the chain rule can be used. This is trivial if there is only a singular directed path between the variables. However, consider the following problem:

\begin{figure}[h]
	\centering
	\includegraphics[width=150pt]{images/diamond/diamond.pdf}
	\caption{Diamond-style computational graph}\label{diamond}
\end{figure}

\begin{example}
	Let \( x \), \( y \), \( z \) and \( w \) be real numbers such that \( y = f \left( x \right) \), \( z = g \left( x \right) \) and \( w = h \left( y, z \right) \). The computational graph for this set of variables and operations is in figure \ref{diamond}. The derivative \( \frac{\partial w}{\partial x} \) cannot be computed directly by applying theorem \ref{chain_rule}. Instead a sum over both directed paths must be used:
	\[ \frac{\partial w}{\partial x} = \frac{\partial w}{\partial y} \frac{\partial y}{\partial x} + \frac{\partial w}{\partial z} \frac{\partial z}{\partial x} \]
	This is the result of theorem \ref{chain_rule_vector} where
	\[ h : \VUfield R^2 \to \VUfield R :
	\begin{pmatrix}
		y \\
		z
	\end{pmatrix} \mapsto w \]

Generally, in order to find the derivative \( \frac{\partial y}{\partial x} \) the sum of chains of partial derivatives over all directed paths from \( x \) to \( y \) must be taken. This approach becomes computationally infeasible as the number of combinations of all edges increases very quickly with the length of the path. Back-propagation tries to overcome this explosive growth of the number of summands by performing factorization. Therefore, even when an edge is a part of several paths, its gradient is only computed once. This approach reduces computational complexity at the cost of higher memory requirements, a trade-off which may or may not be beneficial depending on the type of problem. For deep feedforward neural networks, the computational graphs are usually complicated and heavily interconnected, which makes such a trade-off a necessity.

The back-propagation algorithm is used to compute a gradient of a given single output variable \( y \) with respect to a set of given input variables \( x^{(i)} \), that is \( \frac{\partial y}{\partial x^{(i)}} \text{ for } i \in \left\{ 1, \dots, n \right\} \). The algorithm can be seen as successively assigning to each node \( u \) of the computational graph the gradient \( \frac{\partial y }{\partial u} \). There are two simplifications to be made for the computational graph:

\begin{remark}\label{backprop-no-dead-ends}
	It can be assumed without loss of generality that from all nodes \( u \) of a given computation graph, there exists a directed path from \( u \) to the output node \( y \). If it were not so, then the gradient for such a node would be \( \frac{\partial y}{\partial u} = 0 \)
\end{remark}
\begin{remark}
	The computational graph can without loss of generality be assumed to only contain such nodes \( u \) that
	\[ \left( \exists i \in \left\{ 1, \dots, n \right\} \right) \left( \exists \text{ a directed path from } x^{(i)} \text{ to } u \right) \]
	This assumption only serves as an optimisation as the excluded nodes do not affect the result of the algorithm and thus can be omitted.
\end{remark}

As a result of the fact that the computational graph is acyclic, all the nodes \( u_1, \dots, u_k \) can be ordered in such a way that if there exists an edge from \( u_j \) to \( u_i \), then \( i < j \) (Reverse topological ordering). Moreover as a result of remark \ref{backprop-no-dead-ends} and the fact that the graph is connected, there holds \( y = u_1 \) and so \( \frac{\partial y}{\partial u_1} = 1 \).

Then, for each node \( u_i \) of the computational graph, its gradient can be written as
\[ \frac{\partial y}{\partial u_i} = \sum_{u_j \in \mathrm{Desc} \left( u_i \right)} \frac{\partial y}{\partial u_j} \frac{\partial u_j}{\partial u_i} \]
where
\[ \mathrm{Desc} \left( u_i \right) = \left\{ u \middle| \exists \text{ a directed edge from } u_i \text{ to } u \right\} \]
If the nodes are reverse topologically ordered and the gradients computed in this order, then each of the necessary partial derivatives needed to compute \( \frac{\partial y}{\partial u_i} \) has already been computed. Using this approach, the gradient for each node is computed only once during the run of the algorithm. The general algorithm for back-propagation is described in algorithm \ref{backprop_algo} (A simplified version of algorithm 6.5 from \cite{goodfellow_deep_2016}).

\begin{algorithm}
	\caption{The general back-propagation algorithm for computational graphs}
	\label{backprop_algo}
	\begin{algorithmic}
		\Require $ \mathcal{G} $ \Comment A pruned computational graph
		\Require $ \left\{ x^{(i)} \right\}_{i = 1}^n $ \Comment A set of input variables
		\Require $ y $ \Comment An output variable
		\Statex
		\Function{build\_grad}{$ x, \mathcal{G}, grad\_table $}
			\If{$ x \in grad\_table $}
				\Return $ grad\_table[x] $
			\EndIf
			\State $ i \gets 1 $
			\For{$ d \in \Call{Desc}{ x, \mathcal{G}} $} \Comment Descendants of $ x $
				\State $ op \gets \Call{get\_operation}{d} $ \Comment Operation producing $ d $
				\State $ D \gets \Call{build\_grad}{d, \mathcal{G}, grad\_table} $
				\State $ G^{(i)} \gets \Call{op.backprop}{\Call{get\_inputs}{d, \mathcal{G}}, x, D} $ \Comment Back-propagation over a single edge
				\State $ i \gets i + 1 $
			\EndFor
			\State $ grad\_table[x] \gets \sum_i G^{(i)} $
		\EndFunction
		\Statex
		\State $ grad\_table[z] \gets 1 $ \Comment A table associating variables and their gradients
		\For{$ x^{(i)} $}
			\Call{build\_grad}{$ x^{(i)}, \mathcal{G}, grad\_table $}
		\EndFor
		\Return $ grad\_table $
	\end{algorithmic}
\end{algorithm}

\section{Application of back-propagation to deep feedforward neural networks}\label{backprop_application}

A deep feedforward neural network (also known as a \VUname{multilayer perceptron}) can be seen as a special case of a computational graph. The \( i \)-th layer of such a network can be represented as:
\[ \VUvec h^{(i)} = f^{(i)} \left( \VUvec{h}^{(i - 1)} ; \VUvec{\theta}^{(i)} \right) = \sigma^{(i)} \left( {\VUmat{W}^{(i)}}^T \VUvec h^{(i - 1)} + \VUvec{b}^{(i)} \right) \]
where \( \VUvec{h}^{(i - 1)} \) is the input of the layer, \( \VUvec{h}^{(i)} \) its output and
\[ \VUvec{\theta}^{(i)} = \left( {\VUmat{W}^{(i)}}^T \middle| \VUvec{b}^{(i)} \right)^T \]
are the parameters of the layer (the weights \( \VUmat{W}^{(i)} \) and the bias \( \VUvec{b}^{(i)} \)).

In such a notation, a neural network with input \( \VUvec{x} \) can be represented as
\[ \widehat{\VUvec{y}} = f^{(n)} \circ f^{(n - 1)} \circ \dots \circ f^{(1)} \left( \VUvec{x} \right) \]
where for each layer \( f^{(i)} \) there exist corresponding parameters \( \VUvec{\theta}^{(i)} \). If \( L \) denotes the loss function of the network and \( \VUvec{y} \) is the desired output of the network for the input \( \VUvec{x} \), then the back-propagation algorithm is used to minimize the loss \( J = L \left( \widehat{\VUvec{y}}, \VUvec{y} \right) \).

For simplicity, in the following text the input and predicted output are also denoted as
\[ \VUvec{h}^{(0)} = \VUvec{x} \qquad \text{and} \qquad \VUvec{h}^{(n)} = \widehat{\VUvec{y}} \]

First, the derivative of the loss with respect to the predicted output is taken:
\[ \nabla_{\widehat{\VUvec{y} }} J = \nabla_{\widehat{\VUvec{y}}} L \left( \widehat{\VUvec{y}}, \VUvec{y} \right) \]
Then, for the \( i \)-th layer, theorem \ref{chain_rule_tensor} is used for the activation function \( \sigma^{(i)} \):
\[ \nabla_{\VUvec{a}^{(i)}} J = \nabla_{\VUvec{h}^{(i)}} J \odot {\sigma^{(i)}}' \left( \VUvec{a}^{(i)} \right) \]
where
\[ \VUvec{a}^{(i)} = {\VUmat{W}^{(i)}}^T \VUvec h^{(i - 1)} + \VUvec{b}^{(i)} \]
From this value, the gradient with respect to the parameters of the layer can be computed as
\[ \nabla_{\VUmat{W}^{(i)}} J = \nabla_{\VUvec{a}^{(i)}} J {\VUvec{h}^{(i - 1)}}^T \]
\[ \nabla_{\VUvec{b}^{(i)}} J = \nabla_{\VUvec{a}^{(i)}} J \]
And for computing the gradients of the layer below:
\[ \nabla_{\VUvec{h}^{(i - 1)}} J = {\VUmat{W}^{(i)}}^T \nabla_{\VUvec{a}^{(i)}} J \]
If the gradients are computed backwards to the order of layers, this ordering is a reverse topological ordering of the computational graph and thus all the values required in each step of the algorithm have already been computed.

This chapter has been sourced from \cite{goodfellow_deep_2016} and \cite{olah_calculus_2015}.
\end{example}


\section{Gaussian integral}

\textbf{Normal distribution}, a term often appeared in statistics, is defined
by equation (\ref{gauss-distr})

\begin{equation}
	p(x)={1\over\sigma\sqrt{2\pi}}
	e^{-\frac12\left(x-\mu\over\sigma\right)^2}
	\label{gauss-distr}
\end{equation}

in which $\sigma$ is the standard deviation (Accordingly, $\sigma^2$ is the
variance), and $\mu$ is the mean value. Since normal distribution is a
\textbf{probability density function}, it should always satisfies (\ref{pdf1})
and (\ref{pdf2}), where set $\mathbb{X}$ denotes all possible values of the
continuous variable $x$.

\begin{equation}
	\forall x\in\mathbb{X},p(x)\ge0
	\label{pdf1}
\end{equation}
\begin{equation}
	\int_{x\in\mathbb{X}} p(x)dx=1
	\label{pdf2}
\end{equation}

It is very easy to prove that $p(x)>0$ because the exponential function is
always positive for all real input. In this section, Condition in equation
(\ref{pdf2}) will be proven.

\subsection{Perform substitution to simplify the integral}

By simplifying the exponential function, we make the following substitutions.

$$
\begin{aligned}
	\frac12\left(x-\mu\over\sigma\right)^2=u^2 \\
	u={1\over\sqrt2}\left(x-\mu\over\sigma\right) \\
	du={dx\over\sigma\sqrt2} \\
	\therefore dx=\sigma\sqrt2du
\end{aligned}
$$

With the substitutions applied, the integral is transformed.

\begin{equation}
	{1\over\sigma\sqrt{2\pi}}\int_{-\infty}^\infty
	e^{-\frac12\left(x-\mu\over\sigma\right)^2}dx=
	{1\over\sigma\sqrt{2\pi}}\cdot\sigma\sqrt2\int_{-\infty}^\infty
	e^{-u^2}du
	={1\over\sqrt\pi}\int_{-\infty}^\infty e^{-u^2}du
	\label{gauss-distr-int}
\end{equation}

By proving $\int_{-\infty}^\infty e^{-u^2}du$, the Gaussian integral, is the
square root of $\pi$, we can then testify the normal distribution satisfies the
second identity of probability density functions.

\subsection{Find the square of the Gaussian integral}

Squaring the integral turns it into a double integral over the
cartesian plane of $(u,v)$, which is also visualized in figure \ref{gauss-3d}.

$$
\begin{aligned}
	\left(\int_{-\infty}^\infty e^{-u^2}du\right)^2
	&=\int_{-\infty}^\infty e^{-u^2}du\int_{-\infty}^\infty e^{-v^2}dv \\
	&=\int_{-\infty}^\infty\int_{-\infty}^\infty e^{-(u^2+v^2)}dudv
\end{aligned}
$$

\begin{figure}[ht]
	\centering{\includegraphics[width=4in]{graphics/gauss-3d.png}}
	\caption{Plot of function $z=e^{-(u^2+v^2)}$}\label{gauss-3d}
\end{figure}

\subsection{Simplify the integral by converting coordinate systems}

Ostensibly, the integral problem seems to become more complicated; however, it
becomes easier with the transformation from Cartesian coordinates to polar
coordinates via applying the rules in equation (\ref{car-to-pol}).

\begin{equation}
	\begin{cases}
		u=r\cos\theta \\
		v=r\sin\theta
	\end{cases}
	\label{car-to-pol}
\end{equation}

Since it is replacing two variables at the same time, the transformation itself
can be interpreted as a vector-valued function. Because $u$ and $v$ are
non-linearly transformed, a function will be needed to transform the tiny piece
$dudv$. Although it is a non-linear transform, a linear approximation for this
transform is made possible by \textbf{Jacobian matrix}.

$$
J=
\begin{bmatrix}
	{\partial u\over\partial r} & {\partial u\over\partial\theta} \\
	{\partial v\over\partial r} & {\partial v\over\partial\theta}
\end{bmatrix}
=
\begin{bmatrix}
	\cos\theta & -r\sin\theta \\
	\sin\theta & r\cos\theta
\end{bmatrix}
$$

Accordingly, $dxdy$ is approximated by the absolute value of the
\textbf{Jacobian determinant}.

$$
\left|\det(J)\right|=\left|r\cos^2\theta+r\sin^2\theta\right|=r
$$

As a result, the complicated double integral over Cartesian plane becomes an
easier double integral over polar regions.

$$
\int_{-\infty}^\infty\int_{-\infty}^\infty e^{-(u^2+v^2)}dudv
=\int_0^{2\pi}\int_0^\infty e^{-r^2}rdrd\theta
$$

\pagebreak
\subsection{Evaluate the double integral}

Since the inner function does not depend on the variable $\theta$, the outer
integral is treated separately.

$$
\int_0^{2\pi}\int_0^\infty e^{-r^2}rdrd\theta
=\int_0^{2\pi}d\theta\int_0^\infty e^{-r^2}rdr=2\pi\int_0^\infty e^{-r^2}rdr
$$

Now, let $a\triangleq-r^2$ so that the exponent is further simplified.

$$
\begin{aligned}
	\because a=-r^2 \\
	\therefore da=-2rdr \\
\end{aligned}
$$
$$
\begin{aligned}
	2\pi\int_0^\infty e^{-r^2}rdr
	&=\pi\int_{-\infty}^0 e^ada \\
	&=\pi\cdot\left.e^a\right|_{-\infty}^0=\pi\cdot1=\pi
\end{aligned}
$$

\subsection{Calculate Gaussian integral from the double integral}

Take the square root of the double integral yields the answer to the original
Gaussian integral.

$$\int_{-\infty}^\infty e^{-u^2}du=\sqrt\pi$$

Plugging the above result into equation (\ref{gauss-distr-int}) makes
(\ref{gauss-distr}) a genuine probability density function.

$$
{1\over\sqrt\pi}\int_{-\infty}^\infty e^{-u^2}du
={1\over\sqrt\pi}\cdot\sqrt\pi=1
$$


\chapter{Finding Derivatives using Fourier Spectral Methods}
Spectral methods are a class of numerical techniques that often utilize the FFT. Spectral methods can be implemented easily in Matlab, but there are some conventions to note. First note that Matlab's ``fft'' and ``ifft'' functions store wave numbers in a different order than has been used so far. The wave numbers in Matlab and in most other FFT packages are ordered, $0,1,...,\frac{n}{2},-\frac{n}{2}+1,-\frac{n}{2}+2,...,-1$. Secondly, Matlab does not take full advantage of real input data. The DFT of real data satisfies the symmetry property $\hat{v}(-k)=\hat{v}(k)$, so it is only necessary to compute half of the wave numbers. Matlab's ``fft" command does not take full advantage of this property and wastes memory storing both the positive and negative wave numbers. Third, spectral accuracy (exponential decay of the magnitude of the Fourier coefficients) is better for smooth functions, so where possible be sure your initial conditions are smooth -- \textbf{When using a Fourier spectral method this requires that your initial conditions are periodic}.
%%%%%%%%
% subsection
%%%%%%%%
\section{Taking a Derivative in Fourier Space}
Let $u(x)$ be a function which is sampled at the $n$ discrete points $x_i \in {h,2h,...,ih,..,2\pi-h,2\pi}$ and $h=2\pi/n$ in real space. Now take the FFT
\begin{eqnarray}
\text{FFT}(u_j) \equiv \hat{u}_k \qquad \text{where} \quad k \in {\frac{-n}{2}+1,...\frac{n}{2}}.
\end{eqnarray}
The Fourier transform of $\frac{\partial^2 u_j}{\partial x^2}$ can be easily computed from $\hat{u}_k$\footnote{More details can be found in Trefethen~\cite[Chap. 3]{Tre00}}:
\begin{eqnarray}
\textrm{FFT}(\frac{\partial^\nu u_j}{\partial x^\nu}) \equiv (ik)^\nu \hat{u}_k \qquad \text{where} \quad \hat{u}_{n/2}=0 \quad, \text{if \quad $\nu$ \quad \text{is odd}}. \label{eq:Spectral_Method_Deriv}
\end{eqnarray}
Thus, differentiation in real space becomes multiplication in Fourier space. We can then take the inverse fast Fourier Transform (IFFT) to yield a solution in real space. In the next section we will use this technique to implement forward Euler and backward Euler timestepping schemes to compute solutions for several PDEs.

%subsection
\subsection{Exercises}
\begin{enumerate}
\item[1)] Let $u(x)=\sum_k \hat{u}_k \exp(ikx)$ be the Fourier series representation of a function $u(x)$. Explain why
$$\frac{\mathrm{d}^{\nu}u}{\mathrm{d}x^{\nu}}=\sum (ik)^{\nu}\hat{u}_k,$$
provided the series converges.
\item[2)] \footnote{This question was prompted by an REU and UROP project due to Sudarshan Balakrishan which is available at \url{http://www.math.lsa.umich.edu/undergrad/REU/projects.html}.} Consider the linear KdV equation
$$u_t+u_{xxx}=0$$
with periodic boundary conditions for $x\in(0,2\pi]$ and the initial data
\begin{align*}
&{} u(x,0)= \left\{{0\quad\textup{ if } 0< x\leq\pi}\atop{1\quad\textup{ if } \pi<x\leq2\pi }\right.
\end{align*}
\begin{enumerate}
\item[a)] Using separation of variables, show that the ``solution'' is
$$u(t,x)=\frac{1}{2}-\frac{2}{\pi}\sum_{j=0}^{\infty}\frac{\sin((2j+1)x-(2j+1)^3t)}{2j+1}.$$
Quotation marks are used because the expression for the solution that is given does not converge when differentiated either once in time or twice in space.
\item[b)] As explained by Olver~\cite{Olv10}, this solution has a fractal structure at times that are an irrational multiple of $\pi$ and a quantized structure at times that are rational multiples of $\pi$. The Matlab program in listing \ref{lst:DispQuant} uses the Fast Fourier transform to find a solution to the linearized KdV equation.  Explain how this program finds a solution to the linearized KdV equation.
\item[c)] Compare the numerical solution produced by the Matlab program with the analytical solution. Try to determine which is more accurate and see if you can find evidence or an explanation to support your suggestions.
\end{enumerate}

\lstinputlisting[style=matlab_style,label=lst:DispQuant,caption={A Matlab program which solves the linearized KdV equation using the Fast Fourier transform.}]{./FindingDerivatives/Programs/DispQuant.m}

\end{enumerate}


\subsection{Set union and intersection}

We discuss functions. Just because we can write a function of sets which exist, does not mean the results of the functions exist. For that we need axioms discussed later.

\subsubsection{Union function}

We define a function on two sets, \(a\lor b\), such that the result contains all elements from either sets.

\(\forall a \forall x \forall y [a\in (x\lor y) \leftrightarrow (a\in x \lor a\in y)]\)

This is commutative: \(a\lor b = b\lor a\)

This is associative: \((a\lor b)\lor c = a\lor (b\lor c)\)

\subsubsection{Intersection function}

We define a function, \(a\land b\), on two sets, such that the result contains all elements which are in both.

\(\forall a \forall x \forall y [a\in (x\land y) \leftrightarrow (a\in x \land a\in y)]\)

This is commutative: \(a\land b = b\land a\)

This is associative: \((a\land b)\land c = a\lor (b\land c)\)

\subsubsection{Distribution of union and intersection}

Union is distributive over intersection: \(a\lor (b\land c)=(a\lor b)\land (a\lor c)\)

Intersection is distributive over union: \(a\land (b\lor c)=(a\land b)\lor (a\land c)\)