\section{The Lagrangian}
\subsection{The Sufficiency Theorem}
We now consider the general case of the constrained optimisation, without explicitly appealing to the convexity assumption.
Suppose we want to minimise $f(x)$ subject to $g(x)=b,x\in X$, to deal with the functional constraint, we introduce the Lagrangian of the problem:
\begin{definition}[Lagrangian]
    The Lagrangian $L:\mathbb R^n\times\mathbb R^m\to\mathbb R$ of the problem is defined by
    $$L(x,\lambda)=f(x)+\lambda^\top(b-g(x))$$
    For $\lambda=(\lambda_1,\ldots,\lambda_m)^\top$, the components $\lambda_i$ constitutes the Lagrange multiplier.
\end{definition}
A nice theorem associated with this notion is the following:
\begin{theorem}[The Lagrange Sufficiency Theorem]\label{lagrange_suff}
    Let $x^\star$ be feasible for the problem.
    Suppose there exists a $\lambda^\star\in\mathbb R^m$ such that
    $$\forall x\in X,L(x^\star,\lambda^\star)\le L(x,\lambda^\star)$$
    Then $x^\star$ is optimal.
\end{theorem}
\begin{proof}
    For any feasible $x$ and any $\lambda\in\mathbb R^m$, we have
    $$L(x,\lambda)=f(x)+\lambda^\top(b-g(x))=f(x)$$
    So our assumption gives, for any feasible $x$,
    $$f(x^\star)=L(x^\star,\lambda^\star)\le L(x,\lambda^\star)=f(x)$$
    as desired.
\end{proof}
\begin{example}
    Consider the problem of minimising $x_1^2+3x_2^2$ subject to $4x_1+x_2=7$.
    We claim that $(12/7,1/7)$ is optimal.
    The Lagrangian has the expression
    $$L(x_1,x_2,\lambda)=x_1^2+3x_2^2+\lambda(7-4x_1-x_2)$$
    Take $\lambda=6/7$, we have
    $$L(x_1,x_2,6/7)=(x_1-12/7)^2+3(x_2-1/7)^2+3\ge 3=L(12/7,1/7,6/7)$$
    So $(12/7,1/7)$ is optimal by the preceding theorem.
\end{example}
Note that our choice of multiplier $\lambda^\star=6/7$ is kind of ``out of nowhere''.
It makes one wonder if there is a systematic method to guarantee to produce a $\lambda^\star$ that works.
Sadly, it is not always possible to find a multiplier that works as a certificate of optimality, let alone a sufficient algorithm to compute one.
However, inspired by the theorem, we can come up with a way to compute $\lambda^\star$ if the problem behaves sufficiently nice.\\
If $x^\star$ and $\lambda^\star$ do exist, we must have
$$\inf_{x\in X}L(x,\lambda^\star)=f(x^\star)>-\infty$$
So to calculate $\lambda^\star$, we first identify the possible candidates
$$\Lambda=\{\lambda\in\mathbb R^n:\inf_{x\in X}L(x,\lambda)>-\infty\}$$
Then, for each $\lambda\in\Lambda$, we consider the probelm of minimising $L(x,\lambda)$ subject to $x\in X$.
This is hopefully easiler since the functional constraint has been removed (with, of course, the trade-off of having to consider the problem for a wide range of functions).
Denote then minimiser as $x(\lambda)$.
We now find a $\lambda^\star\in\Lambda$ such that $x(\lambda^\star)$ is feasible.
In general, it might not be possible to go through all these steps, but if it is possible, then $x^\star=x(\lambda^\star)$ would be guaranteed to be the optimal solution by the theorem.
\begin{example}[Maximum Likelihood Estimator of the Multinomial Distribution]
    Given constants $n_1,\ldots,n_k>0$, we consider the problem to maximise $\sum_in_i\log p_i$ subject to $\sum_ip_i=1,\forall i,p_i>0$.
    The Lagrangian is
    $$L(p,\lambda)=\lambda+\sum_{i=1}^k(n_i\log p_i-\lambda p_i)$$
    Then easily $\Lambda=\mathbb R_{>0}$.
    Now we have
    $$\frac{\partial L}{\partial p_i}=\frac{n_i}{p_i}-\lambda=0\implies p_i(\lambda)=\frac{n_i}{\lambda}$$
    But also
    $$D^2L=\begin{pmatrix}
        -n_1/p_1^2&0&0\\
        0&\ddots&0\\
        0&0&-n_k/p_k^2
    \end{pmatrix}$$
    which is negative definite, so we have found the maximum.
    The constraint $\sum_ip_i=1$ gives $\lambda=\sum_in_i$, so
    $$p_i^\star=\frac{n_i}{\sum_jn_j}$$
    is optimal.
\end{example}
\subsection{Inequality Constraints}
We can consider a somewhat more general version of the optimisation problem, where we replace the functional constraints by their inequality analog.
For vectors $x,y$, we write $x\le y$ if $x_i\le y_i$ for each $i$.\\
Let $f:\mathbb R^n\to\mathbb R,g:\mathbb R^n\to\mathbb R^m,X\in\mathbb R^n$.
We want to consider the problems to minimise $f(x)$ subject to $g(x)\le b,x\in X$.
Notice that we can actually rewrite this problem to an equality form by introducing a slack variable $z\in\mathbb R^m$ by considering the minimisation of $f(x)$ subject to $g(x)+z=b,x\in X,z\ge 0$.\\
Then, the Lagrangian of the modified problem would be
$$L(x,z,\lambda)=f(x)+\lambda^\top(b-g(x)-z)=f(x)+\lambda^\top(b-g(x))-\lambda^\top z$$
Note that if some of $\lambda$ is positive then we can easily send $L$ to $-\infty$, so $\Lambda\subset\{\lambda\in\mathbb R^m:\lambda\le 0\}$.
So if we want to use the Lagrange multiplier method, then a sign constraint is further imposed on $\lambda$.
In particular,
$$\Lambda=\{\lambda\in\mathbb R^m:\lambda\le 0,\inf_{x\in X}(f(x)+\lambda^\top(b-g(x)))>-\infty\}$$
Now we proceed to the second step of using Lagrange multiplier.
Note that for $\lambda\le 0$, we have $\inf_{z\ge 0}(-\lambda^\top z)=0$, so the optimal $z=z(\lambda)$ would satisfy the complementary slackness condition, i.e. $\lambda^\top z=0$.
So if the $i^{th}$ Lagrange multiplier is nonzero, then $z_i=0$, so the $i^{th}$ functional constraint is tight, i.e. hold with equality.
If the $i^{th}$ functional constraint is not tight, then $z_i>0$, thus it must be the case that $\lambda_i=0$.
The rest of the method can then be carried on as usual.
\begin{example}
    Consider the minimisation of $x_1-3x_2$ subject to $x_1^2+x_2^2\le 4$ and $x_1+x_2\le 2$.
    We introduce the slack variables $z_1,z_2\ge 0$, by which we can transform the problem to the minimisation of $x_1-3x_2$ subject to
    $$\begin{cases}
        x_1^2+x_2^2+z_1=4\\
        x_1+x_2+z_2=2\\
        z_1,z_2\ge 0
    \end{cases}$$
    So the Lagrangian is
    $$L=x_1-3x_3+\lambda_1(4-x_1^2-x_2^2-z_1)+\lambda_2(2-x_1-x_2-z_2),\lambda_1,\lambda_2\le 0$$
    Note that
    $$D^2L=\begin{pmatrix}
        -2\lambda_1&0\\
        0&-2\lambda_1
    \end{pmatrix}\succeq 0$$
    Hence we only need $\partial L/\partial x_1=\partial L/\partial x_2=0$, whcih yields
    $$\begin{cases}
        1-2\lambda_1x_1-\lambda_2=0\\
        -3-2\lambda_1x_2-\lambda_2=0
    \end{cases}$$
    If $\lambda_1=0$, then $\lambda_2=1$ and $\lambda_2=-3$, contradiction.
    So $\lambda_1<0$.
    if $\lambda_2<0$, then by complementary slackness, $z=0$, therefore
    $$\begin{cases}
        1-2\lambda_1x_1-\lambda_2=0\\
        -3-2\lambda_1x_2-\lambda_2=0\\
        x_1^2+x_2^2=4\\
        x_1+x_2=2
    \end{cases}$$
    Solving this equation gives $(x_1,x_2)\in\{(2,0),(0,2)\}$, so $(x_1,x_2,\lambda_1,\lambda_2)$ is either $(2,0,1,-3)$ or $(0,2,-1,1)$, but both violates the sign constraint, which gives contradiction again.
    Hence $\lambda_2=0$.
    By complementary slackness $z_1=0$, so the first functional constraint is tight, hence
    $$\begin{cases}
        1-2\lambda_1x_1=0\\
        -3-2\lambda_1x_2=0\\
        x_1^2+x_2^2=4
    \end{cases}$$
    Solve to get $\lambda_1=-\sqrt{10}/4$ and
    $$(x_1,x_2)=\left( -\sqrt{\frac{2}{5}},3\sqrt{\frac{2}{5}} \right)$$
    which is feasible hence optimal.
\end{example}
While we can solve it, it is always good to know beforehand that this method will work.
This brings us to the next topic on the necessity condition.
\subsection{The Necessity Theorem}
Let's go back to the problem of minimising $f(x)$ subject to $g(x)=b,x\in X$.
Denote the Lagrangian by
$$L(x,\lambda)=f(x)+\lambda^\top(b-g(x))$$
We have already established Theorem \ref{lagrange_suff} saying that a feasible $x^\star$ is optimal if there exists some $\lambda^\star$ with $L(x^\star,\lambda^\star)\le L(x,\lambda^\star)$ for any $x\in X$.
In particular, we have
$$\inf_{x\in X,g(x)=b}f(x)=f(x^\star)=\inf_{x\in X}L(x,\lambda^\star)$$
We, of course, want some useful statement on the converse of this statement.
\begin{theorem}[Lagrangian Necessity]\label{lagrange_nece}
    For $b\in\mathbb R^n$, let
    $$\phi(b)=\inf_{x\in X,g(x)=b}f(x)$$
    If $\phi$ is defined and convex, then there is a $\lambda=\lambda(b)$ with
    $$\phi(b)=\inf_{x\in X,g(x)=b}f(x)=\inf_{x\in X}L(x,\lambda(b))$$
    Furthermore, if $\phi$ is differentiable, then $\lambda(b)$ can be taken to be $D\phi(b)$.
\end{theorem}
\begin{definition}
    Such a function $\phi(b)$ is called the value function of the family of the problems of minimising $f(x)$ subject to $g(x)=b,x\in X$.
\end{definition}
\begin{proof}
    Suppose $\phi$ is convex and fix $b\in\mathbb R^m$, then by Theorem \ref{hyperplane}, there is some $\lambda\in\mathbb R^m$ (which can be taken to be $D\phi(b)$) such that
    \begin{align*}
        \phi(b)&=\inf_{c\in\mathbb R^m}(\phi(c)+\lambda^\top(b-c))\\
        &=\inf_{c\in\mathbb R^m}\inf_{x\in X,g(x)=c}(f(x)+\lambda^\top(b-c))\\
        &=\inf_{c\in\mathbb R^m}\inf_{x\in X,g(x)=c}(f(x)+\lambda^\top(b-g(x)))\\
        &=\inf_{x\in X}L(x,\lambda)
    \end{align*}
    which is just what we wanted.
\end{proof}
\begin{remark}
    By adding slack variables
    $$\inf_{x\in X,g(x)\le b}f(x)=\inf_{x\in X,z\ge 0,g(x)+z=b}f(x)$$
    We can see that the theorem can still hold when the functional constraint is an equality.
\end{remark}
To apply this theorem, we can require helps on determining the convexity of $\phi$.
\begin{theorem}
    Suppose $X$ is convex, $f$ is convex, the functional constraint is $g(x)\le b$ and $g_j$ is convex for all $1\le j\le m$, then $\phi$ is convex.
\end{theorem}
\begin{proof}
    Exercise.
\end{proof}
\subsection{Shadow Prices}
We can view the optimisation problem in some real-life economical setting, which gives intuition to the construction of the Lagrangian.\\
Consider a factory owner who makes $n$ different products out of $m$ raw materials.
He needs to choose amount $x_i$ to make of the $i^{th}$ product for each $i=1,\ldots,n$.
Given a vector of amounts $x=(x_1,\ldots,x_n)^\top$ of products to manufacture, the factory requires the amount $g_j(x)$ of the $j^{th}$ raw material for $j=1,\ldots,m$, and the amount of available $j^{th}$ raw material available is $b_j$.
The only nonnegative amounts of products can be produced and given the amounts $x$ of products, the profit earned is $f(x)$.\\
Obviously, what the factory owner wants is to maximise $f(x)$ subject to $g(x)\le b,x\ge 0$.
We let $\phi(b)$ be the maximised profit as a function of the amount of raw materials.\\
To solve this problem, we can consider a fictional world where there is a market for the raw material with the unit price of the $j^{th}$ raw material being $\lambda_j$, then the total profit is exactly
$$L(x,\lambda)=f(x)+\lambda^\top(b-g(x))$$
So the problem has transformed into the maximisation problem of the Lagrangian.
For each price vector $\lambda$, the optimal amount of products is the vector $x(\lambda)$.
We can now find a price vector $\lambda^\star$ such that $x^\star=x(\lambda)$ is feasible.
Such a $\lambda^\star$ is called the shadow price vector of the raw materials.
This is the prices of the raw materials such that the maximisation problem makes no difference on whether there is such a fictional market.\\
The supporting hyperplane can also have an economical interpretation.
Suppose the fictional market of raw materials suddenly appears, then the factory owner would buy the basket $(\epsilon_1,\ldots,\epsilon_m)^\top$ of raw materials if $\phi(b+\epsilon)-\phi(b)\ge\lambda^\top\epsilon$, so when $\epsilon$ is small,
$$\phi(b+\epsilon)-\phi(b)=D\phi(b)^\top\epsilon$$
So if $\lambda_j>\partial\phi/\partial b_j$, then the factory owner should not be buying any of that raw material.
Hence the shadow price has to be $D\phi(b)$ as before.\\
To check that all these are consistent, we check the followings:
The inequality constraint $g(x)\le b$ gives a sign constraint $\lambda\ge 0$, which is consistent with the assumption that price should not be negative.
Also, if the $j^{th}$ constraint is not tight, then $g_j(x^\star)<b_j$, then the owner does not use up all the $j^{th}$ raw material, so the shadow price has $\partial\phi/\partial b_j=0$ as there is no extra profit in acquiring some more of that material.
On the other hand, the $j^{th}$ slack variable would be positive, hence $\lambda_j=0$ by complementary slackness, which makes sense economically.
\subsection{Duality}
We can introduce the notion of a dual optimisation problem by a corresponding economic motivation.
Consider the point of view of the seller of the raw materials who then buy the products produced by the factory.
If the amount of finished products is $x$ and the price of raw materials is $\lambda$, then the seller's profit would be the negative of the profit of factory owner, which is $f(x)+\lambda^\top(b-g(x))$.
So what the seller wants would be to minimise
$$\sup_{x\ge 0}(f(x)+\lambda^\top(b-g(x)))$$
\begin{definition}
    Consider the primal problem of minimising $f(x)$ subject to $g(x)=b,x\in X$ which has Lagrangian
    $$L(x,\lambda)=f(x)+\lambda^\top(b-g(x))$$
    Then the set of feasible Lagrange multipliers is
    $$\Lambda=\{\lambda\in\mathbb R^m:\inf_{x\in X}L(x,\lambda)>-\infty\}$$
    Then the dual objective function is defined as
    $$h:\Lambda\to\mathbb R,h(\lambda)=\inf_{x\in X}L(x,\lambda)$$
    The dual problem of the primal problem is to maximise $h(\lambda)$ subject to $\lambda\in\Lambda$.
\end{definition}
\begin{remark}
    One should note that if one need to study the dual problem of a problem with inequality constraints, then we need to introduce the slack variables first.
    Also, we can formulate the dual problem to a maximisation problem analogously.
\end{remark}
\begin{theorem}[Weak Duality]\label{weak_dual}
    With the definitions introduced above,
    $$\sup_{\lambda\in\Lambda} h(\lambda)\le\inf_{x\in X,g(x)=b}f(x)$$
\end{theorem}
\begin{proof}
    For any $x$ feasible for the primal problem and $\lambda$ feasible for the dual problem, then
    $$h(\lambda)=\inf\{L(\xi,\lambda):\xi\in X\}\le L(x,\lambda)=f(x)$$
    As $L(x,\lambda)=f(x)$ whenever $x$ is feasible.
\end{proof}
So we can restate Theorem \ref{lagrange_suff} in the following way instead:
If $x^\star$ and $\lambda^\star$ are feasible for the primal and dual problems respectively, and $h(\lambda^\star)=f(x^\star)$, then $x^\star$ is optimal.
\begin{remark}
    The difference
    $$\inf_{x\in X,g(x)=b}f(x)-\sup_{\lambda\in\Lambda}h(\lambda)$$
    is called the duality gap.
    It is in general nonnegative by Theorem \ref{weak_dual}.
    By Theorem \ref{lagrange_nece}, if the value function of the primal problem is convex, then there is some $\lambda^\star\in\Lambda$ such that
    $$\inf_{x\in X,g(x)=b}f(x)=\inf_{x\in X}L(x,\lambda^\star)=h(\lambda^\star)$$
    So the duality gap is zero.
    This is called \textit{strong duality}.
\end{remark}
\subsection{Duality in Linear Programs}
\begin{definition}
    Linear Programs are problems of maximising $c^\top x$ subject to $Ax\le b,x\ge 0$ where $A$ is an $m\times n$ matrix, $b\in\mathbb R^m$ and $c\in\mathbb R^n$.
\end{definition}
Of course, we can analyze the dual problem too.
Introduce the slack variable $z\in\mathbb R^m$ and transform the problem to maximising $c^\top x$ subject to $Ax+z=b,x\ge 0,z\ge 0$.
So the Lagrangian is
$$L(x,z,\lambda)=c^\top x-\lambda^\top(b-Ax-z)=b^\top\lambda +(c-A^\top\lambda)^\top x-\lambda^\top z$$
So
$$\Lambda=\{\lambda\in\mathbb R^m:\sup_{x,z\ge 0}L(x,z,\lambda)<\infty\}=\{\lambda\in\mathbb R^m:A^\top\lambda\ge c,\lambda\ge 0\}$$
Now the dual objective function is
$$\sup_{x\ge 0,z\ge 0}L(x,z,\lambda)=b^\top\lambda$$
for $\lambda\in\Lambda$.
Hence the dual problem is to minimise $b^\top\lambda$ subject to $A^\top\lambda\ge c,\lambda\ge 0$.
This looks just like our original problem.
\subsection{The Barrier Method}
Consider the optimisation problem of minimising a differentiable convex function $f:\mathbb R^n\to\mathbb R$ subject to $g(x)\le b$, where $g:\mathbb R^n\to\mathbb R^m$ has every one of its components differentiable.
To tackle it (computationally), we consider the family of unconstrained minimisation problems of
$$f(x)-\epsilon\sum_{i=1}^m\log(b_i-g_i(x))$$
for $\epsilon>0$.
Note that the constraint $g(x)<b$ is implicitly imposed due to the domain of $\log$.
The idea of it, as one might have already observed, is to add an extra term which blows up to positive infinity if the condition $g(x)<b$ is not satisfied.
The reason why we use a term in the $\epsilon\log$ form is to smoothen the blow-up term for computational issues.
\begin{theorem}\label{barrier_method}
    Suppose $x^\star$ is optimal for our original problem and $x_\epsilon$ is optimal for the problem shown above, then
    $$0\le f(x_\epsilon)-f(x^\star)\le m\epsilon$$
\end{theorem}
To prove it, we shall introduce some preliminary notion.
The Lagrangian for the original problem is $L(x,z,\lambda)=f(x)+\lambda^\top (b-z-g(x))$ for the slack variable $z\ge0$ and the set of feasible Lagrange multipliers are
$$\Lambda=\{\lambda\in\mathbb R^m:\inf_{x\in\mathbb R^n,z\ge 0}L(x,z,\lambda)>-\infty\}$$
\begin{claim}
    Let $\lambda\in\mathbb R^m$ be such that $\lambda\le 0$ and that there exists some $x_\lambda\in\mathbb R^n$ with
    $$Df(x_\lambda)=\sum_{i=1}^m\lambda_iDg_i(x_\lambda)$$s
    Then $\lambda\in\Lambda$ and the dual objective function for this problem is just
    $$h(\lambda)=b^\top\lambda+f(x_\lambda)-\lambda^\top g(x_\lambda)$$
\end{claim}
\begin{proof}
    Recall that every convex differentiable function is minimised at any of its stationary points.
    Also note that $f(x)-\lambda^\top g(x)$ is convex.
    So for $x\in\mathbb R^n$ and $z\ge 0$ we have the bound
    $$L(x,z,\lambda)=b^\top\lambda+f(x)-\lambda^\top g(x)-\lambda^\top z\ge b^\top\lambda+f(x_\lambda)-\lambda^\top g(x_\lambda)$$
    Equality holds when $x=x_\lambda$ and $(z,\lambda)$ satisfies complementary slackness.
    The rest follows.
\end{proof}
\begin{proof}[Proof of Theorem \ref{barrier_method}]
    Recall that a differentiable function on an open set has derivative $0$ on any of its local minima.
    As $x_\epsilon$ is optimal to the modified problem, we necessarily have
    $$Df(x_\epsilon)=-\epsilon\sum_{i=1}^m\frac{Dg_i(x_\epsilon)}{b_i-g(x_\epsilon)}$$
    So we set, for $i=1,\ldots,m$ we define $\lambda_i=-\epsilon/(b_i-g_i(x_\epsilon))$.
    Then
    $$Df(x_\epsilon)=\sum_{i=1}^m\lambda_iDg_i(x_\epsilon)$$
    So by the preceding claim $\lambda\in\Lambda$.
    Therefore, by Theorem \ref{weak_dual},
    $$f(x_\epsilon)\ge f(x^\star)\ge h(\lambda)=f(x_\epsilon)+\lambda^\top(b-g(x_\epsilon))=f(x_\epsilon)-m\epsilon$$
    which is the bound we wanted.
\end{proof}
\begin{remark}
    As $f$ and $g_i$ are convex for all $i$, the value function $\phi(b)=\inf\{f(x):g(x)\le b\}$ is convex.
    So by fixing $b$ and assuming there exists an optimiser $x^\star$, Theorem \ref{lagrange_nece} then tells us there is some $\lambda^\star$ with
    $$L(x^\star,z^\star)=\inf\{L(x,z,\lambda^\star):x\in\mathbb R^n,z\ge 0\},z^\star=b-g(x^\star)$$
\end{remark}
Theorem \ref{barrier_method} inspires us to invent an algorithm to approach the optimal solution.
Pick an initial guess $x_0\in\mathbb R^n$ with $g(x_0)<b$ and some initial $\epsilon_0>0$.
For $k\ge 0$, we solve the modified problem with $\epsilon=\epsilon_k$ approximately (using e.g. gradient descent or Newton's method) starting from $x_k$.
Then $x_{k+1}$ is defined to be the approximated optimal solution in this way, and reduce $\epsilon$ by $\epsilon_{k+1}=r\epsilon_k$ for some $r\in (0,1)$.


\section{Extreme Values}
Extreme values of a function are the minimum and maximum values it attains on an interval.
The absolute extreme values would the the extreme values accross the function's entire domain.

\begin{example}
	Find the extreme values of $x^2$ over the following intervals.
	\begin{enumerate}
		\item $(-\infty, \infty)$
		\item $[0,2]$
		\item $(0,2]$
		\item $(0,2)$
	\end{enumerate}
\end{example}
\begin{answer}
	\begin{enumerate}
		\item For the max, there is no maximum value because we can keep increasing $x$ to get a larger output.
			The min is 0 when $x=0$.
		\item The max is 4 when $x=2$.
			The min is 0 when $x=0$.
		\item The max is 4 when $x=2$.
			Although the function approaches 0 in the limit as $x$ approaches 0, there is no min because 0 itself is not a value $x^2$ can take on the interval.
		\item Although the function approaches 4 as $x$ approaches 2, there is no max because 4 itself is not a value $x^2$ can take on the interval.
			Like in the previous question, there is no min.
	\end{enumerate}
	We see that a function can fail to have a max or min value, but this cannot happen on a finite, closed interval.
\end{answer}

\begin{theorem}[Extreme Value Theorem]
	If $f$ is continuous on some finite, closed interval $[a,b]$, then $f$ must have both a minimum and maximum value on the interval.
\end{theorem}

We can find these extreme values by following these steps.
\begin{enumerate}
	\item Find any relative/local minima and maxima.
	\item Find the function values for these local minima and maxima.
	\item Find the function values at the endpoints of the interval, $a$ and $b$.
	\item The smallest such function value will be the absolute minima, while the largest such function value will be the absolute maxima.
\end{enumerate}

\begin{theorem}
	If a function $f$ has a local extrema at a point $c$ interior to its domain and $f^\prime$ exists at $c$, then
	\begin{equation*}
		f^\prime(c) = 0.
	\end{equation*}
\end{theorem}

So, the only candidate values we need to check are the endpoints and where the derivative is 0.
Such points are called critical points.

\begin{example}
	Find the absolute extrema of $y = x^3 + x^2 - 8x + 5$ on the interval $[-3,2]$.
\end{example}
\begin{answer}
	\begin{enumerate}
		\item We'll take the derivative to find all the critical points.
				$y^\prime = 3x^2 + 2x - 8$, which is 0 when $x=-2$ and $x=4/3$.
		\item $(-2)^3 + (-2)^2 - 8(-2) + 5 = 17$ and $(4/3)^3 + (4/3)^2 - 8(4/3) + 5 = -41/27$.
		\item $(-3)^3 + (-3)^2 - 8(-3) + 5 = 11$ and $(2)^3 + (2)^2 - 8(2) + 5 = 1$.
		\item Of these values, $(-2,17)$ is the absolute maxima and $(4/3, -41/27)$ is the absolute minima.
	\end{enumerate}
\end{answer}


\section*{Exercises}

\begin{ex}
  Let $A=\begin{mymatrix}{ccc} 1 & 1 & 0 \\ 1 & 2 & 0 \\ 0 & 0 &
    2 \end{mymatrix}$, and consider $\R^3$ with the inner product given by
  $\iprod{\vect{u},\vect{v}} = \vect{u}^TA\vect{v}$. Which of the
  following vectors are orthogonal to each other?
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{c}  1 \\  1 \\  1 \end{mymatrix},\quad  % 2,  3,   2
    \vect{u}_2 = \begin{mymatrix}{c} -1 \\  2 \\ -2 \end{mymatrix},\quad  % 1,  3,  -4
    \vect{u}_3 = \begin{mymatrix}{c}  7 \\ -5 \\ -2 \end{mymatrix},\quad  % 2, -3,  -4
    \vect{u}_4 = \begin{mymatrix}{c} 10 \\ -2 \\ -7 \end{mymatrix}.       % 8,  6, -14
  \end{equation*}
  \begin{sol}
    We have
    $\iprod{\vect{u}_1,\vect{u}_2} = \vect{u}_1^T A \vect{u}_2 = 0$,
    $\iprod{\vect{u}_1,\vect{u}_3} = \vect{u}_1^T A \vect{u}_3 = -5$,
    $\iprod{\vect{u}_1,\vect{u}_4} = \vect{u}_1^T A \vect{u}_4 = 0$,
    $\iprod{\vect{u}_2,\vect{u}_3} = \vect{u}_2^T A \vect{u}_3 = 0$,
    $\iprod{\vect{u}_2,\vect{u}_4} = \vect{u}_2^T A \vect{u}_4 = 32$,
    and
    $\iprod{\vect{u}_3,\vect{u}_4} = \vect{u}_3^T A \vect{u}_4 = 54$.
    Therefore, $\vect{u}_1\orth\vect{u}_2$,
    $\vect{u}_1\orth\vect{u}_4$, and $\vect{u}_2\orth\vect{u}_3$. None
    of the other pairs of vectors are orthogonal.
  \end{sol}
\end{ex}

\begin{ex}
  On $C[-1,1]$, which of the following functions are orthogonal to each other?
  \begin{equation*}
    f_1(x) = x,\quad
    f_2(x) = x^2,\quad
    f_3(x) = x^3-x,\quad
    f_4(x) = 1-x^4.
  \end{equation*}
  \begin{sol}
    $f_1\orth f_2$, $f_1\orth f_4$, $f_2\orth f_3$, and $f_3\orth f_4$.
  \end{sol}
\end{ex}

\begin{ex}
  Consider the inner product space $\Poly_3$ of polynomials of degree
  at most $3$, with the inner product defined by
  \begin{equation*}
    \iprod{f,g} = \int_{-1}^{1} f(x)g(x)\,dx.
  \end{equation*}
  \begin{enumerate}
  \item Find the orthogonal complement of $\set{x^2,x}$.
  \item Find the orthogonal complement of $\set{x+1}$.
  \end{enumerate}
  \begin{sol}
    \begin{enumerate}
    \item Let $p(x)=ax^3+bx^2+cx+d$. Then
      $\iprod{p(x), x^2} = \frac{2}{5}b + \frac{2}{3}d$ and
      $\iprod{p(x), x} = \frac{2}{5}a + \frac{2}{3}c$.  Therefore,
      $p(x)$ is in the orthogonal complement of $\set{x^2,x}$ if and
      only if $\frac{2}{5}b + \frac{2}{3}d=0$ and
      $\frac{2}{5}a + \frac{2}{3}c=0$. It follows that
      $b=-\frac{5}{3}d$ and $a=-\frac{5}{3}c$. The general solution is
      $p(x) = -\frac{5}{3}cx^3 - \frac{5}{3}dx^2 + cx + d$. A basis
      for the orthogonal complement is
      $\set{-\frac{5}{3} x^3 + x, -\frac{5}{3} x^2 + 1}$.
    \item A basis for the orthogonal complement of $\set{x+1}$ is
      $\set{5x^3-1, 3x^2-1, 3x-1}$.
    \end{enumerate}
  \end{sol}
\end{ex}

\begin{ex}
  Consider $\R^3$ as an inner product space with the usual dot
  product.  For each of the following bases of $\R^3$, state whether
  it is orthonormal, orthogonal, or neither.
  \begin{enumerate}
  \item $\set{
      \begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 0 \\ 1 \end{mymatrix}
    }$.
  \item $\set{
      \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 1 \\ 1 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 0 \\ 1 \end{mymatrix}
    }$.
  \item $\set{
      \begin{mymatrix}{r}  1 \\ 0 \\ 2 \end{mymatrix},
      \begin{mymatrix}{r}  0 \\ 1 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} -2 \\ 0 \\ 1 \end{mymatrix}
    }$.
  \item $\def\arraystretch{1.2}
    \set{
      \begin{mymatrix}{r} \frac{3}{5} \\ \frac{4}{5} \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 0 \\ -1 \end{mymatrix},
      \begin{mymatrix}{r} \frac{4}{5} \\ -\frac{3}{5} \\ 0 \end{mymatrix}
    }$.
  \end{enumerate}
  \begin{sol}
    (a) Orthonormal (therefore also orthogonal). (b) Neither
    orthogonal nor orthonormal. (c) Orthogonal (not orthonormal). (d)
    Orthonormal (therefore also orthogonal).
  \end{sol}
\end{ex}

\begin{ex}
  Suppose $B=\set{\vect{u}_1,\vect{u}_2,\vect{u}_3}$ is an orthogonal
  basis for an inner product space $V$, such that
  $\norm{\vect{u}_1}=2$, $\norm{\vect{u}_2}=\sqrt{3}$, and
  $\norm{\vect{u}_3}=\sqrt{5}$.  Moreover, suppose that
  $\vect{v}\in V$ is a vector such that
  $\iprod{\vect{v},\vect{u}_1} = 1$,
  $\iprod{\vect{v},\vect{u}_2} = 2$, and
  $\iprod{\vect{v},\vect{u}_3} = -4$.  Find the coordinates of
  $\vect{v}$ with respect to $B$.
  \begin{sol}
    $\vect{v} = \frac{1}{4}\vect{u}_1 + \frac{2}{3}\vect{u}_2 - \frac{4}{5}\vect{u}_3$.
  \end{sol}
\end{ex}

\begin{ex}
  Suppose $B=\set{\vect{u}_1,\vect{u}_2,\vect{u}_3}$ is an
  orthogonal basis of $\R^3$. We have been told that
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{c} 1 \\ 1 \\ 0 \end{mymatrix},
  \end{equation*}
  but it is not known what $\vect{u}_2$ and $\vect{u}_3$ are. Find the
  first coordinate of the vector
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{c} 1 \\ 0 \\ 2 \end{mymatrix}
  \end{equation*}
  with respect to the basis $B$.
  \begin{sol}
    We have $\vect{v} = a_1\vect{u}_1 + a_2\vect{u}_2 + a_3\vect{u}_3$
    where
    $a_1 =
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}
    = \frac{1}{2}$. So the first coordinate is $\frac{1}{2}$.
  \end{sol}
\end{ex}


\subsection{Metrics}

We defined a norm as:

\(||v||=v^TMv\)

A metric is the distance between two vectors.

\(d(u,v)=||u-v||=(u-v)^TM(u-v)\)

\subsubsection{Metric space}

A set with a metric is a metric space.


\subsubsection{Second Derivative Test \& Hessian Matrix}
\noindent
Recall from single-variable calculus that if we have a critical point (derivative is 0), we can find whether the point is a minimum or maximum by using the second derivative test.
If the second derivative is negative at the critical point, then the critical point is a maximum, and if the second derivative is positive at the critical point, then the critical point is a minimum.
A similar process works in higher dimensions.\\

\noindent
In higher dimensions:
\begin{itemize}
	\item If $f_{xx}$ and $f_{yy}$ are positive at a critical point, then the critical point is a minimum.
	\item If $f_{xx}$ and $f_{yy}$ are negative at a critical point, then the critical point is a maximum.
	\item If $f_{xx}$ and $f_{yy}$ don't agree on sign, then the point is a saddle point.
\end{itemize}

\noindent
Although there are only 3 options for what a critical point on a surface can be, we need a process of identifying them that works for for higher dimensional objects too.

\begin{figure}[H]
	\centering
	\includegraphics*[width=0.9\textwidth]{./Images/differentialMultivariableCalculus/second_derivative_test.png}
	\caption{Three possibilities depending on signs of the seconds derivative}
\end{figure}

\begin{definition}
	The Hessian Matrix (for $f(x,y)$):
	\begin{equation*}
		H = \begin{bmatrix}
			f_{xx} & f_{xy} \\
			f_{yx} & f_{yy}
		\end{bmatrix}
	\end{equation*}
\end{definition}
\noindent
Note that $\det{H} = f_{xx}f_{yy} - f_{xy}^2$.\\

\noindent
If $(x_0, y_0)$ is a critical point:
\begin{itemize}
	\item $\det{H(x_0,y_0) > 0}$ means $(x_0, y_0)$ is an extrema.
	\begin{itemize}
		\item If $f_{xx}(x_0, y_0) > 0$, then $(x_0, y_0)$ is a minima.
		\item If $f_{xx}(x_0, y_0) < 0$, then $(x_0, y_0)$ is a maxima.
	\end{itemize}
	\item $\det{H(x_0, y_0)} < 0$ means $(x_0, y_0)$ is a saddle point.
	\item $\det{H(x_0, y_0)} = 0$ means the 2nd derivative test is inconclusive.
\end{itemize}


\section{09/07}
\begin{definition}{Random Variable}{}
    A \emph{random variable} is a function from some sample space $S$ to a set
    of numbers.
\end{definition}

\begin{definition}{Probability of a Random Variable}{}
    Let $X:\Omega \to E$ be some random variable and let $S \subseteq E$. Then
    \[\prob{X \in S} = \prob{\omega \in \Omega \suchthat X(\omega) \in S}\]
    If $S$ is a discrete probability space, then
    \[\prob{X = r} = \sum_{X(\omega) = r}\prob{\omega}\]
\end{definition}

For example, let us roll a fair die and say, should you roll $i$, you receive
$i^2$ dollars. We can define the random variable $X$ to be the money received in
this dice game. Then
\[\prob{X = 25} = \prob{\text{roll} = 5} = \frac{1}{6}\]

On the other hand, say you receive $i$ dollars (that is, the face of the die
roll). Then
\[\prob{X \equiv 0 \bmod{2}} = \prob{\text{die} \in \set{2, 4, 6}} = \frac{1}{2}\]

Suppose we additionally roll a second die and call the random variable equal to
the face-value of this die $Y$. What is $\prob{X + Y = 7}$? Say $R(i, j)$ is the
event that the first die rolls $i$ and the second rolls $j$. Then

\begin{align*}
    \prob{X + Y = 7} &= \prob{R(1, 6) \cup R(2, 5) \cup R(3, 4) \cup R(4, 3) \cup R(5, 2) \cup R(6, 1)}\\
                    &= \prob{R(1, 6)} + \prob{R(2, 5)} + \prob{R(3, 4)} + \prob{R(4, 3)} + \prob{R(5, 2)} + \prob{R(6, 1)}\\
                    &= \frac{6}{36}\\
                    &= \frac{1}{6}
\end{align*}

\subsection{Expected Value}
\begin{definition}{Expected Value}{}
    Let $X$ be a discrete random variable on a sample space $S$. The
    \emph{expected value} (also called mean or average) of $X$ is defined
    \[\expectation{X} = \sum_{r\in X(S)}r\prob{X = r}\]
\end{definition}
For example, if $X$ is the face-value of a fair die, then
\[\expectation{X} = \sum_{i=1}^6i\prob{X = i} = 1\cdot\frac{1}{6} + 1\cdot\frac{1}{6} + \dots + 6\cdot\frac{1}{6} = \frac{7}{2}\]
Notice that $\sfrac{7}{2}$ is \emph{not} an event in our sample space. In
general, the expectation does not need to be a value that can be ``achieved.''
Instead, the Law of Large Numbers formalizes the idea of what expectation is.

\begin{theorem}{Law of Large Numbers}{}
    Let $X_1$, $X_2$, \dots, $X_n$ denote the outcomes of $n$ independent
    experiments, and suppose that experiment has expected value $\mu$. Let
    $\overline{X}_n$ denote our \emph{sample mean}, i.e., write
    \[\overline{X}_n = \frac{1}{n}\sum_{i=1}^nX_i\]
    Then
    \[\lim_{n \to \infty}\prob{\overline{X}_n = \mu} = 1\]
\end{theorem}

Consider the following game: flip two fair coins. If both come up heads, win
\$2, otherwise lose \$1. Let $X$ denote the amount of money the player wins.
Then
\[\expectation{X} = 2 \cdot \frac{1}{4} + -1\cdot\frac{3}{4} = \frac{-1}{4}\]

We can define independence for random variables in the same way we did for
events.
\begin{definition}{Independent Random Variables}{}
    Two random variables $X$ and $Y$ on a sample space $S$ are independent if
    \[\prob{X = x \cap Y = y} = \prob{X = x}\prob{Y = y}\]
    for all $x \in X(S)$ and $y \in Y(S)$
\end{definition}

\subsection{Linearity of Expectation}
The following theorem proves very useful in analyzing randomized algorithms.
\begin{theorem}{Linearity of Expectation}{linexp}
    For any random variables $X$ and $Y$ and real value $c$
    \begin{align*}
        \expectation{X + Y} &= \expectation{X} + \expectation{Y}\\
        \expectation{cX} &= c\expectation{X}
    \end{align*}
\end{theorem}

\begin{proof}
    By definition
    \begin{align*}\expectation{X + Y}
        &= \sum_{x \in X(S)}\sum_{y \in Y(S)} (x + y)\prob{X = x \cap Y = y}\\
        &= \sum_{x \in X(S)}\sum_{y \in Y(S)} x\prob{X = x \cap Y = y} + \sum_{x \in X(S)}\sum_{y \in Y(S)}y\prob{X = x \cap Y = y}\\
        &=\sum_{x \in X(S)}\sum_{y \in Y(S)} x\prob{X = x \cap Y = y} + \sum_{y \in Y(S)}\sum_{x \in X(S)}y\prob{X = x \cap Y = y}\\
        &=\sum_{x \in X(S)}x\sum_{y \in Y(S)} \prob{X = x \cap Y = y} + \sum_{y \in Y(S)}y\sum_{x \in X(S)}\prob{X = x \cap Y = y}\\
        &=\sum_{x \in X(S)}x\prob{X = x} + \sum_{y \in Y(S)}y\prob{Y = y} \hbox{ by the law of total probability}\\
        &=\expectation{X} + \expectation{Y}
    \end{align*}
    Similarly,
    \begin{align*}\expectation{cX}
        &=\sum_{x \in X(S)}cx\prob{X = x}\\
        &=c\sum_{x \in X(S)}x\prob{X = x}\\
        &=c\expectation{X}\qedhere
    \end{align*}
\end{proof}

The power of \nameref{thm:linexp} can be seen by considering the following
scenario: suppose we flip $n$ fair coins and wish to count the expected number
of heads, $X$. Without \cref{thm:linexp}, this is
\[\expectation{X} = \sum_{i=0}^ni\binom{n}{i}\left(\frac{1}{2}\right)^n\]
a computationally difficult expression. On the other hand, if we let $X_i$
denote the following random variable
\[X_i = \begin{cases}1 & \hbox{coin $i$ lands heads}\\
                     0 & \hbox{otherwise}\end{cases}\]
It should be quite clear that
\[X = \sum_{i=1}^n X_i\]
Now,
\begin{align*}\expectation{X_i}
    &=1\prob{X_i = 1} + 0\prob{X_i = 0}\\
    &=\frac{1}{2}
\end{align*}
hence
\[\expectation{X} = \sum_{i=1}^n\expectation{X_i} = \frac{n}{2}\]
an almost \emph{trivial} computation.

The above random variable $X_i$ is called a Bernoulli or \emph{indicator} random
variable, since it \emph{indicates} whether the desired outcome occurred. In
general, if $X$ is an indicator random variable, then
\[\expectation{X} = \prob{X = 1}\]

We can similarly demonstrate the power of \nameref{thm:linexp} with the
following problem: suppose $n$ people arrive at a party and drop off their
coats, for which they receive a receipt. Unfortunately, the attendant loses all
of the stubs and decides to assign the party-goers coats at random. What is the
expected number of people who receive their original coat?

Let $X$ denote the number of people who receive their own coats. Then
\[\expectation{X} = \sum_{i=0}^n i \prob{X = i}\]
Notice that, for general $i$, this is impractical to determine. In fact, the
number $D_{n, k}$ is called a \emph{rencontres number}, and denote the number of
permutations of $\set{1, 2, \dots, n}$ with exactly $k$ fixed points. This can
be recursively computed with
\[D_{n, k} = \binom{n}{k}D_{n-k,0}\]
However, we can determine $\expectation{X}$ by applying \nameref{thm:linexp}.
Define
\[X_i = \begin{cases}1 & \hbox{person $i$ receives their own coat}\\
    0 & \hbox{otherwise}\end{cases}\]
and observe that $X = \sum X_i$. Additionally,
\[\expectation{X_i} = \prob{X_i = 1} = \frac{(n - 1)!}{n!} = \frac{1}{n}\]
from which it follows that
\[\expectation{X} = \sum_{i=1}^n \expectation{X_i} = n\frac{1}{n}=1\]

\subsection{3-SAT}
\begin{problem}{Boolean Satisfiability}{}
    Given a boolean formula, determine if there exists an assignment of the
    variables that causes the formula to evaluate to \True.
\end{problem}
A formula is in \emph{Conjunctive Normal Form} (CNF) if it can be written as a
disjunction of conjunctions, that is, a formula $B$ is in CNF if
\[B = T_1 \land T_2 \land \dots \land T_n\]
and each $T_i$ is of the form $t_1 \lor t_2 \lor \dots \lor t_i$. We say a
formula is $k$-CNF if each $T_i$ consists of exactly $k$ conjunctions.

It turns out that this problem is equivalent to the \emph{3-SAT} problem.
\begin{theorem}{}{}
    Given a boolean formula $B$, there is an \emph{equisatisfiable} formula $B'$
    in 3CNF whose length is at most 3 times that of $B$. That is, the formula
    $B$ is satisfiable if and only if $B'$ is satisfiable.
\end{theorem}
\begin{proof}
    Consider some clause of conjunctions $l_1 \lor l_2 \lor \dots \lor l_n$.
    The formula
    \begin{align*}
              & (l_1 \lor l_2 \lor x_2)\\
        \land & (\neg x_2 \lor l_3 \lor x_3)\\
        \land & (\neg x_3 \lor l_4 \lor x_4)\\
        \vdots &\\
        \land & (\neg x_{n-3} \lor l_{n-2} \lor x_{n-2})\\
        \land & (\neg x_{n-2} \lor l_{n-1} \lor l_n)
    \end{align*}
    is equisatisfiable to the clause.
\end{proof}

\begin{problem}{Max 3-SAT}{}
    Given a 3-SAT problem, find an assignment of the variables that
    \emph{maximizes} the number of clauses that evaluate to \True.
\end{problem}

Consider an algorithm where we simply assign each variable 0 or 1 at random. For
clause $i$, define
\[X_i = \begin{cases}1 & \hbox{clause $i$ is satisfied}\\
                     0 & \hbox{otherwise}\end{cases}\]
And set $X = \sum X_i$ to be the number of satisfied clauses. Suppose clause $i$
is $a \lor b \lor c$. Then
\begin{align*}\expectation{X_i}
    &= \prob{a \lor b \lor c}\\
    &= 1 - \prob{\neg a \land \neg b \land \neg c}\\
    &= 1 - \left(\frac{1}{2}\right)^3\\
    &= \frac{7}{8}
\end{align*}
Thus
\[\expectation{X} = \sum_{i=1}^m \expectation{X_i} = \frac{7m}{8}\]

\subsection{Probabilistic Method}
The \emph{Probabilistic Method} is a nonconstructive method to demonstrate the
existence of some sort of mathematical object. This is done by creating a
probability distribution on some sample space and showing that such an object
exists with probability greater than 0, from which you conclude that at least
one such object exists.

\begin{theorem}{}{}
    There is at least one assignment of variables that has $\sfrac{7m}{8}$ true
    clauses.
\end{theorem}

To see this, simply note that if $\expectation{X} = \mu$, there must be at least
one value in the sample space whose value is greater than or equal to $\mu$ and
at least one whose value is less than or equal to $\mu$.


\subsection{Complex numbers}

When Eigenmath starts up, it defines symbol $i$ as $i=\sqrt{-1}$.
Symbol $i$ can be redefined and used for some other purpose if need be.

\bigskip
\noindent
Complex quantities can be entered in either rectangular or polar form.

{\color{blue}
\begin{verbatim}
a + i b
\end{verbatim}
}

\noindent
$a+ib$

{\color{blue}
\begin{verbatim}
exp(1/3 i pi)
\end{verbatim}
}

\noindent
$\exp\left(\tfrac{1}{3}i\pi\right)$

\bigskip
\noindent
Converting a complex number to rectangular or polar coordinates causes
simplification of mixed forms.

{\color{blue}
\begin{verbatim}
A = 1 + i
B = sqrt(2) exp(1/4 i pi)
A - B
\end{verbatim}
}

\noindent
$1+i-2^{1/2}\exp\left(\tfrac{1}{4}i\pi\right)$

{\color{blue}
\begin{verbatim}
rect(last)
\end{verbatim}
}

\noindent
$0$

\bigskip
\noindent
Rectangular complex quantities, when raised to a power, are multiplied out.

{\color{blue}
\begin{verbatim}
(a + i b)^2
\end{verbatim}
}

\noindent
$a^2-b^2+2iab$

\bigskip
\noindent
When $a$ and $b$ are numerical and the power is negative, the evaluation is done as follows.
\begin{equation*}
(a+ib)^{-n}
=\left(\frac{a-ib}{(a+ib)(a-ib)}\right)^n=
\left(\frac{a-ib}{a^2+b^2}\right)^n
\end{equation*}

\noindent
Here are a few examples.

{\color{blue}
\begin{verbatim}
1/(2 - i)
\end{verbatim}
}

\noindent
$\tfrac{2}{5}+\frac{1}{5}i$

{\color{blue}
\begin{verbatim}
(-1 + 3 i)/(2 - i)
\end{verbatim}
}

\noindent
$-1+i$

\bigskip
\noindent
The absolute value of a complex number returns its magnitude.

{\color{blue}
\begin{verbatim}
abs(3 + 4 i)
\end{verbatim}
}

\noindent
$5$

\bigskip
\noindent
The imaginary unit can be changed from $i$ to $j$
by defining $j=\sqrt{-1}$.

{\color{blue}
\begin{verbatim}
j = sqrt(-1)
sqrt(-4)
\end{verbatim}
}

\noindent
$\displaystyle 2j$


\input{../common/common.tex}

\title{Math notes - Fibolucci}
\author{Uwe Hoffmann}
\hypersetup{colorlinks, pdftitle={Math notes - Fibolucci}}

\begin{document}

\setcounter{chapter}{1}
\section*{Fibolucci}

\newthought{Exercise `Fibolucci'} in \textit{Programming, The Derivation of Algorithms}\cite{Kaldewaij90}.\index{Fibolucci}

\vspace{10 mm}
\begin{problem}
Write a program that calculates the function

\begin{equation*}
f(n) = \sum_{i = 0}^{n} \mathit{fib}(i) \mathit{fib}(n - i)  \text{, for } n \geq 0
\end{equation*}
where $\mathit{fib}$ is the Fibonacci sequence defined by:
\begin{equation*}
\begin{split}
& \mathit{fib}(0) = 0 \\
& \mathit{fib}(1) = 1 \\
& \mathit{fib}(n + 2) = \mathit{fib}(n + 1) + \mathit{fib}(n) \text{, for } n \geq 0
\end{split}
\end{equation*}

\end{problem}

To solve the Fibolucci sum we adopt the same notation used in \textit{Programming in the 1990s}\cite{Cohen90}: The notation of function application is the "dot" notation with name of function, followed by arguments, each separated by a dot. The notation of quantified expressions has the operator followed by the bounded variables, then a colon followed by the range for the bounded variables and ended with a colon and the actual expression. So

\[
	(\sum k : i \leq k < j : x_k)
\]

\noindent corresponds to the more classical mathematical notation $\sum_{k = i}^{ j - 1}x_k$.

For our derivation steps in predicate calculus we will use the following notation:

\[
\begin{array}{lcl}
		&&A \\
	      &=& { < \mbox{reason why A equals B } >} \\
                  &&B \\
                &\leq& { < \mbox{reason why B is less than C} >} \\
                  && C
   \end{array}
\]

We start by finding a recursive expression for $f$. We will use properties of quantified expressions as covered in Chapter 3 of \textit{Programming in the 1990s}\cite{Cohen90}. Since $ \mathit{fib}.(0) = 0$ we can use an equivalent definition expression for $f$:
\[
f(n) = (\sum i: 1 \leq i < n: \mathit{fib}.i\  \mathit{fib}.(n - i))
\]

We derive:
\[
\begin{array}{lcl}
		&&f.(n + 2) \\
	      &=& { < \text{definition of  } f\ >} \\
                  && (\sum i: 1 \leq i < n + 2: \mathit{fib}.i\  \mathit{fib}.(n + 2 - i))\\
                &=& { < \text{range split, 1-point rule} >} \\
                  &&  (\sum i: 1 \leq i < n + 1: \mathit{fib}.i\  \mathit{fib}.(n + 2 - i)) + \mathit{fib}.(n+ 1)\  \mathit{fib}.(1) \\
                &=& { <\  \mathit{fib}.(1) = 1\  >} \\
                   && (\sum i: 1 \leq i < n + 1: \mathit{fib}.i\  \mathit{fib}.(n + 2 - i)) + \mathit{fib}.(n+ 1)\\
	       &=& { < \text{definition of } \mathit{fib}\ >} \\
	       	&& (\sum i: 1 \leq i < n + 1: \mathit{fib}.i\  (\mathit{fib}.(n + 1 - i) + \mathit{fib}.(n - i))) + \mathit{fib}.(n+ 1)\\
	      &=& { < \text{splitting the term} >} \\
	       	&& (\sum i: 1 \leq i < n + 1: \mathit{fib}.i\  \mathit{fib}.(n + 1 - i))\ + \\
		&& (\sum i: 1 \leq i < n + 1: \mathit{fib}.i\  \mathit{fib}.(n - i)) + \mathit{fib}.(n+ 1)\\	      &=& { < \text{definition of } f\ >} \\
	       	&& f.(n + 1) +  (\sum i: 1 \leq i < n + 1: \mathit{fib}.i\  \mathit{fib}.(n - i)) + \mathit{fib}.(n+ 1) \\
	      &=& { < \text{range split, 1-point rule, } \mathit{fib}.(0) = 0\  >} \\
                  && f.(n + 1) +  (\sum i: 1 \leq i < n: \mathit{fib}.i\  \mathit{fib}.(n - i)) + \mathit{fib}.(n+ 1) \\
               &=& { < \text{definition of } f\ >} \\
	       	&& f.(n + 1) +  f.n + \mathit{fib}.(n+ 1)
   \end{array}
\]

We get the recursive definition of $f$:
\begin{equation*}
\begin{split}
& f.0 = 0 \\
& f.1 = 0 \\
& f.(n + 2) = \mathit{fib}.(n + 1) + f.(n + 1) + f.n \text{, for } n \geq 0
\end{split}
\end{equation*}

It is straightforward to write a program that computes $f$ from this recursive definition, either iteratively with a loop that step by step computes next values of $f$ starting with $f(2)$ and remembering the last two computed values of $f$ and of $\mathit{fib}$ for the next computations, or in Haskell by simply declaring the above recursions for $f$ and $\mathit{fib}$. This will lead to a runtime of $O(n)$. But can we do better than linear ?

Let's look again at the recursive expressions of the two functions involved, leaving out the base cases and computing one additional next value:

\begin{equation*}
\begin{split}
& f.(n + 2) = \mathit{fib}.(n + 1) + f.(n + 1) + f.n \\
& f.(n + 3) = \mathit{fib}.(n + 2) + f.(n + 2) + f.(n + 1) \\
& \mathit{fib}.(n + 2) = \mathit{fib}.(n + 1) + \mathit{fib}.n \\
& \mathit{fib}.(n + 3) = \mathit{fib}.(n + 2) + \mathit{fib}.(n + 1)
\end{split}
\end{equation*}

The key observation we can make here is that new values of the two functions are linear combinations of previously computed values. Linear combinations implies linear applications with matrix representations from linear algebra. How many previously computed values, i.e. how far back do we need to go: we need the last computed value last and the value computed before that, so 2 previous values. Looks like we could try something in a linear space of dimension 2.

Let's try first with $\mathit{fib}$ which is simpler and doesn't depend on $f$. We define the function $\mathit{Fib}: \mathbb{N}  \rightarrow \mathbb{N}^2$ into the two-dimensional space $\mathbb{N}^2$:

\[
  \mathit{Fib}.n =
  \begin{pmatrix}
       \mathit{fib}.n \\
       \mathit{fib}.(n + 1)
   \end{pmatrix}   \text{, for } n \geq 0
\]

For a recursive expression for $\mathit{Fib}$ we have:
\[
\begin{array}{lcl}
  &&\mathit{Fib}.(n + 1) \\
   &=& { < \text{definition of  } \mathit{Fib}\ >} \\
  &&\begin{pmatrix}
       \mathit{fib}.(n + 1) \\
       \mathit{fib}.(n + 2)
   \end{pmatrix} \\
   &=& { < \text{definition of  } \mathit{fib}\ >} \\
   &&\begin{pmatrix}
       \mathit{fib}.(n + 1) \\
       \mathit{fib}.(n + 1) + \mathit{fib}.n
   \end{pmatrix} \\
   &=& { < \text{matrix multiplication}  >} \\
   &&\begin{pmatrix}
       0 & 1 \\
       1 & 1
   \end{pmatrix}
   \begin{pmatrix}
        \mathit{fib}.n \\
       \mathit{fib}.(n + 1)
   \end{pmatrix} \\
   &=& { < \text{definition of  } \mathit{Fib}\ >} \\
   &&\begin{pmatrix}
       0 & 1 \\
       1 & 1
   \end{pmatrix} \mathit{Fib}.n
  \end{array}
\]

So
\[
  \mathit{Fib}.(n + 1) =
  \begin{pmatrix}
       0 & 1 \\
       1 & 1
   \end{pmatrix}   \mathit{Fib}.n = \ldots =
   \begin{pmatrix}
       0 & 1 \\
       1 & 1
   \end{pmatrix} ^{n + 1}  \mathit{Fib}.0
\]

The same approach can be used for $f$. We define a function $F: \mathbb{N}  \rightarrow \mathbb{N}^4$ into the four-dimensional space $\mathbb{N}^4$:

\[
  F.n =
  \begin{pmatrix}
       \mathit{fib}.n \\
       \mathit{fib}.(n + 1) \\
       f.n \\
       f.(n + 1)
   \end{pmatrix}   \text{, for } n \geq 0
\]

For a recursive expression for $F$ we have:
\[
\begin{array}{lcl}
  &&F.(n + 1) \\
   &=& { < \text{definition of  } F\ >} \\
  &&\begin{pmatrix}
       \mathit{fib}.(n + 1) \\
       \mathit{fib}.(n + 2) \\
       f.(n + 1) \\
       f.(n + 2)
   \end{pmatrix} \\
   &=& { < \text{definitions of  } \mathit{fib} \text{ and } f \ >} \\
   &&\begin{pmatrix}
      \mathit{fib}.(n + 1) \\
       \mathit{fib}.(n + 1) + \mathit{fib}.n\\
       f.(n + 1) \\
       f.(n + 1) + f.n + \mathit{fib}.(n + 1)
   \end{pmatrix} \\
   &=& { < \text{matrix multiplication}  >} \\
   &&\begin{pmatrix}
       0 & 1 & 0 & 0\\
       1 & 1 & 0 & 0\\
       0 & 0 & 0 & 1\\
       0 & 1 & 1 & 1
   \end{pmatrix}
   \begin{pmatrix}
         \mathit{fib}.n \\
       \mathit{fib}.(n + 1) \\
       f.n \\
       f.(n + 1)
   \end{pmatrix} \\
   &=& { < \text{definition of  } F\ >} \\
   &&\begin{pmatrix}
       0 & 1 & 0 & 0\\
       1 & 1 & 0 & 0\\
       0 & 0 & 0 & 1\\
       0 & 1 & 1 & 1
   \end{pmatrix} F.n
  \end{array}
\]
and
\[
  F.(n + 1) =
  \begin{pmatrix}
        0 & 1 & 0 & 0\\
       1 & 1 & 0 & 0\\
       0 & 0 & 0 & 1\\
       0 & 1 & 1 & 1
   \end{pmatrix}   F.n = \ldots =
   \begin{pmatrix}
       0 & 1 & 0 & 0\\
       1 & 1 & 0 & 0\\
       0 & 0 & 0 & 1\\
       0 & 1 & 1 & 1
   \end{pmatrix} ^{n + 1} F.0
\]

Calculating $F.n$ also calculates $f.n$ so if we can calculate $F.n$ faster than linear we also solve the original problem faster than linear. $F.n$ is basically an exponentiation so let's look at the exponentiation function $exp(x, n) = x^n$. The following recursive expression holds for $exp$:

\begin{equation*}
 exp.x.n =
 \begin{cases}
 exp. (x \ x).(n / 2) & \text{if } n = 0 \text{ mod } 2 \\
 x \ exp.x.(n - 1) & \text{if }n = 1 \text{ mod } 2
 \end{cases}
\end{equation*}

At least at every other step in the above recursion n is halved so computing $exp(x, n)$ has $O(log\ n)$ runtime which also implies $O(log\ n)$ runtime for $F$.

Before we write the actual code for computing $F$ let's first see if we can find a more compact representation for the powers of matrix $A$ involved in the computation:

\begin{equation*}
A =
\begin{pmatrix}
       0 & 1 & 0 & 0\\
       1 & 1 & 0 & 0\\
       0 & 0 & 0 & 1\\
       0 & 1 & 1 & 1
       \end{pmatrix}
\end{equation*}

We are searching for patterns in the powers of $A$:
\begin{equation*}
A^2 =
\begin{pmatrix}
       1 & 1 & 0 & 0\\
       1 & 2 & 0 & 0\\
       0 & 1 & 1 & 1\\
       1 & 2 & 1 & 2
       \end{pmatrix},\ A^3 =
 \begin{pmatrix}
       1 & 2 & 0 & 0\\
       2 & 3 & 0 & 0\\
       1 & 2 & 1 & 2\\
       2 & 5 & 2 & 3
       \end{pmatrix},\ A^4 =
 \begin{pmatrix}
       2 & 3 & 0 & 0\\
       3 & 5& 0 & 0\\
       2 & 5 & 2 & 3\\
       5 & 10 & 3 & 5
       \end{pmatrix}
\end{equation*}

We make the conjecture that $A^k$ for any natural $k$ is of the form:

\begin{equation}\label{tuplerep}
A^k =
\begin{pmatrix}
       a & b & 0 & 0\\
       b & a + b & 0 & 0\\
       c & d & a & b\\
       e & f & b & a + b
       \end{pmatrix}, \text{ for some } a, b, c, d, e, f \in \mathbb{N}
\end{equation}

and prove this by induction. The base case for $k = 1$ is established with values $(0, 1, 0 , 0, 0 ,1)$ for $(a, b, c, d, e, f)$. Assuming that the conjecture holds for $A^k$ we look at $A^{k + 1}$ and get:

\begin{equation*}
A^{k + 1} = A^k\ A =
\begin{pmatrix}
       b & a + b & 0 & 0\\
       a + b & a + 2 b & 0 & 0\\
       d & b + c + d & b & a + b\\
       f & a + b + e + f & a + b & a + 2 b
       \end{pmatrix}
\end{equation*}

so $A^{k + 1}$ has the same form as stated in the conjecture if we substitute $(b, a + b, d, b + c + d, f, a + b + e + f)$  for $(a, b, c, d, e, f)$. This proves conjecture (\ref{tuplerep}).

It means that in our program we can use a tuple representation $(a, b, c, d, e, f)$ of 6 values instead of the whole 16 values to represent the powers of $A$. We need to define multiplication in this tuple space consistent with the matrix multiplication:

\begin{equation*}
\begin{split}
(a, b, c, d, e, f) & (a', b', c', d', e',f') =  \\
( & a a' + b b', \\
   & a b' + b (a' + b'), \\
   & c a' + d b' + a c' + b e', \\
   & c b' + d (a' + b') + a d' + b f', \\
   & e a' + f b' + b c' + (a + b) e', \\
   & e b' + f (a' + b') + b d' + (a + b) f' )
\end{split}
\end{equation*}

We read this definition off the matrix multiplication:

\begin{equation*}
\begin{pmatrix}
       a & b & 0 & 0\\
       b & a + b & 0 & 0\\
       c & d & a & b\\
       e & f & b & a + b
       \end{pmatrix}
  \begin{pmatrix}
       a' & b' & 0 & 0\\
       b' & a' + b' & 0 & 0\\
       c' & d' & a' & b'\\
       e' & f' & b' & a' + b'
       \end{pmatrix}
\end{equation*}

The last expression we need is:

\begin{equation*}
A^n F.0 = \begin{pmatrix}
       a & b & 0 & 0\\
       b & a + b & 0 & 0\\
       c & d & a & b\\
       e & f & b & a + b
       \end{pmatrix}
 \begin{pmatrix}
       0\\
       1\\
       0\\
       0
       \end{pmatrix}   =
    \begin{pmatrix}
       b\\
       a + b\\
       d\\
        f
       \end{pmatrix}
\end{equation*}

so we are interested in $d$ which corresponds to the $F.n$ coordinate of the vector.

Putting all the pieces together we get the final Haskell program:

\lstinputlisting[language=Haskell, basicstyle=\small, frame=trBL, caption={Haskell code}]{fibolucci.hs}

\bibliographystyle{plainnat}
\bibliography{../common/math}

\end{document}


\chapter{Introduction}

\section{Gamma distribution}
Consider a number of events that follows a homogenous Poisson process with rate paramter $1/\beta$. Denote the time intervals between the events $X_1,...,X_\alpha$. The total time of the events are given by
\begin{equation*}
    X = X_1 + \cdots + X_\alpha.
\end{equation*}
Then $X$ is gamma distributed with parameters $(\alpha, \beta)$. The probability density function for the gamma distribution is given as follows.
\begin{equation*}
    f(x;\alpha,\beta) = \frac{x^{\alpha - 1}}{\Gamma(\alpha)\beta^\alpha} e^{\frac{-x}{\beta}}
\end{equation*}
The function $\Gamma(\alpha)$ is called the gamma function and is denoted
\begin{equation*}
    \Gamma(\alpha) = \int_{0}^{\infty} t^{\alpha - 1} e^{-t} dt.
\end{equation*}
Furthermore $\alpha>0$, $\beta>0$ and $X>0$.

\subsection{Sufficient statistics}

\subsection{Tranformation of variables}
For tranformation of variables in distribution the following theorem is presented.
\begin{theorem}
Let $X$ have a pdf $f_X(x)$ and let $Y = g(X)$, where $g$ is a monotone function. Let FIX s51 casella $X$ and $Y$. Suppose that $f_X(x)$ is continuous on $X$ and that
\begin{equation*}
f_Y (y) =
\begin{cases}
f_X(g^{-1}(y)) \left| \frac{d}{dy} g^{-1}(y) \right| & y \in \mathcal{Y} \\
0 & \text{otherwise.}
\end{cases}
\end{equation*}
\end{theorem}

\subsection{Generating sample}
The general setup is that we have data $X$ and sufficient statistcs $T$. An assumption is a random vector $U$ with known distribution. Furthermore assume functions $(\chi(\cdot, \cdot),\tau(\cdot, \cdot))$ such that
\begin{equation*}
    (\chi(U,\theta),\tau(U, \theta)) \overset{\theta}{\sim} (X,T).
\end{equation*}
For instance the vector $U$ could be from an uniform distribution between 0 and 1, while $\chi(U,\theta)$ is the inverse of the cumulative distribution and $\tau(U,\theta)$ is the statistics. For some instances this might give the wrong distribution to the samples. This is when $\tau(u, \theta)$ is not only dependent on $u$ in a function $r(u)$. To solve this weights $W_t(u)$ for the vector $U$ is proposed.

To simulate from a gamma distribution a trick is proposed. Let $X = \beta Y$ where$X ~ f(x;a,b)$. The trick is to simulate $Y$ and then find $X$. The distribution of $Y$ is found by using tranformation of variables. From transformation of variables we have that
\begin{align*}
g(x) &= \frac{x}{\beta} \\
g^{-1}(y) &= \beta y \\
\frac{dg^{-1}(y)}{dy} &= \beta \\
\end{align*}
Then $Y$ has the distribution
\begin{equation*}
    f(y;\alpha) = \frac{y^{\alpha - 1}}{\Gamma(\alpha)} e^{-y},
\end{equation*}
which is the same as the gamma distribution with $\beta = 1$. To generate a sample the cumulative distribution function must be found. A cumulative distribution function is found by
\begin{equation*}
    F(x) = \int_{-\infty}^{x} f(u) du.
\end{equation*}
For the gamma distribution the cumulative distribution function becomes
\begin{equation*}
    F(x;\alpha, \beta) = \frac{\gamma(x, \alpha, \beta)}{\Gamma(\alpha)},
\end{equation*}
where $\gamma$ is the incomplete gamma function. Hence for $Y$ the cumulative distribtution is
\begin{equation*}
    F(y;\alpha) = \frac{\gamma(y, \alpha, 1)}{\Gamma(\alpha)}.
\end{equation*}
\section{Goodness of fit testing}


\section*{Summary of Section 1}
\begin{itemize}
  \item In higher dimensions,
      \begin{enumerate}
        \item a line passing through the origin is described as
        \begin{equation*}
          L = \{ c \vec{v} : c \in \bbr \}
        \end{equation*}
        \item a plane passing through the origin is described as
        \begin{equation*}
          P = \{ c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr \}
        \end{equation*}
        when $\vec{v}_1$ is not a scalar multiple of $\vec{v}_2$.
      \end{enumerate}
    We generalize this using the notion of $\spn$. $L$ is the span of the set $\{\vec{v}\}$ and $P$ is the span of the set $\{ \vec{v}_1, \vec{v}_2 \}$.

    \item More generally, for any subset $\cals \subset \bbr^n$
    \begin{align*}
      \spn(\cals) = \{ c_1 \vec{v}_1 + \dots + c_k \vec{v}_k \quad : \quad
      & c_1, \dots, c_k \mbox{ are real numbers}, \\
      & \vec{v}_1, \dots, \vec{v}_k \mbox{ are vectors in } S \}
    \end{align*}
    defines a subspace of $\bbr^n$. A subspace is a subset of $\bbr^n$ which is closed under scalar multiplication and addition.

    However, there might be reduncacies in $\cals$. For example,
    \begin{align*}
      \spn(\vec{v}_1) &= \spn(\vec{v}_1, \vec{v}_2)
      && \mbox{ if $\vec{v}_1$ is a scalar multiple of $\vec{v}_2$} \\
      \spn(\vec{e}_1 - \vec{e}_2, \vec{e}_2 - \vec{e}_3, \vec{e}_3 - \vec{e}_1) &= \spn(\vec{e}_1 - \vec{e}_2, \vec{e}_2 - \vec{e}_3)
    \end{align*}

    \item This is ``corrected'' by linear independence (Definition \ref{def:linearIndependence}). For a subspace $V$ of $\bbr^n$, a set $\calb \subseteq V$ is said to be a \emph{basis} of $V$ if $\spn(\calb) = V$ and ${\calb}$ is linearly independent.

    In the above examples, $L$ has basis $\calb = \{ \vec{v} \}$ and $P$ has basis $\{ \vec{v}_1, \vec{v}_2 \}$.

    \item A basis allows us to define coordinates for a vector space.
    \begin{enumerate}
      \item For $L$ as above, with $\calb = \{ \vec{v} \}$
      \begin{equation*}
        [c\vec{v}]_{\calb} = [ c ]
      \end{equation*}
      \item For $P$ as above, with $\calb = \{ \vec{v}_1, \vec{v}_2\}$
      \begin{equation*}
        [c_1 \vec{v}_1 + c_2 \vec{v}_2]_{\calb} = \begin{bmatrix}
          c_1 \\ c_2
        \end{bmatrix}
      \end{equation*}
      \item More generally, if $V$ has basis $\calb = \{ \vec{v}_1, \dots, \vec{v}_k\}$ then
      \begin{equation*}
        [c_1 \vec{v}_1 + \dots + c_k \vec{v}_k]_{\calb} = \begin{bmatrix}
          c_1 \\ \vdots \\c_k
        \end{bmatrix}
      \end{equation*}
    \end{enumerate}
\end{itemize}

\begin{figure}
  \centering
  \begin{tikzpicture}[
     >=latex,
     % font=\footnotesize,
     x={(1cm, 1cm)},
     y={(1cm, -.5cm)},
   ]
     \def\xmin{-2}
     \def\xmax{4}
     \def\ymin{-2}
     \def\ymax{4}
     \draw[thin, dashed]
       \foreach \x in {\xmin, ..., \xmax} {
         (\x, \ymin) -- (\x, \ymax)
       }
       \foreach \y in {\ymin, ..., \ymax} {
         (\xmin, \y) -- (\xmax, \y)
       }
     ;
     \draw[<->, very thick]
       (1, 0) node[above] {$\vec{v}_1$}
       -- (0, 0)
       -- (0, 1) node[below] {$\vec{v}_2$}
     ;
     \draw[->]
       (0, 0)
       -- (2, 3) node[above] {$\begin{bmatrix} 2 \\ 3 \end{bmatrix}$}
     ;
   \end{tikzpicture}
  \caption{Generalized coordinates on a plane. In the basis $\calb=\{ \vec{v}_1, \vec{v}_2 \}$, the vector $\vec{v} = 2 \vec{v}_1 + 3 \vec{v}_2$ has coordinates $[\vec{v}]_{\calb} = \protect\begin{bmatrix} 2 \\ 3 \protect\end{bmatrix}$.}
\end{figure}


\section*{Exercises}

\begin{ex}
  The following are augmented matrices for four systems of
  equations. Determine which of them, if any, are row equivalent.
  \begin{equation*}
    \begin{mymatrix}{rrrr|r}
       1 & 3 &  5 &  1 & 12 \\
      -1 & 1 & -1 &  2 &  5 \\
       2 & 0 &  4 & -2 &  0
    \end{mymatrix}
    \quad
    \begin{mymatrix}{rrrr|r}
      1 & 2 & 4 & -1 &  8 \\
      2 & 4 & 3 &  1 & 15 \\
      3 & 6 & 1 & -1 &  8
    \end{mymatrix}
    \quad
    \begin{mymatrix}{rrrr|r}
      3 & 6 & -3 & 1 & 6 \\
      2 & 4 & 2 & 1 & 13 \\
      0 & 0 & 1 & 0 & 2
    \end{mymatrix}
    \quad
    \begin{mymatrix}{rrrr|r}
      1 & 2 & 4 & 1 & 10 \\
      0 & 1 & 1 & 1 & 5 \\
      2 & 1 & 5 & 0 & 8
    \end{mymatrix}
  \end{equation*}

% The {\rref}s are supposed to be:
% \mat{
%    \begin{array}{rrrr|r}
%      1 & 0 & 2 & 0 & 3 \\
%      0 & 1 & 1 & 0 & 2 \\
%      0 & 0 & 0 & 1 & 3
%    \end{array}
%    }
%    \quad
%    \mat{
%    \begin{array}{rrrr|r}
%      1 & 2 & 0 & 0 & 3 \\
%      0 & 0 & 1 & 0 & 2 \\
%      0 & 0 & 0 & 1 & 3
%    \end{array}
%    }
%    \quad

\end{ex}

\begin{ex}
  Find a tuple $(x,y,z)$ that is a solution to one system of equations
  but not the other.
  \begin{equation*}
    \begin{mymatrix}{rrrr|r}
      1 & 0 & 1 & 5 & 0 \\
      2 & 1 & 4 & 3 & 0 \\
      3 & 1 & 5 & 9 & 0 \\
    \end{mymatrix}
    \quad
    \begin{mymatrix}{rrrr|r}
      1 & 0 & 3 & 6 & 0 \\
      2 & 1 & 8 & 2 & 0 \\
      2 & 0 & 6 & 13 & 0 \\
    \end{mymatrix}
  \end{equation*}
\end{ex}


\subsection{Fourier transforms}

\subsubsection{Taylor series of matrices}

We can also use Taylor series to evaluate functions of matrices.

Consider \(e^M\)

We can evaluate this as:

\(e^M=\sum_{k=0}^\infty \dfrac{1}{k!}M^k\)


\subsection{Algebraic definition of the cross product}

From its geometric description, we can prove that the cross product
satisfies the following properties.

\begin{proposition}{Properties of the cross product}{properties-cross-product}
  Let $\vect{u}, \vect{v}, \vect{w}$ be vectors in $\R^3$, and $k$ a
  scalar. Then the following hold.%
  \index{properties of cross product}%
  \index{cross product!properties}%
  \index{vector!cross product!properties}%
  \index{vector!properties of cross product}%
  \begin{enumerate}
  \item
    $\vect{u}\times \vect{v}= -(\vect{v}\times \vect{u})$.
  \item $\vect{u}\times \vect{u}=\vect{0}$.
  \item $(k \vect{u})\times \vect{v}= k (\vect{u}\times \vect{v})
    =\vect{u}\times (k \vect{v})$.
  \item $\vect{u}\times (\vect{v}+\vect{w}) =\vect{u}\times \vect{v}+\vect{u}\times \vect{w}$.
  \item $(\vect{v}+\vect{w}) \times \vect{u}=\vect{v} \times \vect{u}+\vect{w}\times \vect{u}$.
  \end{enumerate}
\end{proposition}

\begin{proof}
  Formula $1$. follows immediately from the definition. The vectors
  $\vect{u}\times \vect{v}$ and $\vect{v}\times \vect{u}$ have the
  same magnitude, $\norm{\vect{u}}\norm{\vect{v}}\sin \theta$, and an
  application of the right hand rule shows they have opposite
  direction.

  Formula $2$. is proven as follows. If $k$ is a non-negative scalar,
  the direction of $(k \vect{u}) \times \vect{v}$ is the same as
  the direction of
  $\vect{u}\times \vect{v}, k (\vect{u}\times \vect{v})$ and
  $\vect{u}\times (k \vect{v})$. The magnitude is $k$ times the
  magnitude of $\vect{u}\times \vect{v}$ which is the same as the
  magnitude of $k (\vect{u}\times \vect{v})$ and
  $\vect{u}\times (k \vect{v})$. Using this yields equality in
  $2$. In the case where $k <0$, everything works the same way except
  the vectors are all pointing in the opposite direction and you must
  multiply by $\abs{k}$ when comparing their magnitudes.

  The distributive laws, $3$. and $4$., are harder to establish. For
  now, we will content ourselves with noticing that if we know that
  $3$. is true, $4$. follows. Namely, assuming $3$., and using $1$.,
  we have
  \begin{align*}
    (\vect{v}+\vect{w}) \times \vect{u}
    & =-\vect{u}\times (
      \vect{v}+\vect{w}) \\
    & =-(\vect{u}\times \vect{v}+\vect{u}\times \vect{w}) \\
    & =\vect{v}\times \vect{u}+\vect{w}\times \vect{u}.
  \end{align*}

  \vspace{-3ex}
\end{proof}

In turn, we can use the properties from
Proposition~\ref{prop:properties-cross-product} to get an algebraic
description of the cross product. We begin by determining the cross
products of the special vectors $\vect{i}$, $\vect{j}$, and
$\vect{k}$. They are as follows:
\begin{equation*}
  \begin{array}{c@{\quad\quad}c@{\quad\quad}c}
    \vect{i}\times \vect{j}=\vect{k},
    & \vect{j}\times \vect{i}=-\vect{k},
    & \vect{i}\times \vect{i}=\vect{0}, \\
    \vect{k}\times \vect{i}=\vect{j},
    & \vect{i}\times \vect{k}=-\vect{j},
    & \vect{j}\times \vect{j}=\vect{0}, \\
    \vect{j}\times \vect{k}=\vect{i},
    & \vect{k}\times \vect{j}=-\vect{i},
    & \vect{k}\times \vect{k}=\vect{0}.
  \end{array}
\end{equation*}
With this information and the laws of
Proposition~\ref{prop:properties-cross-product}, we can compute the
cross product of any two vectors from their coordinates.%
\index{cross product!coordinate description} Let
\begin{equation*}
  \vect{u}=\begin{mymatrix}{c}u_1\\u_2\\u_3\end{mymatrix}
  \quad\mbox{and}\quad
  \vect{v}=\begin{mymatrix}{c}v_1\\v_2\\v_3\end{mymatrix}.
\end{equation*}
Then we have:
\begin{eqnarray*}
  \vect{u}\times\vect{v}
  &=& (u_1\vect{i}+u_2\vect{j}+u_3\vect{k})\times (v_1\vect{i}+v_2\vect{j}+v_3\vect{k}) \\
  &=& \begin{array}[t]{@{}c@{~}c@{~}c@{~}c@{~}c@{~}c}
        & u_1v_1(\vect{i}\times\vect{i})
        &+& u_1v_2(\vect{i}\times\vect{j})
        &+& u_1v_3(\vect{i}\times\vect{k}) \\
        +& u_2v_1(\vect{j}\times\vect{i})
        &+& u_2v_2(\vect{j}\times\vect{j})
        &+& u_2v_3(\vect{j}\times\vect{k}) \\
        +& u_3v_1(\vect{k}\times\vect{i})
        &+& u_3v_2(\vect{k}\times\vect{j})
        &+& u_3v_3(\vect{k}\times\vect{k})
      \end{array}
  \\
  &=&\begin{array}[t]{@{}c@{~}c@{~}c@{~}c@{~}c@{~}c}
       & u_1v_1\vect{0}
       &+& u_1v_2\vect{k}
       &-& u_1v_3\vect{j} \\
       -& u_2v_1\vect{k}
       &+& u_2v_2\vect{0}
       &+& u_2v_3\vect{i} \\
       +& u_3v_1\vect{j}
       &-& u_3v_2\vect{i}
       &+& u_3v_3\vect{0}
     \end{array}
  \\
  &=& (u_2v_3-u_3v_2) \vect{i}+
      (u_3v_1 - u_1v_3) \vect{j}+
      (u_1v_2-u_2v_1) \vect{k}.
\end{eqnarray*}
The resulting formula for the cross product is summarized in the
following Proposition.

\begin{proposition}{Coordinate description of cross product}{cross-product-coordinate}
  The cross product can be computed as follows:
  \begin{equation*}
    \begin{mymatrix}{c}u_1\\u_2\\u_3\end{mymatrix}
    \times
    \begin{mymatrix}{c}v_1\\v_2\\v_3\end{mymatrix}
    =
    \begin{mymatrix}{c}
      u_2v_3 - u_3v_2 \\
      u_3v_1 - u_1v_3 \\
      u_1v_2 - u_2v_1 \\
    \end{mymatrix}.
  \end{equation*}
\end{proposition}

We will now look at an example of how to compute a cross product.

\begin{example}{Find a cross product}{find-cross-product}
  Find $\vect{u} \times \vect{v}$ for the vectors
  \begin{equation*}
    \vect{u}
    =
    \begin{mymatrix}{r}
      1 \\
      -1 \\
      2
    \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}
    =
    \begin{mymatrix}{r}
      3 \\
      -2 \\
      1
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  Using Proposition~\ref{prop:cross-product-coordinate}, we compute
  \begin{equation*}
    \begin{mymatrix}{c}1\\-1\\2\end{mymatrix}
    \times
    \begin{mymatrix}{c}3\\-2\\1\end{mymatrix}
    =
    \begin{mymatrix}{c}
      (-1)(1) - (2)(-2) \\
      (2)(3) - (1)(1) \\
      (1)(-2) - (-1)(3) \\
    \end{mymatrix}
    =
    \begin{mymatrix}{c}
      3 \\
      5 \\
      1
    \end{mymatrix}.
  \end{equation*}
\end{solution}

We use this concept in the following examples.

\begin{example}{Area of a parallelogram}{area-parallelogram}
  Find the area of the parallelogram determined by the following
  vectors $\vect{u}$ and $\vect{v}$:
  \begin{equation*}
    \vect{u}
    =
    \begin{mymatrix}{r}
      1 \\
      -1 \\
      2
    \end{mymatrix}, \quad
    \vect{v}
    =
    \begin{mymatrix}{r}
      3 \\
      -2 \\
      1
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  Notice that these vectors are the same as the ones given in Example~\ref{exa:find-cross-product}.  Recall from the geometric description
  of the cross product that the area of the parallelogram is the
  magnitude of $\vect{u} \times \vect{v}$.  From
  Example~\ref{exa:find-cross-product},
  $\vect{u} \times \vect{v} = \mat{3,5,1}^T$.
  Thus the area of the parallelogram is
  \begin{equation*}
    \norm{\vect{u} \times \vect{v}} =
    \sqrt{3^2+5^2+1^2} = \sqrt{35}.
  \end{equation*}
\end{solution}

\begin{example}{Area of a parallelogram}{area-parallelogram2}
  Find the area of the parallelogram with vertices $(1,0,1)$,
  $(2,2,3)$, $(-1,1,3)$, and $(0,3,5)$.
\end{example}

\begin{solution}
  Let $P=(1,0,1)$, $Q=(2,2,3)$, $R=(-1,1,3)$, and $S=(0,3,5)$.
  \begin{center}
    \begin{tikzpicture}[scale=1]
      \fill[blue!5] (0,0) -- (2,2) -- (5,2) -- (3,0) -- cycle;
      \draw[->,thick,blue!70!black] (0,0) -- (2,2);
      \draw[->,thick,red] (0,0) -- (3,0);
      \draw[thick] (2,2)--(5,2);
      \draw[thick] (3,0)--(5,2);
      \draw[fill](0,0) circle [radius=2.25pt] node[left=1ex]{$P$};
      \draw[fill](2,2) circle [radius=2.25pt] node[left=1ex]{$Q$};
      \draw[fill](3,0) circle [radius=2.25pt] node[right=1ex]{$R$};
      \draw[fill](5,2) circle [radius=2.25pt] node[right=1ex]{$S$};
    \end{tikzpicture}
  \end{center}
  First, we check that this really is a parallelogram. We have to have
  $\longvect{PQ}=\longvect{RS}$. Indeed, this is the case, as
  $\longvect{PQ}=\mat{2-1,2-0,3-1}^T=\mat{1,2,2}^T$ and
  $\longvect{RS}=\mat{0-(-1),3-1,5-3}^T=\mat{1,2,2}^T$.  We also
  compute $\longvect{PR}=\longvect{QS}=\mat{-2,1,2}^T$. The
  area of the parallelogram is
  \begin{equation*}
    \norm{\longvect{PQ}\times\longvect{PR}}
    =
    \norm{
      \begin{mymatrix}{r} 1 \\ 2 \\ 2 \end{mymatrix}
      \times
      \begin{mymatrix}{r} -2 \\ 1 \\ 2 \end{mymatrix}
    }
    =
    \norm{\begin{mymatrix}{r} 2 \\ -6 \\ 5 \end{mymatrix}}
    = \sqrt{2^2 + (-6)^2 + 5^2}
    = \sqrt{65}.
  \end{equation*}
\end{solution}

We can also use this concept to find the area of a triangle, as in the
following example.

\begin{example}{Area of triangle}{area-triangle}
  \index{cross product!area of triangle}%
  \index{area!of triangle}%
  \index{triangle!area of}%
  Find the area of the triangle determined by the points
  $(1,2,3)$, $(0,2,5)$, and $(5,1,2)$.
\end{example}

\begin{solution}
  Let $P=(1,2,3)$, $Q=(0,2,5)$, and $R=(5,1,2)$. The area
  of the triangle is exactly half of the area of the parallelogram
  determined by the vectors $\longvect{PQ}$ and $\longvect{PR}$.
  \begin{center}
    \begin{tikzpicture}[scale=1]
      \fill[blue!5] (0,0) -- (2,2) -- (3,0) -- cycle;
      \draw[->,thick,blue!70!black] (0,0) -- (2,2);
      \draw[->,thick,red] (0,0) -- (3,0);
      \draw[thick] (2,2)--(5,2);
      \draw[thick] (3,0)--(5,2);
      \draw[dashed] (2,2)--(3,0);
      \draw[fill](0,0) circle [radius=2.25pt] node[left=1ex]{$P$};
      \draw[fill](2,2) circle [radius=2.25pt] node[left=1ex]{$Q$};
      \draw[fill](3,0) circle [radius=2.25pt] node[right=1ex]{$R$};
    \end{tikzpicture}
  \end{center}
  We have $\longvect{PQ}=\mat{-1,0,2}^T$ and
  $\longvect{PR}=\mat{4,-1,-1}^T$. The area of the parallelogram is
  the magnitude of the cross product:
  \begin{equation*}
    \norm{\longvect{PQ}\times\longvect{PR}}
    =
    \norm{
      \begin{mymatrix}{r} -1 \\ 0 \\ 2 \end{mymatrix}
      \times
      \begin{mymatrix}{r} 4 \\ -1 \\ -1 \end{mymatrix}
    }
    = \norm{\begin{mymatrix}{r} 2 \\ 7 \\ 1 \end{mymatrix}}
    = \sqrt{2^2+7^2+1^2}
    = \sqrt{54}.
  \end{equation*}
  Hence the area of the triangle is $\frac{1}{2}\sqrt{54}= \frac{3}{2}\sqrt{6}$.
\end{solution}

In general, the area of the triangle determined by three points
$P,Q,R$ in $\R^3$ is given by
\begin{equation*}
  \frac{1}{2}\norm{\longvect{PQ} \times  \longvect{PR}}.
\end{equation*}