\lab{Public Key Cryptography}{RSA}
\label{lab:RSA}

\objective{Implement the RSA cryptosystem as an example of public key cryptography.}

A \emph{public key cryptosystem} uses separate keys for encryption and decryption.
If Alice wishes to send Bob a message, she encrypts it using Bob's \emph{public key}, which is available to everyone.
Then Bob decrypts it with his \emph{private key}, which only he knows.
As long as it is difficult to find the private key from the public key, this is a secure system.

%Public key cryptosystems are advantageous because for $n$ people to have secure communications with each other, they need exactly $n$ public-private key pairs.
%A symmetric cryptosystem would require on the order of $n^2$ keys.
%Also, they eliminate the need for secure key sharing algorithms like the Diffie-Hellman key exchange (see Lab \ref{lab:DiffieHellman}).

As an analogy, consider a locked box with two kinds of keys.
One kind of key, called the public key, is available to anyone and can lock the box but not unlock it.
The other kind of key, called the private key, is only available to one person and can unlock the box.
To send a message to someone with the private key, the sender will put the message in the box and lock it.
Since the only person that can unlock the box is the recipient, the message is safe until it arrives. This is how public key cryptography works.

One of the oldest and most popular public key cryptosystems is RSA cryptography.

%All material in this lab through the section ``Security of RSA'' is required.
%After that, all sections and subsections are independent of each other and may be covered as desired.

\section*{The RSA system}
Suppose Alice wants to receive secret messages using RSA.
To do so, she first needs to generate a public key (for encryption) and a private key (for decryption).
Alice does this by choosing two prime numbers, $p$ and $q$, and setting $n=pq$.
Then she sets $\phi(n) = (p-1)(q-1)$.\footnote{
The function $\phi: \mathbb{Z} \rightarrow \mathbb{N}$ is called the \emph{Euler phi function}. In general $\phi(n)$ is the number of positive integers less than $n$ that are relatively prime to $n$.}
Alice chooses her encryption exponent to be some integer $e$ that is relatively prime to $\phi(n)$.
Then she uses the Extended Euclidean Algorithm to find $d'$ such that $ed' + \phi(n)x' = 1$, and adds or subtracts multiples of $\phi(n)$ until $d = d'+k\phi(n)$ is between 0 and $\phi(n)$.
Alice publishes her public key $(e, n)$ for the world to see and keeps her private key $(d,n)$ a secret ($e$ for encryption, $d$ for decryption).

Now imagine Bob wants to send Alice a message, which he represents as an integer $m<n$ (say, using the A=01, B=02 scheme).
Bob computes
\[
c \equiv m^e \pmod{n}
\]
and sends the ciphertext $c$ to Alice.

To decrypt the message, Alice computes
\[
m' \equiv c^d \pmod{n}.
\]
At this point, she uses the following theorem, which is easily proved with a combination of ring and group theory.
\begin{theorem}
For any integers $m$ and $n$ such that $n$ does not divide $m$, the following equality holds:
\[
m^{\phi(n)}\equiv 1 \pmod{n}.
\]
\end{theorem}
Then Alice concludes that
\[
m' = c^d \equiv m^{ed} = m^{\phi(n)(ek-x')+1} \equiv m \pmod{n}.
\]
Now Alice can read Bob's message.

As an example, let us encrypt and decrypt the message SECRET=190503180520.
First we define $p$, $q$, $n$, and $\phi(n)$.
\begin{lstlisting}
>>> p = 1000003
>>> q = 608609
>>> n = p*q
>>> phi_n = (p-1)*(q-1)
\end{lstlisting}

Now we choose an encryption exponent $e = 1234567891$ and compute $d = 589492582555$ using the Extended Euclidean Algorithm.
\begin{lstlisting}
>>> e = 1234567891
>>> d = 589492582555
\end{lstlisting}

Finally we are ready to encrypt the message.
Note that $m<n$. If this were not the case, we would need to break up $m$ into shorter pieces.
Also, we force $m$ to be a \li{long} integer so that the exponentiation operation does not overflow.
The function \li{pow(a, b, n)} computes $a^b \pmod{n}$.
\begin{lstlisting}
>>> m = long(190503180520)
>>> c = pow(m, e, n)
\end{lstlisting}

We decrypt the message by raising $c$ to the $d^{th}$ power, modulo $n$.
\begin{lstlisting}
>>> m == pow(c, d, n)
True
\end{lstlisting}

\subsection*{Logistical considerations}
The cryptosystem described is not particularly easy to use, since the message must be converted to an integer and back again by hand.
The provided \li{rsa_tools} module contains some functions to fix this problem.
The function \li{string_to_int()} turns any string into an integer (using a mapping more complicated than $A=01, B=02$), and the function \li{int_to_string()} changes it back again.

\begin{lstlisting}
>>> import rsa_tools as r
>>> r.string_to_int('SECRET')
91556947314004
>>> r.int_to_string(91556947314004)
'SECRET'
\end{lstlisting}

At this point we have a problem, because the message 91556947314004 is larger than $n=608610825827$.
We can only use RSA to encrypt messages that are smaller than $n$.
In fact, the function \li{string_size()} provided in \li{rsa_tools} tells us the maximum number of characters we can encrypt with this choice of $n$.

\begin{lstlisting}
>>> r.string_size(608610825827)
4
\end{lstlisting}

The function \li{partition()}, also in the \li{rsa_tools} will break our message into pieces of length 4.
We specify the ``fill value'' \li{x} that will be used to make all pieces the same length.

\begin{lstlisting}
>>> r.partition('SECRET', 4, 'x')
['SECR', 'ETxx']
\end{lstlisting}
Now we can proceed to encrypt and decrypt the strings \li{'SECR'} and \li{'ETxx'} as before.

% Problem 1: Implement the 'myRSA' class.
\begin{problem}
Write a class called \li{myRSA} that can generate keys, encrypt messages, and decrypt messages.
Use methods from the provided \li{rsa_tools.py} module to convert strings to ints and vice-versa.

Write a method called \li{generate_keys} that accepts a pair of primes and an encryption exponent and sets the \li{public_key} and \li{_private_key} attributes.
(Starting \li{_private_key} with an underscore hides the attribute from the user.)
Also include an \li{encrypt} method that accepts a string and encrypts it, using \li{public_key} as the default encryption key.
If a different public key is provided, then the message should be encrypted with the provided key.
Finally, write a \li{decrypt} method that decrypts an encrypted message with \li{_private_key}.

In the next problem we will write a test function for this class.
\label{prob:rsa1}
\end{problem}

\subsection*{Security of RSA}
Suppose an enemy Eve wants to read the message that Bob sent to Alice.
Like Bob, she has access to Alice's public key $(e, n)$.
Let's assume she has also intercepted the ciphertext $c$.

One way for Eve to read Bob's message is to directly find $m$ such that $m^e \equiv c \pmod{n}$.
Such a computation is known as taking a \emph{discrete logarithm}.
When $n$ is very large, this computation is essentially impossible.

Another option is for Eve to compute $(d, n)$ and then find $c^d$.
However, computing $d$ means computing $\phi(n)$.
Computing $\phi(n)$ from $n$ directly requires factoring $n$.
When $n$ has hundreds of digits, finding its factors with known algorithms can take years.

Thus, the security of RSA depends on selecting large primes so that $n$ has many digits.

\section*{Exceptions}

Every programming language has a formal way of handling errors. In Python, we raise and handle \emph{Exceptions}. There are different kinds of exceptions, each with its appropriate usage.
In Python, as in most languages, exceptions are organized into a class hierarchy.
A complete list of Python exceptions can be found \href{https://docs.python.org/2/library/exceptions.html}{here}.
The following code displays some examples of common exceptions:

\begin{lstlisting}
# A 'NameError' exception indicates that a nonexistant name was used.
>>> print(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

# An 'AttributeError' exception indicates that a nonexistant method or
# attribute was called on some object.
>>> x = list()
>>> x.fly()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'fly'

# A 'SyntaxError' exception indicates bad coding syntax.
>>> def myFunction(a, b)
  File "<stdin>", line 1
    def myFunction(a, b)
                       ^
SyntaxError: invalid syntax
\end{lstlisting}

Many exceptions, like the ones listed above, are due to coding mistakes and typos.
Exceptions can also be used programmatically to indicate a problem to the user.
To raise an exception, use the keyword \li{raise}, followed by the name of the exception.
As soon as an exception is raised, the program stops running unless the exception is handled.

\begin{lstlisting}
# raise a specific type of exception, with an error message included.
>>> x = 7
>>> if x > 5:
...     raise ValueError("'x' should not exceed 5.")
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ValueError: 'x' should not exceed 5.
\end{lstlisting}

For now we are only concerned with raising exceptions, but documentation on handling exceptions can be found \href{https://docs.python.org/2/tutorial/errors.html}{here}.

% Problem 2: Exceptions and testing my_RSA.
\begin{problem}
Write a \li{test_myRSA} function that accepts a message, two primes, and an encryption exponent.
If the first argument is not a string, raise a \li{TypeError} with error message ``message must be a string.''
If the final three arguments are not integers, raise a \li{TypeError} with error message ``p, q, and e must be integers.''
(Hint: use the built-in \li{type} function.)

Create an instance of the \li{myRSA} class.
Encrypt and decrypt the message using the keys generated by the integer arguments.
If the original message and the decryption are not exacly the same, raise a \li{ValueError} with error message ``decrypt(encrypt(message)) failed.''
Otherwise, return \li{True}.
\end{problem}

\section*{Making RSA Secure}

Since the security of RSA depends on the use of large prime numbers for $p$ and $q$, we need a fast way to find such numbers for RSA to be a practical cryptosystem.
It is very slow to check that a large number is prime by finding its factors.
In fact, it is the difficulty of factoring that makes RSA secure.

However, there exist fast algorithms that determine when a number is ``probably prime.''
One such algorithm comes from a special case of Theorem \ref{thm:fermat} known as \emph{Fermat's Little Theorem}:
\[
a^{p-1} \equiv 1 \pmod{p}
\]
for all primes $p$ and integers $a$ that are not divisible by $p$.
Therefore, to check if a fixed number $p$ is likely a prime, we can compute $a^{p-1} \pmod{p}$ for several values of $a$.
If we ever get something different than 1, then we know that $p$ is composite.
The value of $a$ that proved $p$ was composite is called a \emph{witness number}.
In fact, the converse is true: if $a^{p-1} \equiv 1 \pmod{p}$ for every $a<p$ then $p$ is prime.

With a few exceptions, if $p$ is composite then more than half the integers in $[2, p-1]$ are witness numbers.
Thus, if we run the above test on $p$ just a few times and don't find a witness number, there is a high probability that $p$ is prime.
This test is called Fermat's test for primality.

In practice, a probabilistic algorithm like Fermat's test is used to identify numbers that have a high chance of being prime.
Then, deterministic algorithms are used to verify the primality of a candidate.

% Problem 3: Implement Fermat's test for primality.
\begin{problem}
Write a function called \li{is_prime} that implements Fermat's test for primality.
Run the test at most five times, using integers randomly chosen from $[2, n-1]$ as possible witnesses.
If a witness number is found, return the number of tries it took to find it.
If no witness number is found after five tries, return $0$.
(Hint: use the built-in \li{pow} function so the exponentiation operation does not overflow.)

For most composite values of $n$, if you call \li{is_prime(n)} one hundred times, you should expect to get $0$ at most five times.
However, some composite numbers do not follow this rule.
For example, how many times do you have to call \li{is_prime()} on $340561$ to get an answer of $0$?
\label{prob:prime_confidence}
\end{problem}

\section*{PyCrypto: RSA in Python}
% A professional implementation of RSA is found in the PyCrypto module.

PyCrypto is a professional implementation of RSA in Python.
The package is called \li{Crypto}, and is included with most Python distributions.
It can be downloaded separately \href{https://pypi.python.org/pypi/pycrypto}{here}.
Documentation for PyCrypto can be found \href{https://www.dlitz.net/software/pycrypto/api/current/}{here}.
This library contains many random number generators and encryption classes.
Many programs use PyCrypto for their security needs.

\begin{warn}
Make certain that you are using the latest version of PyCrypto.
Security software is updated often to fix security vulnerablilities and bugs.
The current version of PyCrypto at the time of writing is version 2.6.1.
\end{warn}

The RSA module in PyCrypto is located in the PublicKey module.
The library allows us to explicity construct a key, or generate a key automatically.
\begin{lstlisting}
>>> from Crypto.PublicKey import RSA

# generate a 2048-bit RSA key
>>> keypair = RSA.generate(2048)

# Save the public key as a string for distribution.
>>> public_key = keypair.publickey()
>>> share_this = public_key.exportKey()
\end{lstlisting}

The \li{share_this} variable may be distributed as a public key.
Another user may then import it and use it to encrypt a message.

\begin{lstlisting}
>>> encrypter = RSA.importKey(share_this)
>>> ciphertext = encrypter.encrypt('abcde', 2048)
>>> ciphertext
('\xb1\t\xc6L\xd1u\x80.C@\x07$#\x8e\xca\x8a\x05*\xdf\x1f.N\xa9\x80
\x08\xcb*8~7\x1d\x87&&Ke\xd5\xed_H\xb9\xd0x\xac!\xf3\xa9\xdc\xbfy5
s\x92\x8d\x15\xf7vY\x99G\xb6\x03j[\xa3\xc6\x92a\n\x91\x08N\xbc\xe4
\xcd\xe2\x9b\xeb\x1eT\xe5\xef\x96\x83\x10\xb7\x0c\xd1\x9bK]z\xa5!\
xcc\xe0/\xd3L\xd4\xa9xx?*\xfb\xf6\xcaM8\xe6\x9d\xd4u\xbd\xda\xa8tf
X\x02\xfa\xff\x99\xff\xbb"\xd1\x87\'\xb9d\x1c\x1b\x9fcWd\x83\xea}\
x1f\xff\xd3\x9b0\x8e\x0f\x91\n\x16\r\r\xa5\xa5S\xafw\x07N`$\x9c]\x
ac\x96\xe3\x80l\xc9\xe5\xe4Od\xa5\t>\x16j\xa1\xb9\x9c5\xc0\xfe\xe3
\xe5i\xcd\xaf\xdc\xcad\x82\x10u\x91\xa0"\xcf\xe3A\x11\x82\x87\xb9\
xdf\xd7\x86\x87\xff\x11#-\xb5!Q)\xf7\xf1a\x94\xb3e?\xd0\x96W4\xb4\
xca\xcf\x18\xd1I\xcd\x1dl\xd7\xe0Y\xdf}F\x93\x92\xe1(d\xcc\xdc\xa7
 x\xc5`',)
>>> keypair.decrypt(ciphertext)
'abcde'
\end{lstlisting}

The RSA encryption and decryption methods on these keys are textbook approaches.

\begin{comment}
However, to increase security, we will want to pad the messages so every message encrypted with a particular key will become exactly as large (in bits) as the key itself.
A commonly used padding algorithm is implemented in PyCrypto in the \li{Crypto.Cipher.PKCS1_OAEP} module.
\begin{lstlisting}
>>> from Crypto.Cipher import PKCS1_OAEP as oaep

# generate a new key from the original RSA key.
# This key can encrypt and decrypt
>>> paddedkey = oaep.new(keypair)
>>> encrypted = paddedkey.encrypt('hello world')
>>> paddedkey.decrypt(encrypted)
'hello world'
\end{lstlisting}
\end{comment}

% Problem 4: Implement a PyCrypto-powered RSA system.
\begin{problem}
Write a new RSA class called \li{PyCrypto} that acts like the \li{myRSA} class, but implement it with methods from PyCrypto's \li{RSA} package.

Store both of your RSA keys in a single attribute called \li{_keypair}, and store a sharable string representation of the public key in the \li{public_key} attribute.
Initialize \li{_keypair} and \li{public_key} in the constructor.
The \li{encrypt} method accepts a string and encrypts it, using \li{_keypair} as the default encryption key.
If the string representation of a different public key is provided, then the message should be encrypted with the provided key.
The \li{decrypt} method decrypts an encrypted message using \li{_keypair}.

With a partner, test your classes by exchanging public keys and sending secret messages.
\end{problem}

\begin{warn}
The following warning comes from the PyCrypto \href{https://www.dlitz.net/software/pycrypto/}{website}:

``The export of cryptography software is (still) governed by arms control regulations in Canada, the United States, and elsewhere.
The export or re-export of this software may be regulated by law in your country.''
\end{warn}

% 	================ END OF LAB ================	%

% Group project problem from the old lab.
% Assumes we did a problem on digital signatures.
\begin{comment}
\begin{problem}[Group Project]
Split into several groups of at least three individuals.
Each group member should generate a private and public keypair using PyCrypto.
Share your public keys with everyone in the group.

You might find it useful to transmit your messages as strings.
Here are two functions that will read and write specially formatted strings for your messages.
\begin{lstlisting}
def print_msg(m_encrypted):
    #print the encrypted message as a specially formatted string
    out = ("----BEGIN MESSAGE----",
           m_encrypted[0],
           "----END MESSAGE----",
           "----BEGIN SIGNATURE----",
           m_encrypted[1],
           "----END SIGNATURE----")
    return '\n'.join(out)

def read_msg(m_string):
    d1 = m_string.find('----BEGIN MESSAGE----') + 21
    d2 = m_string.find('----END MESSAGE----', d1)

    d3 = m_string.find('----BEGIN SIGNATURE----', d2) + 23
    d4 = m_string.find('----END SIGNATURE----', d3)
    message  = m_string[d1:d2]
    sign = m_string[d3:d4]
    return message.strip(), sign.strip()
\end{lstlisting}

\begin{enumerate}
\item Encrypt and sign a message for someone in the group and send it to the entire group.
Attach the sender's name to the encrypted message (a claim that the message came from a particular sender).
In this exercise we will verify that the actual origin of the message is indeed the claimed origin.
Each group member should do the following:
\begin{enumerate}
\item Decrypt and verify the message addressed for you.
\item Attempt to decrypt and verify message addressed for someone else in the group.
\item Report the message and verified signature.
\end{enumerate}

\item Encrypt and sign a different message for someone in the group and send it, anonymously, to the entire group.
For example, if Bob encrypts a message and sends it to Alice, he should claim the message came from another individual.
Each group member should do the following:
\begin{enumerate}
\item Decrypt and verify the message address to you.
\item Report the message and verified signature.
\end{enumerate}
\end{enumerate}
\end{problem}

\end{comment}


\textbf{Let $f(x)=\sum_{n=-\infty}^{\infty}\frac{1}{(1+i)(n+1/2)^6}e^{inx}$. Compute the Fourier coefficients of $f$ using a DFT or FFT on 11 points to obtain $\hat{v}_k$ and verify the aliasing formula for the coefficients:
\begin{align*}
\hat{v}_k-\hat{f}_k=\sum_{m\neq 0}\hat{f}_{k+mN}
\end{align*}}
\newline

For this problem we inmediately have
\begin{align*}
\hat{f}_k=\frac{1}{(1+i)(k+1/2)^6}.
\end{align*}
To compute $\hat{v}_k=\frac{1}{N}\sum_0^{N-1}f(x_j)e^{-ikx_j}$ we need first $f(x_j)$. We can compute the values of the function using the information given by the problem, truncating the sum at $n=1000$ (because for higher values of $n$ we reach machine precision when calculating the remainder terms of the sum):
\begin{align*}
f(x_j)=\sum_{n=-1000}^{1000}\frac{1}{(1+i)(n+1/2)^6}e^{inx_j}.
\end{align*}
Once we have these values it is easy to compute $\hat{v}_k$. The right hand side of the equation is calculated for every value of $k$ in a loop in MATLAB, see code below. We have obtained an accuracy of $10^{-14}$ between the two sides of the equation. So the aliasing formula is verified.

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 2
x0=0;
xf=2*pi;
N=11;
K=5;
L=1e3;
x=linspace(x0,xf,N+1);
x(end)=[];
k=-K:K;
l=-L:L;

fhat = 1./((1+1i)*(k+(1/2)).^6);
fxj = 1./((1+1i)*(l+(1/2)).^6)*exp(1i*l'*x);
vhat = (1/N)*fxj*exp(-1i*x'*k);

lhs = vhat-fhat;
m = l;
m(1001) = [];

rhs = zeros(1,11);
for l = 1:length(k)
   rhs(l) = sum(1./((1+1i)*((k(l)+m*N)+(1/2)).^6));
end
diff = lhs-rhs
\end{verbatim}


\subsection{Eigenvalues and eigenvectors}

Which vectors remain unchanged in direction after a transformation?

That is, for a matrix \(A\), what vectors \(v\) are equal to scalar multiplication by \(\lambda \) following the operation of the matrix.

$Av=\lambda v$


\subsection*{CFL conditions}
\subsubsection*{Problem 1-2}
For stability we need
\begin{align*}
|g(w)|&\leq 1,\\
\left|1+i\frac{a\Delta t}{ \Delta x}\sin\left(\Delta xw_{\nu}\right)-\frac{4b\Delta t}{\Delta x^2}\sin^2\left(\frac{\Delta xw_{\nu}}{2}\right)\right| &\leq 1.
\end{align*}
We can use the identity $|g(w)|^2 = \Re\left[g\right]^2+\Im\left[g\right]^2$ to obtain
\begin{align*}
\left(1-4bcy^2\right)^2+\left(2ac\Delta xy\sqrt{1-y^2}\right)^2 &\leq 1,
\end{align*}
where $c$ is the Courant Number and $y = \sin\left(w\Delta x/2\right)\in[-1,1]$. We continue expanding the terms,
\begin{align*}
1+16b^2c^2y^4-8bcy^2+4a^2c^2\Delta x^2y^2(1-y^2) &\leq 1,\\
16b^2c^2y^4-8bcy^2+4a^2c^2\Delta x^2y^2(1-y^2) &\leq 0,
\end{align*}
Let $z = y^2$,
\begin{align*}
16b^2c^2z^2-8bcz+4a^2c^2\Delta x^2z(1-z) &\leq 0,\\
16b^2c^2z-8bc+4a^2c^2\Delta x^2(1-z) &\leq 0,
\end{align*}
The previous equation represents a straight line on $z\in[0,1]$. To guarantee that the entire line is negative, the endpoints must be.
\begin{itemize}
\item $z=1$:
\begin{align*}
16b^2c^2-8bc &\leq 0,\\
2bc &\leq 1,\\
2b\frac{\Delta t}{\Delta x^2} &\leq 1,\\
2b\Delta t &\leq \Delta x^2.
\end{align*}
\item $z=0$:
\begin{align*}
-8bc+4a^2c^2\Delta x^2&\leq 0,\\
a^2c\Delta x^2&\leq 2b,\\
a^2\Delta t&\leq 2b.
\end{align*}
\end{itemize}
Thus, the CFL conditions are
\begin{align}
2b\Delta t &\leq \Delta x^2,\label{eq:CFL_1}\\
a^2\Delta t&\leq 2b.\label{eq:CFL_2}
\end{align}
\subsubsection*{Problem 3}
For this problem the we have introduced an artificial diffusion by modifying the discrete PDE when adopting the Lax-Friedrichs scheme. We can obtain the desired CFL condition by removing $b$ from the 2 previous CFL conditions. From \eqref{eq:CFL_1} and \eqref{eq:CFL_2},
\begin{align*}
a^2\Delta t&\leq 2b,\\
2b &\leq \frac{\Delta x^2}{\Delta t}.
\end{align*}
Joining both together we get
\begin{align*}
a^2\Delta t&\leq 2b\leq \frac{\Delta x^2}{\Delta t},\\
a^2\Delta t&\leq \frac{\Delta x^2}{\Delta t},\\
\Delta t^2&\leq \frac{\Delta x^2}{a^2}.
\end{align*}
Wehre we have used the fact that the artificial diffusion introduced is $2b = \Delta x^2/\Delta t$. Finally, we obtain the desired CFL condition when $b=0$,
\begin{align*}
\Delta t&\leq \left|\frac{\Delta x}{a}\right|.
\end{align*}

\subsubsection*{Problem 4}
For this problem the we have introduced an artificial diffusion by modifying the discrete PDE when adopting the Lax-Wendroff scheme. We can obtain the desired CFL condition by removing $b$ from the 2 previous CFL conditions. From \eqref{eq:CFL_1} and \eqref{eq:CFL_2},
\begin{align*}
a^2\Delta t&\leq 2b,\\
2b &\leq \frac{\Delta x^2}{\Delta t}.
\end{align*}
Joining both together we get
\begin{align*}
a^2\Delta t&\leq 2b\leq \frac{\Delta x^2}{\Delta t},\\
a^2\Delta t&\leq \frac{\Delta x^2}{\Delta t},\\
\Delta t^2&\leq \frac{\Delta x^2}{a^2}.
\end{align*}
Where we have used the fact that the artificial diffusion introduced is $2b = a^2\Delta t$. Finally, we obtain the desired CFL condition when $b=0$,
\begin{align*}
\Delta t&\leq \left|\frac{\Delta x}{a}\right|.
\end{align*}


\section{Exponential family distributions}

The exponential family is a set of probability distributions with a common parametric form that lead to convenient mathematical properties when doing Bayesian inference\cite{??}. \\
The exponential family distributions include many of the most common distributions, including the normal, exponential, gamma, beta, Dirichlet, Bernoulli and Poisson \cite{???}.\\

The parametrisations of the (multivariate) exponential family which we use in this thesis is:
\[
p(\bfx|\btheta) = h(\bfx) \exp \{ \eta(\btheta) T(\bfx) - A(\btheta) \}
\]
where the functions that parametrise the distribution are:
\begin{itemize}
	\item T(\bfx): the sufficient statistics of the distribution.
	\item $\eta(\btheta)$: the natural parameters.
	\item $h(\bfx)$: the base mesure.
	\item $A(\eta)$: the log-partition function.
\end{itemize}

The standard probability density (top) and the exponential family form (bottom) for the probability distributions frequently used in this thesis are the following:\\

\textbf{Univariate normal distribution}:
\beq
	p(x|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp{-\frac{(x-\mu)^2}{2\sigma^2}}
\eeq
\begin{equation}
\begin{split}
	& \eta(\mu,\sigma^2) = \bmat \frac{\mu}{\sigma^2} & -\frac{1}{2\sigma^2} \emat \\
	& h(x) = \frac{1}{\sqrt{2\pi}}
\end{split}
\qquad \qquad
\begin{split}
	& T(x) = \bmat x & x^2 \emat \\
	&  A(\eta) = -0.25\frac{\eta_1^2}{\eta_2} - 0.5\log(\frac{1}{2\eta_2}) % A(\mu,\sigma) = \frac{\mu^2}{2\sigma^2} + \log \sigma
\end{split}
\end{equation}

\textbf{Multivariate normal distribution}:
\beq
	p(\bfx|\bmu,\bSigma) = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\bSigma|^{1/2}} \exp{-\frac{1}{2} (\bfx - \bmu)^T \bSigma{-1} (\bfx - \bmu)} \\
\eeq
\begin{equation}
\begin{split}
	& \eta(\bmu,\bSigma)  = \bmat \Sigma^{-1} \mu & -0.5\Sigma^{-1} \emat \\
	& h(x) = (2\pi)^{-\frac{k}{2}}
\end{split}
\qquad \qquad
\begin{split}
	& T(x) = \bmat x & xx^T \emat \\
	& A(\eta) = -0.25\eta_1^T \eta_2^{-1} \eta_1 - 0.5\log(|-2\eta_2|)
\end{split}
\end{equation}

\textbf{Gamma distribution}:
\beq
p(x|\alpha,\beta) = \frac{1}{\Gamma(\alpha)} \beta^\alpha x^{\alpha-1} e^{-\beta x}
\eeq
\begin{equation}
\begin{split}
	& \eta(\alpha,\beta) = \bmat \alpha - 1 & -\beta \emat \\
	& h(x) = 1
\end{split}
\qquad \qquad
\begin{split}
	& T(x) = \bmat \log x & x \emat \\
	& A(\eta) = \log(\Gamma(\eta_1 + 1)) - (\eta_1 + 1) \log(-\eta_2)
\end{split}
\end{equation}

\textbf{Beta distribution}:
\beq
p(x|a,b) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} x^{a-1} (1-x)^{b-1}
\eeq
\begin{equation}
\begin{split}
	& \eta(a,b) = \bmat a & b\emat \\
	& h(x) = \frac{1}{x(1-x)}
\end{split}
\qquad \qquad
\begin{split}
	& T(x) = \bmat \log x & \log (1-x) \emat \\
	& A(\eta) = \log(\Gamma(\eta_1)) +\log(\Gamma(\eta_2)) - \log(\Gamma(\eta_1+\eta_2))
\end{split}
\end{equation}

\textbf{Bernoulli distribution}:
\beq
	p(x|\theta) = \theta^x (1-\theta)^{1-x}
\eeq
\begin{equation}
\begin{split}
& \eta(\theta) = \frac{\theta}{1-\theta} \\
& h(x) = 1
\end{split}
\qquad \qquad
\begin{split}
& T(x) = x \\
& A(\eta) = \log(1+\exp^\eta)
\end{split}
\end{equation}

\textbf{Poisson distribution}:
\beq
p(x|\theta) = \theta^x (1-\theta)^{1-x}
\eeq
\begin{equation}
\begin{split}
& \eta(\theta) = \frac{\theta}{1-\theta} \\
& h(x) = 1
\end{split}
\qquad \qquad
\begin{split}
& T(x) = x \\
& A(\eta) = \log(1+\exp^\eta)
\end{split}
\end{equation}

The properties that all exponential family distributions share, and make it valuable for statistical modelling, are, among others: (1) The sufficient statistics can summarize arbitrary amounts of independent identically distributed data, and (2) The existence of conjugate priors within the same exponential family that guarantee a closed-form posterior distribution.

• Products of exponential family distributions are exponential family distributions, but unnormalized.

https://arxiv.org/pdf/0911.4863.pdf

Let $\bfx | \btheta_x$ and $\bfy | \btheta_y$ be random variables with exponential family distributions:
\begin{align}
	\bfx \times \bfy &=
	h_x(\bfx) \exp \{ \eta_x(\btheta_x) T_x(\bfx) - A_x(\btheta_x) \} \times h_y(\bfy) \exp \{ \eta_y(\btheta_y) T_y(\bfy) - A_y(\btheta_y) \} \\
	&= h_x(\bfx) h_y(\bfy) \exp \{ \eta_x(\btheta_x) T_x(\bfx) + \eta_y(\theta_y) T_y(\bfy) - A_x(\btheta_x) A_y(\btheta_y) \} \\
	&= \hat{h}(\bfx,\bfy) \exp \{ \hat{\eta}(\btheta_x,\btheta_y) \hat{T}(\bfx,\bfy) - \hat{A}(\btheta_x, \btheta_y) \}
\end{align}
% To exemplify the second property, consider the normally-distributed random variable with unknown mean and unknown variance:
% \baln
% 	x \sim \mathcal{N}(\mu, \tau^{-1})
% \ealn


\section{Vectors}
\subsection*{Vectors}
\textbullet An ordered finite list of numbers.\\
\textbullet Block or stacked vectors($a = [b, c, d]$), Subvectors ($a_{r:s} = (a_r,...,a_s)$), Zero vectors (all elements equal to zero), Unit vectors(($e_i = 1$)), Ones vector($1_n$) \& Sparsity($nnz(x)$)

\subsection*{Vector addition}
\textbullet \textit{Commutative}: $a + b = b + a$\\
\textbullet \textit{Associative}: $(a + b) + c = a + (b + c)$\\
\textbullet $a+0 = 0+a = a$\\
\textbullet $a - a = 0$
\subsection{Scalar-vector multiplication}
$(-2)(1, 9, 6)=(-2, -18, -12)$\\
\textbullet \textit{Commutative}: $\alpha a = a \alpha$\\
\textbullet \textit{Left-distributive}: $(\beta + \gamma)a = \beta a + \gamma a$\\
\textbullet \textit{Right-distributive}: $a(\beta + \gamma) = a\beta + a\gamma $ \\

Linear combinations: $\beta_1 a_1 + ... + \beta_m a_m$\\
\textbullet With Unit vectors: $b = b_1 e_1 +...+ b_n e_n$ \\
\textbullet If $\beta_1 +...+ \beta_m = 1$, linear combination is said to be \textit{affine combination}

\subsection{Inner product}
$a^T b = a_1 b_1 +a_2 b_2 +...+ a_n b_n$
\textbf{Properties}:\\
\textbullet \textit{Commutativity}:$a^T b = b^T a$\\
\textbullet \textit{Scalar multiplication Associativity}:\\ $(\gamma a)^T b = \gamma(a^T b)$\\
\textbullet {Vector addition Distributivity}:\\
$(a + b)^T c = a^T c + b^T c.$

\textbf{General examples}:\\
\textbullet Unit vector: $e_i^T a = a_i$ \\
\textbullet Sum: $\textbf{1}^T a = a^1 + ... + a^n$ \\
\textbullet Average: $(\textbf{1}/n)^T a=(a^1+...+a^n)/n$\\
\textbullet Sum of squares: $a^Ta = a^2_1 +...+a^2_n$\\
\textbullet Selective sum: If $b_i = 1 or 0,$ \textit{$b^Ta$} is the sum of elements for which $b_i = 1$,

\textbf{Block vectors}\\
$a^T b = a_1^T b_1 + ... + a_k^T b_k$

\subsection{Complexity of vector computations}
\textit{Space}:  8n bytes\\
\textit{Complexity of vector operations}: $x^Ty=2n-1$ flops ($n$ scalar multiplications and $n-1$ scalar additions)\\
\textit{Complexity of sparse vector operations}: If x is sparse, then computing $ax$ requires \textbf{nnz}(x) flops, If x and y are sparse, computing x + y requires no more than min{\textbf{nnz}(x), \textbf{nnz}(y)}. computing $x_T y$ requires no more than 2 min{\textbf{nnz}(x), \textbf{nnz}(y)} flops


\section{$\sigma$-Algebras, $\sigma$-Fields}

\subsection{Definition}
Given a set $X$, a $\sigma$-Algebra $\mathscr{A}$ is a collection of subsets of a given set $X$, which has to satisfy the conditions:

\begin{itemize}
\item $\emptyset, X \in \mathscr{A}$
\item If $A\in\mathscr{A}$, then $A^c := X\setminus A \in \mathscr{A}$
\item $\mathscr{A}$ has (possibly infinitely many) countable subsets $A_i \in \mathscr{A},\ i \in \mathbb{N}$. Then $\bigcup_{i=1}^{\infty}A_i \in \mathscr{A}$. That is, the $\sigma$-algebra is closed under \textit{countable unions} of its subsets.
\end{itemize}

An element of the $\sigma$-Algebra are $\mathscr{A}$-measurable sets, or measurable with respect to the $\sigma$-Algebra $\mathscr{A}$. A $\sigma$-algebra is a subset of the power set of $S$, i.e. $\mathscr{A} \subseteq P(X)$.

The difference to between an algebra and a $\sigma$-algebra is that an algebra is closed under \textit{finite} unions of subsets, i.e. $A,B\in \mathscr{A}$ then $A\cup B \in \mathscr{A}$, while a $\sigma$-algebra is closed under \textit{countable} unions, i.e. for $\{A_n: A_n \in \mathscr{A}, n\in \mathbb{N}\}$, $\bigcup_{n=1}^{\infty} A_n \in \mathscr{A}$. Sometimes the terms \textit{field} and \textit{$\sigma$-field} are used instead of \textit{algebra} or \textit{$\sigma$-algebra}. Sometimes measure and integration theory are developed from $\sigma$-rings instead o f $\sigma$-algebras, which are slightly different animals.

\subsubsection{Example: The Smallest Possible $\sigma$-Algebra}

\begin{equation}
\mathscr{A} = \{\emptyset, X\}
\end{equation}

\subsubsection{Example: The Largest Possible $\sigma$-Algebra}
The largest possible $\sigma$-Algebra is the power set:
\begin{equation}
\mathscr{A} = P(X)
\end{equation}

However, there are important examples where it is not possible to define a $\sigma$-algebra on the full power set.

\subsection{Intersection Property}
The intersection of $\sigma$ algebras is also a $\sigma$-algebra. That way, a $\sigma$-algebra with the desired properties can be constructed by creating individual $\sigma$-algebras with the properties in question and forming their intersection.

\begin{equation}
\bigcap_i \mathscr{A} \ \ \mathrm{is\ also\ a\ }\sigma-\mathrm{algebra}
\end{equation}

\subsection{Generated $\sigma$-Algebras}

For a subset of the powerset $\mathscr{M} \subseteq P(X)$, that does not necessarily have to satisfy the properties of the $\sigma$-Algebra, the \textit{smallest} $\sigma$-algebra that contains $\mathscr{M}$ is the $\sigma$-algebra \textit{generated} by $\mathscr{M}$. It can be constructed through the intersection of all $\sigma$-algebras  on $X$ that contain $\mathscr{M}$.

\begin{equation}
\sigma(\mathscr{M}) = \bigcap_{\mathscr{A}\supseteq\mathscr{M}}\mathscr{A}
\end{equation}

$\sigma(\mathscr{M})$ is the $\sigma$-algebra \textit{generated} by $\mathscr{M}$.

\subsubsection{Example: Generated $\sigma$-Algebra}

Take $X = \{a,b,c,d\}$, and $\mathscr{M} = \{ \{a\}, \{b\}\}$. Note that $\mathscr{M}$ is not a $\sigma$-algebra. To find the smallest $\sigma$-algebra that contains $\mathscr{M}$, one adds the elements necessary to fulfill the conditions on a $\sigma$-algebra. These are the empty set $\emptyset$ and the full set $X$, the union $\{a,b\}$, and the complements.

\begin{equation}
\sigma(\mathscr{M}) = \left\{ \emptyset, X, \{a\}, \{b\}, \{a,b\}, \{b,c,d\}, \{a,c,d\}, \{c,d\} \right\}
\end{equation}

\subsection{Borel $\sigma$-Algebras ($\mathscr{B}$)	}

Borel $\sigma$-Algebras is the $\sigma$-Algebra generated by \textit{open sets}, for example $\sigma(\mathbb{R}^n)$ or $\sigma(\mathscr{M})$ with $\mathscr{M} = (0,1)$.

\section{Measures, Measurable Spaces and Measure Spaces}
A \textit{measure} is a function that gives a volume measure for subsets of a $\sigma$-algebra, where a $\sigma$-algebra is a special sort of collection of subsets of some set $X$.
The combination $(X,\mathscr{A})$ of a set $X$ and a $\sigma$-Algebra that is defined on $X$ is called a \textit{measurable space}. A measure is a function defined on the $\sigma$-Algebra and maps to the positive real line (including $\infty$!):

\begin{equation}
\mu: \mathscr{A} \rightarrow [0,\infty)\cup\{\infty\}
\end{equation}

It has to satisfy the conditions:

\begin{itemize}
\item $\mu(\emptyset) = 0$
\item $\mu(\bigcup_i A_i) = \sum_i \mu(A_i)$ if $A_i \cap A_j = \emptyset$ when $i\neq j$ (additive)
\item $\mu(\bigcup^{\infty}_i A_i) = \sum^{\infty}_i \mu(A_i)$ if $A_i \cap A_j = \emptyset$ when $i\neq j$ ($\sigma$-additive)
\end{itemize}

Where the infinite sum corresponds to gradually approximating the full volume by taking the union of countably infinitely many subsets. The collection $(X,\mathscr{A},\mu)$ is a \textit{measure space}.

The reason why the $\sigma$-algebra is included in the definition of a measurable space is because unless a $\sigma$-algebra is defined on a set $X$, there is no guarantee that a measure exists on $X$.

\subsection{Example: Counting Measure}

\begin{equation}
\mu(A) = \left\{\begin{array}{l} |A| \mathrm{\ if\ }A\mathrm{\ has\ finite\ elements}\\ \infty\mathrm{\ else}\end{array}\right.
\end{equation}

\subsection{Example: Dirac Measure}
\begin{equation}
\delta_p(A) = \left\{\begin{array}{l} 1 \mathrm{\ if\ }p\in A\\ 0 \mathrm{\ else} \end{array}\right.
\end{equation}

\subsection{Example: Volume Measure}
For $X=\mathbb{R}^n$, the conventional volume measure satisfies the properties of:

\begin{itemize}
\item $\mu([0,1]^n) = 1$
\item $\mu(x + A) = \mu(A)$ for some $x\in \mathbb{R}^n$ (translation invariance)
\end{itemize}

\section{Measure Problem on $\mathbb{R}$}

This is an important example of where it is not possible to define a measure on the whole powerset. This is why measure theory is founded on $\sigma$-algebras.\\

Take the following measure problem:\\

We search a measure $\mu$ on $P(\mathbb{R})$ with the properties:

\paragraph{}
\begin{enumerate}
\item $\mu([a,b]) = b-a,\ b>a$
\item $\mu(x+A) = \mu(A),\ A\in\mathbb{R},\ x\in\mathbb{R}$
\end{enumerate}

The only solution to this problem is the trivial map, $\mu(\mathbb{R}) = 0$. A proof goes as follows:

\paragraph{Claim}
Let $\mu$ be a measure on $P(\mathbb{R})$ with $ \mu ((0,1]) < \infty  $ and (2). The only measure that satisfies this condition is the zero measure, $\mu = 0$, which violates (1).

Take the interval $I:=(0,1]$ with equivalence relation $x\sim y \equiv x-y \in \mathbb{Q}$. That is, $x$ and $y$ are considered equivalent if they differ by a rational number. That is, we define sets of equivalent numbers, equivalence classes  $ [x] := \{ x+r: r \in \mathbb{Q}, x+r \in I \} $. The equivalence classes are a disjoint decomposition (a partition) of the unit interval $I$ in terms of possibly infinite, countable number of elements.

Take a choice set  $A\subseteq I$ that consists of one number from each of the equivalence classes $[x]$ that make up $I$. (cf. sections \ref{sec:choicesets}, \ref{sec:axiomofchoice}). It has the property:
\begin{itemize}
\item For each $[x]$ there is an $a\in A$ with $a \in [x]$
\item For all $a,b \in A$, if $a,b \in [x] \implies a=b$
\end{itemize}

Take the translations of a set $A$, $A_n := r_n + A$ where $(r_n)_{n\in\mathbb{N}}$  are an enumeration of $\mathbb{Q}\cap(1,1]$.

\paragraph{Proof}
The axiom of choice guarantees that it is possible to form a choice set $A$ and the definition of equivalence classes guarantees that the translations of $A_n$ are still choice sets.

\paragraph{Claim}
$A_n \cap A_m = \empty$ if $n\neq m$.

\paragraph{Proof}
Take $x \in A_n \cap A_m \implies \begin{array}{ll} x = r_n + a, & a_n \in A\\ x = r_m + a_m, & a_m \in A \end{array}$

Then, $r_n + a_n = r_m + a+m  \implies a_n - a_m = r_m - r_n \in \mathbb{Q} \implies a_n \sim a_m$ according to the definition of the equivalence classes. Also, $a_m, a_n \in [a_m] \implies a_n = a_m \implies r_n = r_m \implies n=m$.

\paragraph{Claim}
$(0,1] \subseteq \bigcup_{n\in\mathbb{N}}A_n \subseteq (-1,2]$

\paragraph{Proof}
Given $r_n \in \mathbb{Q}\cap (-1,1]$, $-1< r_n \leq 1$ and $a_n \in A,\ 0<a_n\leq 1$ given that $A\subset (0,1]$, $-1 < r_n + a_n \leq 2$. Therefore $A_n \subseteq (-1,2]\ \forall n\in\mathbb{N}$. Therefore the union $(0,1] \subseteq \bigcup_{n\in\mathbb{N}}A_n \subseteq (-1,2]$.

\paragraph{Assume} $\mu$ a measure on $P(\mathbb{R})$ with $\mu((0,1]) < \infty$ and (2).

By (2):  $\mu(r_n + A) = A$ for all $n\in\mathbb{N}$.

By the property $(0,1] \subseteq \bigcup_{n\in\mathbb{N}}A_n \subseteq (-1,2]$, $\mu((0,1]) \leq \mu(\bigcup_{n\in \mathbb{N}} A_n) \leq \mu((-1,2])$.

Further, $\mu((0,1]) = c < \infty$.

So we can use the disjoint union to express $\mu((-1,2]) = \mu((-1,0] \sqcup (0,1] \sqcup (1,2]) = 3c$.

That implies, $c \leq \sum_{n=1}^{\infty}\mu(A_n) \leq 3c \implies c \leq \sum_{n=1}^{\infty}\mu(A) \leq 3c$. Given that the series is infinite, the only way for this to be true is if $\mu(A)=0$.

\paragraph{Conclusion}
The measure $\mu(A) = 0$, which implies $\mu((0,1]) = 0$. Because of translation invariance and $\sigma$-additivity, $\mu(\mathbb{R}) = \mu(\bigcup_{m\in\mathbb{Z}^+} (m,m+1]) = 0$. That means that the only measure that satisfies the conditions stated in the measure problem assigns $0$ to the length of the whole real line.

\section{Measurable Maps}
\paragraph{Definition} Given two measurable spaces $(\Omega_1,\mathscr{A}_1), (\Omega_2, \mathscr{A}_2)$, a measurable map with respect to $\mathscr{A}_1, \mathscr{A}_2$ is $f:\Omega_1 \rightarrow \Omega_2$ if $f^{-1}(A_2)\in\mathscr{A}_1$ for all $A_2 \in \mathscr{A}_2$. That is, $f$ connects the two $\sigma$-algebras $\mathscr{A}_1$ and $\mathscr{A}_2$ in that the \textit{preimage} (cf. section \ref{sec:preimage}) of an element $A_2 \in \mathscr{A}_2$, $f^{-1}(A_2)$ is an element of $\mathscr{A}_1$.

\subsection{Example: Characteristic Function, Indicator Function}
Take the measurable spaces $(\Omega,\mathscr{A})$ and $(\mathbb{R},\mathscr{B}(\mathbb{R}))$.

\begin{equation}
\chi_A: \omega \rightarrow \mathbb{R},\ \ \chi_A(\omega) = \left\{\begin{array}{l} 1,\ \omega\in A\\ 0,\ \omega\notin A \end{array}\right.
\end{equation}

For all measurable $A\in\mathscr{A}$, $\chi_A$ is a measurable map. The four possible preimages are:

\begin{equation}
\begin{array}{l}
\chi_A^{-1}(\empty) = \empty,\ \chi_A^{-1}(\mathbb{R}) = \Omega\\
\chi_A(\{ 1 \}) = A\ \chi_A(\{ 1 \}) = A^c
\end{array}
\end{equation}

Where all of the preimages are contained in $\mathscr{A}_1$.

\subsection{Example: Composition of Measurable Maps}

Take measurable spaces $(\Omega_1,\mathscr{A}_1),(\Omega_2,\mathscr{A}_2),(\Omega_3,\mathscr{A}_3)$,  connected through measurable maps:

\begin{equation}
\begin{array}{l}
f: \Omega_1 \rightarrow \Omega_2 \\
g: \Omega_2 \rightarrow \Omega_3
\end{array}
\end{equation}

Then $f \circ g: \Omega_1 \rightarrow \Omega_3$ is also measurable, because $(g\circ f)^{-1}(A_3) = \underbrace{f^{-1}(\underbrace{g^{-1}(A_3)}_{\in \mathscr{A}_2})}_{\in \mathscr{A}_1}$.

\subsection{Example: Sums and Products of Measurable Maps}
Given $(\Omega,\mathscr{A}), (\mathbb{R},\mathscr{B}(\mathbb{R})$, if $f,g: \Omega \rightarrow \mathbb{R}$ 	are measurable maps, then $f+g, f-g, f\times g$ and $|f|$ are also measurable maps. This follows from the property that compositions of measurable maps are also measurable.

\section{Lebesgue Integrals}
\subsection{Lebesgue Integrals for Step Functions}
Take a measure space $(X,\mathscr{A}, \mu)$ and the measurable space $(\mathbb{R},\mathscr{B})$, where $X$ is any set, $\mathscr{A}$ is a special collection of subsets of $X$, $\mu$ is a map $\mu: \mathscr{A} \rightarrow [0,\infty]$, and $\mathscr{B}$ is a Borel $\sigma$-algebra on $\mathbb{R}$. Then let $f:X \rightarrow \mathbb{R}$ be a measurable map, so that $f^{-1}(E) \in \mathscr{A}$ for all $E\subseteq \mathscr{B}(\mathbb{R})$.

\paragraph{Characteristic Function}
The integral of a characteristic function $\chi_A$ is $I(\chi_A) = \mu(A)$.

\paragraph{Simple Functions}
Simple functions are, for example, step functions, staircase functions, etc., that can be expressed in terms of a sum of characteristic functions. That is, for $A_1,A_2,...,A_n\in\mathscr{A}$, $c_1,c_2,...,c_n\in\mathbb{R}$:

\begin{equation}
f(x) = \sum_{i=1}^n c_i \chi_{A_i}(x)
\end{equation}

Since characteristic functions are measurable and sums of characteristic functions are measurable, simple functions are measurable. Then the integral would be:

\begin{equation}
I(f) = \sum_{i=1}^n c_i \mu(A_i)
\end{equation}

However, there is a problem with this definition, because of the possibility of having to subtract infinitely large intervals. The options are to restrict simple functions to either:

\begin{itemize}
\item restrict $A_i$ to be finite size sets.
\item restrict $c_i$ to be positive.
\end{itemize}

\paragraph{S+}
The set of positive functions $S^+ := \{ f:X\rightarrow\mathbb{R} : f \mathrm{\ simple\ function},\ f\geq0\}$ where $f$ is measurable and has finitely many values (picture a staircase, rather than a smooth curve). For $f\in S^+$, choose representation $f(x) = \sum_{i=1}^n c_i \chi_{A_i}(x)$, $c_i \geq 0$.

\paragraph{Lebesgue Integral for Simple Functions}
The Lebesgue integral of $f\in S^+$ with respect to the measure $\mu$:

\begin{equation}
\int_X f(x) d\mu(x) = \int_X f d\mu = I(f) = \sum_{i=1}^n c_i \mu(A_i)\ \in [0,\infty]
\end{equation}

This is a well-defined object that is independent of the specific representation of $f(x)$. It has the properties:

\begin{itemize}
\item $I(\alpha f + \beta g) = \alpha I(f) + \beta I(g)$ for $\alpha,\beta \geq 0$
\item $f \leq g \implies I(f) \leq I(g)$ (monotonicity)
\end{itemize}

The Lebesgue integral for simple functions enables defining the integral for more complex functions by approximating them.

\paragraph{Definition}
Given a non-negative function $f:X \rightarrow [0,\infty)$, there are positive simple functions $h\in S^+$ that approximate it from below $\{h: h\in S^+, h\leq f\}$ with $h=\sum_{i=1}^n c_i \chi_{A_i}$. Then the integral of $f$ is given by the largest possible function within that set.

The \textit{Lebesgue Integral} of a function $f$ with respect to a measure $\mu$ is

\begin{equation}
\int_X f d\mu := \mathrm{sup}\{ I(h) : h\in S^+, h\leq f \}
\end{equation}

$f$ is called $\mu$-integrable if $\int_X f d\mu < \infty$.

The only thing that was needed to define the integral was a measure space $(X,\mathscr{A},\mu)$

\section{Monotone Convergence Theorem}

A convergence theorem should show circumstances under which a limit can be pulled into the integral, and monotone implies that the theorem will deal with a monotonically changing series.

\paragraph{Preliminaries}

Take the measurable positive function $f: X\rightarrow [0,\infty)$ which has Lebesgue integral $\int_X f d\mu \in [0,\infty]$.

\subparagraph{Equality}
If $f=g$, $\mu$ almost everywhere, then $\int_X f d\mu = \int_X g d\mu$.

That is, if $f=g$ "almost everywhere" with respect to the measure $\mu$, then the integrals are identical. This is more general than to simply say $f=g$. Rather, it requires $\mu(\{x \in X: f(x) \neq g(x) \}) = 0$. The Lebesgue integral "cannot see" things that happen on $0$-measure sets. For example, if the measure is a length measure $\mu ( [a,b] ) = b-a$, then, if $f$ and $g$ differ at a single point, then $\int_X f $ and $\int_X g$ are still the same.

\subparagraph{Monotonicity}

Similartly, if $f\leq g$,  $\mu$ almost everywhere, then $\int_X f d\mu \leq \int_X g d\mu$.

Given that a positive simple function $h:X \rightarrow [0,infty)$ assumes a finite amount of different values, it is permissible to represent $h(x)$ in terms of the values $t$ that it assumes:

\begin{equation}
h(x) = \sum_{i=1}^n c_i \chi_{A_i} (x) = \sum_{t\in h(X)} t \chi_{\{x\in X: h(x) = t\}}
\end{equation}

Using this representation, Then the Lebesgue integral:

\begin{equation}
I(h) = \sum_{t\in h(X)\setminus \{0\}} t \mu (\{ x\in X : h(x) = t \} )
\end{equation}

Where omitting the $t\in\{0\}$ makes no difference to the integral. Divide the set $X$ into $X = \tilde{X} + \tilde{X}^c$ with $\mu(\tilde{X}^c) = 0 $ and $\tilde{X} \in \mathscr{A}$. Then let:

\begin{equation}
\tilde{h}(X) := \left\{ \begin{array}{l} h(x), x\in \tilde{X}\\ a, x\in \tilde{X}^c \end{array} \right.
\end{equation}

Where $a \in [0, infty)$.  Then:

\begin{equation}
\tilde{h}(x) = \int_{t\in h(X)} t \chi_{x\in \tilde{X}: h(x) = t} + a \chi_{\tilde{X}^c}
\end{equation}

Then the integral:

\begin{equation}
I (\tilde{h}) = \sum_{t\in h(X)} t \mu( \{ x\in \tilde{X} : h(x) = t \}) + a \underbrace{\mu(\tilde{X}^c)}_{0}
\end{equation}

So that $I(h) = I(\tilde{h})$. This means that we can modify a simple function $h$ on a set with measure $0$ however we like without affecting the integral. This is enough to prove that if $f\leq g$ then $\int_X f d\mu \leq \int_X g d\mu$ in the "almost everywhere" sense. Divide $X$ into $\tilde{X}:= \{x \in X: f(x) \leq g(x) \}$ and $\tilde{X}^c$ with measure zero, where $f(x) \leq g(x)$ is not true. Since $I(h) = I(\tilde{h})$:

\begin{equation}
\begin{array}{rl}
\int_X f d\mu &= \mathrm{sup}\{ I(h): h\in S^+, h\leq f \} =  \mathrm{sup}\{ I(\tilde{h}): h\in S^+, \tilde{h}\leq f \leq g\mathrm{\ on\ }\tilde{X}\}\\
&\leq \mathrm{sup}\{ I(\tilde{h}): h\in S^+, \tilde{h}\leq g\mathrm{\ on\ }\tilde{X}\} = \int_X g d\mu
\end{array}
\end{equation}

\subparagraph{Zero Integral}
$f=0$, $\mu$ almost everywhere, then $\int_X d\mu = 0$. (Note that only positive functions $f$ are considered.

\paragraph{Definition: Monotone Convergence Theorem}

Take a measure space $(X,\mathscr{A},\mu)$ and non-negative measurable functions $f_n: X \rightarrow [0,\infty), f: X \rightarrow [0,\infty)$, with $f_1 \leq f_2 \leq f_3 \leq ... $ with $\mu-a.e.$ (almost everywhere) and $\lim_{n\rightarrow \infty} f_n(x) = f(x). \mu-a.e. (x\in X)$, where the limit is taken in a point-wise sense. That is, it applies for a fixed point $x\in X$.

The monotone convergence theorem states that, given a monotonic series of functions, the limit can be pulled into the integral:

\begin{equation}
\lim_{n\rightarrow \infty} \int_X f_n d\mu = \int_X \lim_{n\rightarrow \infty} f_n d\mu = \int_X f d\mu
\end{equation}

\subsection{Application: Series}

Take a series of measurable non-negative functions $(g_n)_{n\in \mathbb{N}}$, $g_n: X \rightarrow [0,\infty]$ measurable for all $n$, which does not necessarily have monotonic behavior. Then:

\begin{equation}
\sum_{n=1}^{\infty} g_n : X \rightarrow [0,\infty] \mathrm{\ is\ measurable}
\end{equation}

Now, while the series $(g_n)_{n\in \mathbb{N}}$ was not necessarily monotonically increasing, the sum is. therefore:

\begin{equation}
\int_X \sum_{n=1}^{\infty} g_n d\mu = \sum_{n=1}^{\infty} \int_X g_n d\mu
\end{equation}

\section{Fatou's Lemma}
The starting point is, again, a measure space $(X,\mathscr{A},\mu)$ and a series of non-negative functions $f_n : X\rightarrow [0,\infty]$ that are measurable for all $n\in \mathbb{N}$. Then:

\begin{equation}
\int_X \liminf_{n\rightarrow \infty} f_n d \mu \leq \liminf_{n\rightarrow\infty}\int_X f_n d\mu
\end{equation}

Where the limit inferior $\liminf_{n\rightarrow \infty} f_n(x)$ of a sequence of numbers $f_n(x)$ is $\lim_{n\rightarrow\infty} (\inf_{k\geq n} f_k (x))$. That is, that is, $\liminf_{n\rightarrow \infty} f_n$ is actually a function $(\liminf_{n\rightarrow \infty} f_n)(x)$, that, for a series of functions $f_n$, picks out the lower limit of the series of functions evaluated at the point $x$ as $n\rightarrow \infty$. Fatou's Lemma shows that the integral over the infimum of a series of functions $f_n$ as $n\rightarrow \infty$ is smaller than the infimum of the series of integrals of those functions as $n\rightarrow \infty$. That seems intuitively correct -- the integral over the minimum value of a collection of positive functions should be smaller or equal than the minimum of the integrals.  Fatou's Lemma follows from the monotone convergence theorem by recognizing the infimum of a series of functions $\inf_{k\geq n}f_k$ as a function $g_n = (\inf_{k\geq n}f_k)(x)$ and recognizing that $g_n$ forms a monotonically increasing series $g_1 \leq g_2 \leq ...$, because as $n$ get's larger, the infimum $\inf_{k\geq n} f_k$ must get either larger or stay the same.

\section{Lebesgue's Dominated Convergence Theorem}

Take a measure space $(X,\mathscr{A},\mu)$, define a set of Lebesgue integrable functions $\mathscr{L}(X,\mathscr{A},\mu)$, which is simply written $\mathscr{L}(\mu)$ because the measurable space is assumed fixed from the context. The set is given by: $\mathscr{L}(\mu) := \{f: X\rightarrow \mathbb{R}, \mathrm{\ measurable}: \int_X |f| d\mu \}$, where the exponent matters. So, the set of "L1-integrable functions", $\mathscr{L}^1(\mu) := \{f: X\rightarrow \mathbb{R}, \mathrm{\ measurable}: \int_X |f|^1 d\mu \}$. Since the Lebesgue integral was so far defined only for positive functions $f$, for $f\in\mathscr{L^1}(\mu)$ we can write $f=f^+ - f^-$ where $f^+$ and $f^-$ are both positive. $f$ can then be integrated by subtracting the integrals of $f^+$ and $f^-$.

\paragraph{Theorem}
Given a sequence of complex valued functions $f_n: X \rightarrow \mathbb{R}$ that are measurable for all $n\in\mathbb{N}$, which pointwise approaches the limit function $\lim_{n\rightarrow \infty} f_n(x) = f(x)$ for $x\in X$ almost everywhere with respect to $\mu$ (almost everywhere meaning that the limit expression holds for the $f_n$ evaluated at points $x$ everywhere except possibly some set of of points with measure zero).

If, pointwise, it is true $|f_n| \leq g$ with $g\in \mathscr{L}^{1}(\mu)$ for all $n\in \mathbb{N}$, then $g$ is an \textit{integrable majorant}. The implication is: $f_1, f_2, f_3, ... \in \mathscr{L}^{1}(\mu), f\in\mathscr{L}^1(\mu)$.

Further, convergence gives:
\begin{equation}
\lim_{n\rightarrow \infty}\int_X f_n d \mu = \int_X f d\mu
\end{equation}

Which looks exactly like the monotone convergence theorem, except that the requirement on the sequence $f_n$ is only that there is an integrable majorant $g_n$ in $\mathscr{L}^1(\mu)$. The theorem gives a sufficient condition under which the almost everywhere convergence of a sequence of functions guarantees convergence in the $\mathscr{L}^1(\mu)$ norm. The theorem can be proved, in particular, via Fatou's Lemma, which guarantees an upper bound on the integral of the limit of the infimum of a sequence of positive functions. The integrable majorant $g\geq |f|$ allows for the construction of a series of non-negative functions $h_n := 2g - |f_n - f| \geq 0$, recognizing that $|f_n - f| \leq |f_n| + |f| \leq 2g$.

\subsection{Triangle Inequalities}
\begin{equation}
|f+g| \leq |f| + |g|
\end{equation}

\begin{equation}
|\int_X f | \leq \int_X |f|
\end{equation}

\section{Carath\'eodory's Extension Theorem}
Carath\'eodory's extension theorem states that, given a set $X$, for a semiring of sets $\mathscr{A}\subseteq P(X)$, with a pre-measure $\mu: \mathscr{A} \rightarrow [0,\infty]$, there exists a unique extension to the semiring and the pre-measure that is the unique $\sigma$-algebra $\sigma(\mathscr{A})$ generated by the semiring and a measure $\tilde{\mu}: \sigma(\mathscr{A})\rightarrow [0,\infty]$. The theorem is of particular importance because it guarantees the existence and uniqueness of the Lebesgue measure.

\subsection{Semirings of Sets}
For a set $X$, semirings of sets are a collection of sets $\mathscr{A}\subseteq P(X)$ that weaker criteria than $\sigma$-algebras.

The criteria are:

\begin{itemize}
\item $\empty \in \mathscr{A}$
\item For $A,B \in \mathscr{A}$, $A\cap B \in \mathscr{A}$
\item For $A,B \in \mathscr{A}$, there is a difference operation $A\setminus B$. It is not required that $A\setminus B$ is an element of $\mathscr{A}$, but it is required that there is a union of sets $\bigcup_i A_i = A\setminus B$ with $A_i \in \mathscr{A}$.
\end{itemize}

\subsubsection{Example: Intervals on $\mathbb{R}$}
$\mathscr{A} := \{ [a,b) : a,b \in \mathbb{R}, a\leq b \}$ is not a $\sigma$-algebra because $\mathbb{R} \notin \mathscr{A}$. However, the generated $\sigma$-algebra $\sigma(\mathbb{\mathscr{A}}) = \mathscr{B}(\mathbb{R})$ is unique and is the Borel $\sigma$-algebra. $\mathscr{A}$ fulfills the criteria of a semiring.

\subsection{Pre-measure}
A pre-measure $\mu : \mathscr{A} \rightarrow [0,\infty]$ in a semiring of sets $\mathscr{A}$ fulfills:

\begin{enumerate}
\item $\mu(\empty) = 0$
\item $\mu(\bigcup_{j=1}^{\infty} = \sum_{j=1}^{\infty} \mu(A_j)$ for $A_j \in \mathscr{A}$ if $A_i \cap A_j = \empty$ if $i\neq j$ and also $\bigcup_{j=1}^{\infty} A_j \in \mathscr{A}$, where this condition is not necessarily satisfied for semirings, because, in contrast to $\sigma$-algebras, the union of elements of semirings is not necessarily also in the semiring.
\end{enumerate}

\subsection{Lebesgue Measure}
Take the semiring $\mathscr{A} := \{ [a,b) : a,b \in \mathbb{R}, a\leq b \}$ and the premeasure $\mu : \mathscr{A} \rightarrow [0,\infty]$, $\mu([a,b)) = b-a$, then Carath\'eodory's theorem guarantees that there is a unique extension, which is $\mathscr{B}(\mathbb{R})$ and the Lebesgue measure.

\section{Lebesgue-Stieltjes Measures}
Lebesgue-Stieltjes measures are measures that are constructed for monotonically increasing (non-decreasing) functions $F$. Take non-decreasing functions $F:\mathbb{R} \rightarrow \mathbb{R}$. Such functions could be discontinuous or constant, as long as they do not decrease. Then, on the semiring $\mathscr{A} := \{ [a,b) : a,b \in \mathbb{R}, a\leq b \}$, a pre-measure $\mu_F : \mathscr{A} \rightarrow [0,\infty]$ so that $\mu_F([a,b)) = F(b^-) - F(a^-)$ where the superscript $(-)$ clarifies which side of a discontinuity has to be included, should $F$ be discontinuous at $a$ or $b$. This is consistent with the boundaries of the interval $[a,b)$. (It is equally well possible to approach this using intervals $(a,b]$.)

Now, Carath\'eodory's Theorem ensures that this can be extended to one unique measure and $\sigma$-algebra, $\mu_F : \mathscr{B}(\mathbb{R}) \rightarrow [0,\infty]$, which is called the Lebesgue-Stieltjes measure for the function $F$.

\subsection{Example: Lebesgue Measure}
An easy example is $F(x) = x$. Then $\mu_F([a,b)) = b-a$ which is the Lebesgue measure.

\subsection{Example: Zero Measure}
Take $F(x) = 1$, then $\mu_F([a,b)) = 0$, which is the zero measure.

\subsection{Example: Dirac Measure}
Take the discontinuous step function $F(x) = \left\{\begin{array}{l} 0, x < 0\\ 1 x\geq 0 \end{array}\right.$. Then $\mu_F([-\epsilon,\epsilon)) = 1$ for arbitrarily small $\epsilon$. That is, the Dirac Measure $\delta_0$ is the Lebesgue-Stieltjes Measure with respect to a step function on $\mathscr{\mathbb{R}}$.

\subsection{Example: Density Functions}
Take a function $F:\mathbb{R} \rightarrow \mathbb{R}$ that is monotonically increasing and continuously differentiable, i.e. $F':\mathbb{R} \rightarrow [0,\infty)$. Then the measure is simply the interval $\mu_I([a,b)) = F(b) - F(a) = \int_a^b F'(x) dx$. That means that the Lebesgue-Stieltjes measure $\mu_F: \mathscr{A} \rightarrow \int_{\mathscr{A}}F'(x) dx$ with respect to the monotonically increasing function $F$ is a measure from a Borel set to an integral of the derivative of $F'(x)$. In this context, $F'(x)$ is a \textit{density function}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 4}

Show by induction that the number of degree-2 nodes (nodes with 2 children) in any non-empty binary tree is 1 fewer than the number of leaves.
Conclude that the number of internal nodes in a full binary tree is 1 fewer than the number of leaves.

\subsection*{Solution}

Let $n(T)$ be the numebr of nodes, $d(T)$ be the number of nodes with 2 children and $l(T)$ be the number of leaves, in tree $T$.
Objective is to prove $l(T) = d(T) + 1$ in any tree $T$.
Proof is given by induction on number of nodes $n$ in the binary tree $T$.

\begin{itemize}\itemsep=0pt
\item[] \textit{Base case:} The most trivial binary tree with $n(T)=1$ contains only the root node therefore $l(T) = 0 + 1 = d(T) + 1$.
\item[] \textit{Inductive Hypothesis:} We form the inductive hypothesis as for any binary tree with $n(T) \leq k$, $l(T) = d(T) + 1$.
\item[] \textit{Induction Step:} Using the inductive hypothesis, we need to show $l(T) = d(T)+1$ holds true for any tree with $n(T) = k+1$ where $k > 0$.
Since $n(T) > 1$, $T$ will have tree cases that follows.
  \begin{itemize}\itemsep=0pt
    \item[] If root $r$ in tree $T$ ($n(T) = k+1$) has only a left subtree $T'$, since $r$ is neither a leaf nor a degree-2 node, $l(T) = l(T')$ and $d(T)=d(T')$.
However, $n(T') = n(T) - 1 = k$.
Thus, using the inductive hypothesis, $l(T') = d(T') + 1$.
Therefore, $l(T) = d(T) + 1$.
    \item[] Similarly, if root $r$ in tree $T$ has only a right subtree $T'$, same reasoning will lead to $l(T') = d(T') + 1$.
    \item[] In case both the left subtree ($T'$) and the right subtree ($T''$) of the binary tree $T$ ($n(T) = k+1$) are non-empty, the root $r$ of tree $T$ will be a degree-2 node.
Therefore, $l(T) = l(T') + l(T'')$ and $d(T) = d(T') + d(T'') + 1$ will hold.
    Since both $n(T')$ and $n(T'')$ are less than $k$, using the inductive hypothesis, $l(T') = d(T') + 1$ and $l(T'') = d(T'') + 1$.
    Therefore, $l(T) = l(T') + l(T'') = d(T') + 1 + d(T'') + 1 = d(T) + 1$ and the induction is complete.
  \end{itemize}
  Therefore number of leaves in tree $T$ is always 1 more than number of degree-2 nodes.
\end{itemize}
This prove would immediately lead to conclusion of the second part.
Since all non-leaf nodes in a full binary tree are of second degree, number of internal nodes will be one less than the number of leaves.


\subsection{Limits of real functions}

\subsubsection{Limit operator}

For a function \(f(x)\),

\(\lim_{x\rightarrow a}f(x)=L\)

We can say that \(L\) is the limit if:

\(\forall \epsilon >0 \exists \delta >0 \forall x[0<|x-p|<\delta \rightarrow |f(x)-L|<\epsilon]\)


\section{Week 2}
\subsection{1/12/2020}
\begin{itemize}
    \item For multiplication, the entry in row $i$ column $j$ is the dot of row $i$ of A and column $j$ of B.
    \item Distributed from the left and right
    \item Associative
    \item Suppose A is a square matrix. $A^{-1}$ is a matrix of the same such that:
    \item $AI = IA = A$
    \item $A^{-1}A=AA^{-1}=I$
    \item An inverse matrix may not exist if
    \begin{itemize}
        \item not square
        \item if any row is a linear combination of other rows
    \end{itemize}
    \item If an inverse matrix exists, it is really helpful to solve a system of equations.
    \item A $2\times 2$ matrix is invertible if and only if $ad-bc$, the determinant, is not zero. In this case, we find the inverse of a 2x2 matrix.
    $$A=\begin{bmatrix}a&b\\c&d\end{bmatrix}$$
    $$A^{-1}=\frac{1}{|A|}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}=\frac{1}{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}$$
    \item A diagonal matrix of any size is invertible when none of the diagonal entries are zero.
    \item If they are invertible $(AB)^{-1} = B^{-1}A^{-1}$
    \item Any matrix that causes a known change, should be easy to invert. We have to think of a matrix that would undo what is done.
    Ex. $$E=\begin{bmatrix}1&0&0\\-5&1&0\\0&0&1\end{bmatrix}$$ subtracts 5 times row1 from row2
    \item So we would want something that adds 5 times row1 to row2
    Result:
    $$E=\begin{bmatrix}1&0&0\\5&1&0\\0&0&1\end{bmatrix}$$
    \item $$P^{-1} = P^{12} = \begin{bmatrix}0&1&0\\1&0&0\\0&0&1\end{bmatrix}$$
    \item When the inverse is not immediately obvious, one method of calculating it is called the Gauss-Jordan method. This is done by.
    \begin{enumerate}
        \item Putting the matrix you wish to invert next to the identity matrix
        \item Use row transformations (elimination) to transform your matrix into the identity.
        \item The identity will have been transformed into the inverse matrix.
    \end{enumerate}
    \item Ex. $$A=\begin{bmatrix}2&-1&0&1&0&0\\-1&2&-1&0&1&0\\0&-1&2&0&0&1\end{bmatrix}$$
    \item $2R_2+R_1$
    $$A=\begin{bmatrix}2&-1&0&1&0&0\\0&3&-2&1&2&0\\0&-1&2&0&0&1\end{bmatrix}$$
    \item $3R_3+R_2$
    $$A=\begin{bmatrix}2&-1&0&1&0&0\\0&3&-2&1&2&0\\0&0&4&1&2&3\end{bmatrix}$$
    \item $2R_2+R_3$
    $$A=\begin{bmatrix}2&-1&0&1&0&0\\0&6&0&3&6&3\\0&0&4&1&2&3\end{bmatrix}$$
    \item $6R_1+R_2$
    $$A=\begin{bmatrix}12&0&0&9&6&3\\0&6&0&3&6&3\\0&0&4&1&2&3\end{bmatrix}$$
    $$A=\begin{bmatrix}12&0&0&9&6&3\\0&6&0&3&6&3\\0&0&4&1&2&3\end{bmatrix}\begin{bmatrix}\frac{1}{12}\\\frac{1}{6}\\\frac{1}{4}\end{bmatrix}=\begin{bmatrix}1&0&0&3/4&1/2&1/4\\0&1&0&1/2&1&1/2\\0&0&1&1/4&1/2&3/4\end{bmatrix}$$
\end{itemize}

\subsection{1/13/2021}
\begin{itemize}
    \item We are trying to factorize for the lower and upper triangular matrices.
    \item Ex. $\begin{bmatrix}1&0\\-3&1\end{bmatrix}\begin{bmatrix}2&1\\6&8\end{bmatrix}=\begin{bmatrix}2&1\\0&5\end{bmatrix}$
    \item every $E^{-1}_{ij}$ matrix is lower triangular, with 1's on the main diagonal. The off diagonla entry is +k to undo the elimination done by the - k when multiplied from the left by $E_{ij}$
    \item The product of several elimination matrices is still lower triangular.
\end{itemize}


\section*{Ex.1.2}
\subsubsection*{Contract two nodes at random. Show min--cut is found with exponential small probability.}

Define a graph $G$ with $n$ nodes. Let the nodes $u_1,\ldots,u_{u/2}$ be part of the set $S_1$ and let the nodes $u_{u/2+1},\ldots,u_n$ be part of the set $S_2$. Let all nodes in $S_j, j=1,2$ have internal edges between them, and let there be an edge from $(u_1,u_n)$. Hence the min--cut would be $(u_1,u_n)$. It is easy to see that if the algorithm at any point contracts a node from $S_1$ and $S_2$ the min--cut is not found. Hence, we want to find the probability that at all steps, two nodes from either $S_{1i}$ or $S_{2i}$ is contracted, where $S_{ji}, j=1,2$ is the set of remaining nodes at $S_j$ after $i$ iterations.

Define $X_{ji} = \{ 2 \text{ nodes are picked from } S_{ji} \}, j=1,2$. and note that $X_{1i}\cap X_{2i}= \emptyset$. Furthermore, note that
$$
P(X_{ji}) = \frac{\binom{\abs{S_{ji}}}{2}}{\binom{\abs{S_{1i}} + \abs{S_{2i}}}{2}}
$$
which is simply all the ways you can pick two nodes from $S_{ji}$, out of all the ways you can pick two nodes in the remaining graph.

Define the event $E_i = \{ S_{1i} \text{ or } S_{2i}\}$

Then
\begin{align*}
P(E_i) &= P(X_{1i} \cup X_{2i}) = P(X_{1i}) + P(X_{2i}) - P(X_{1i} \cap X_{2i}) = P(X_{1i}) + P(X_{2i})\\
&=  \frac{\binom{\abs{S_{1i}}}{2}}{\binom{\abs{S_{1i}} + \abs{S_{2i}}}{2}} + \frac{\binom{\abs{S_{2i}}}{2}}{\binom{\abs{S_{1i}} + \abs{S_{2i}}}{2}}\\
&= \frac{\binom{\abs{S_{1i}}}{2} + \binom{\abs{S_{2i}}}{2}}{\binom{\abs{S_{1i}} + \abs{S_{2i}}}{2}}
\end{align*}
where the last equation is strictly less than one as long as there are still elements in both of $S_{ji}$, which there for sure will be if no nodes from $S_{1i}$ and $S_{2i}$ have been contracted in the same iteration $i$.

Set $p^* = \max_{i} P(E_i)$ and note that because of the observation above $p^*<1$.

Hence
\begin{align*}
P(\bigcap_{i=1}^{n-2} E_i) = \prod_{i=1}^{n-2}P(E_i) < \prod_{i=1}^{n-2}p^* = (p^*)^{n-2}
\end{align*}

which converges to 0 exponentially fast.


\subsection{De Morgan's Laws}

\begin{itemize}
\item \(\neg (A\lor B)\Leftrightarrow (\neg A\land \neg B)\)
\item \(\neg (A\land B)\Leftrightarrow (\neg A\lor \neg B)\)
\end{itemize}

This expresses the duality of normal form.

Duality is the principle that binary operators have inverses, and when they are swapped with their inverse, the truth value of the statement is unaffected.


%!TEX root=report.tex
\subsection{SVD}
Singular Value Decomposition (SVD) is a useful method for factorizing matrices.
 According to the SVD-theorem a matrix $X$  can be expressed as
\begin{equation}
X=U \Sigma V^{T}.
\label{eq:theory-svd}
\end{equation}

If the matrix $X$ has the size $m \times n$, the factorized components have the following properties \cite{introduction-to-data-mining}:
\begin{itemize}
\item The columns in $U$ are the eigenvectors of $X X^T$, with the size $m \times m$.
\item The columns in $V$ are the eigenvectors of $X^T X$, with the size $n \times n$.
\item $\Sigma$ is a diagonal matrix, consisting of the square root of the corresponding eigenvalues to the eigenvectors in $U$ and $V$.
It has the size $m \times n$. The square root eigenvalues are also called ``singular values''.
\end{itemize}

In the case where $X$ has many more rows than columns, the $U$ matrix will contain more eigenvectors than are strictly needed to reconstruct $X$ and the $\Sigma$ matrix will have lots of zero-rows.
A so called \textit{economy-size} SVD is therefore often used.
The difference being that all the redundant eigenvectors in $U$ are removed, so it has the size $m \times n$ and the zero-rows from $\Sigma$ are also removed, so its size becomes $n \times n$.

An important property of SVD is that the $V$ matrix contains all eigenvectors and is therefore an orthogonal matrix, for which it is known that
\begin{align}
Q^T = Q^{-1} && Q Q^T = I && Q^T Q = I && \text{, where Q is an orthogonal matrix.}
\end{align}

In the normal SVD (not \textit{economy-size}) $U$ is also a complete orthogonal matrix.
However in \textit{economy-size} some of the columns have been removed, so it is only known that $U^T U = I$.

It is standard procedure to rearrange the matrices in $U \Sigma V^T$, so the largest values in $\Sigma$ appears first in the diagonal.
The reason being that $V$ is a change of basis matrix -- it transforms elements in the original space to a new orthogonal space.
Here the first axis has the largest variance, the second axis has the second highest variance and so on.
The variance described by each axis, is given by the diagonal elements in $\Sigma^2$.
Typically one calculates a percentage called ``variance explained by principal components'' using
\begin{equation}
\rho_{jj} = \frac{\Sigma^2_{jj}}{\sum_{i=1}^n \Sigma^2_{ii}}.
\end{equation}

It is often seen that most of the variance in $X$ can be described using very few eigenvectors, thus one can reduce the dimensionality of a dataset to the most ``principal'' components.
The above mentioned analysis goes under the umbrella term ``Principal Component Analysis (PCA)'', where a ``principal component'' (loadings) should be understood as the axes in the transformed space (i.e., the eigenvectors).


\subsection{Rectified Linear Unit (ReLU)}

\subsubsection{The function}

\(a(z)=\max (0,z)\)

\subsubsection{The derivative}

Its differential is \(1\) for values of \(z\) above \(0\), and \(0\) for values of \(z\) below \(0\).

The differential is undefined at \(z=0\), however this is unlikely to occur in practice.

\subsubsection{Notes}

The ReLU activation function induces sparcity.


\subsection{Defining Image and Kernel}

\subsubsection{Image}

Of a function, the set of vectors in the codomain hit by the domain.
The image of a function $f:\;\R^m\rightarrow \R^n$ is:

\[\boxed{
    \mathrm{Im}(f)=\{\tb{y}\in \R^n | \exists\;\tb{x}\in \R^m\text{ s.t. }f(\tb{x})=\tb{y}\}
}\]

Similar in concept to the range of a function in non-linear context.

\subsubsection{Kernel}

Set of vectors in the domain that are mapped to $\tb{0}$ in the codomain.
Kernel of a function $f:\;\R^m\rightarrow \R^n$ is:

\[\boxed{
    \mathrm{Ker}(f)=\{\tb{x}\in \R^m | f(\tb{x})=\tb{0}\}
}\]

Analogous to the roots/zeros of a polynomial.

\subsection{Examples}

Following linear transformation's image lives in $\R^2$:

\[T(\tb{x})=\left(\begin{array}{cc}
    1 & 1 \\
    -1 & 1
    \end{array}\right) \tb{x}\]

Following linear transformation's image forms a plane that is
$\R^2$:

\[T(\tb{x})=\left(\begin{array}{cc}
    1 & -1 \\
    0 & 2 \\
    2 & 2
    \end{array}\right) \tb{x}\]

\subsection{Span}

If $A$ is an $n\times m$ matrix, then image of $T(\tb{x})=A\tb{x}$ is set of all vectors in $\R^n$
that are linear combinations of column vectors of $A$.\\

\noindent
Thus, the span of a set of $n$ vectors is all linear combinations of those vectors:

\[\boxed{
    \mathrm{span}(\tb{v}_1,\tb{v}_2,\cdots,\tb{v}_n)=\{\sum c_i\tb{v}_i | c_i\in\R\}
}\]

So the span of column vectors of $A$ is the image of the associated linear transformation.

\subsection{Kernel}

Kernel amounts to finding solutions to $A\tb{x}=\tb{0}$. Kernels are closed under linear combinations.
Kernel can never be empty set, it always holds true that $T(\tb{0})=\tb{0}$.

\subsection{Invertible Linear Transformations}

Main conclusions about image and kernel:

\begin{itemize}
    \item Kernel is always (trivially) $\{\tb{0}\}$, else would imply dependance and therefore singularity in the associated matrix $A$
    so $\boxed{\mathrm{ker}(T)=\{\tb{0}\}}$
    \item Image is always the space $\R^n$ if associated matrix $A$ is $n\times n$, so $\boxed{\mathrm{Im}(T)=\R^n}$
\end{itemize}


%!TEX root=report.tex
\subsection{Least angular regression (LAR)}
The LAR model is almost equivalent to the linear regression, except that instead of only minimizing the sum of squares, an $\ell^1$ norm penalty as added on the $\hat{\beta}$-vectors coefficients.
\begin{align}
\min_{\hat{\beta}}\ (Y - \hat{Y})^T (Y - \hat{Y}) \quad \text{subject to} \quad ||\hat{\beta}||_1 \le s
\end{align}

The LAR algorithm \cite{LAR-algorithm} solves this problem by initially letting all the $\beta$ coefficients be zero.
It then finds the attribute, $b_1$, with the highest absolute correlation with the dependent variable Y and increases $b_1$  (or decreases depending on the sign of the correlation)
until it reaches a point where another attribute $b_2$ has as much  correlation with the residuals $R=Y-\hat{Y}$ as  $b_1$ has.
At this point the algorithm then increases/decreases both $b_1$ and $b_2$ in their joint direction until another attribute $b_i$ has the highest residual correlation.
This process can then be continued until there is no benefit in increasing any of the $b_i$s - that is, the full LAR solution is equivalent to the linear regression.
It should also be noted that if a coefficient crosses 0 in a iteration (step),
that coefficient should be set to 0, and the regression direction should subsequently be recomputed.

So why might one use LAR instead of for example the LASSO model?
When using the LASSO one has to chose a specific value of $s$ to get one solution.
However, in the LAR model you actually get the full solution path which means that you can find all the LASSO solutions instead of having to do multiple LASSO with different regularization parameters ($s$).


\label{problem_definition}
Clustering is the task of gathering items in a way that elements belonging
to the same group (the \emph{cluster}) are more similar to each other other than the ones
assigned to the others.\\
More formally, given a input:
\begin{itemize}
    \item $X = \{x_0, \dots ,x_n\}$, the initial set of elements.
    \item $d: X \times X \to \mathbb{R}$, a \emph{metric} measuring the similarity.
\end{itemize}
The final goal is to find the cluster configuration
\begin{equation*}
    C = \left\{ c_0, \dots , c_m \right\} \mid \bigcup_{C} = X
\end{equation*}
partitioning $X$ into $m$ clusters, maximizing the intra-cluster distance
(dual problem of minimizing inter-cluter distance):

\begin{equation}
    \underset{C}{\mathrm{argmax}}
    \sum_{c \in C}
        \sum_{i,j}^{|c|}
            d(c_i,c_j)
\end{equation}

\subsection*{Challenges}
The concept of clustering is simple and powerful; moreover, its versatility
and generality are its greatness and at the same time source of many difficulties.
%% Metrics identification
Given the only strict requirement to be the definition of a metric over the
domain of $X$, clustering is applied to a wide variety of problems.
Clearly, each domain offers different optimization opportunities and
particular challenges.
In particular, the choice of the metric heavily influences the final outcome quality.
As a result even the medical~\cite{siless2013comparison}, the mechanical
engineering~\cite{wilding2011clustering} and the mobile networks~\cite{cheng2009stability}
literatures features different studies that address this particular challenge
suggesting highly specialized distance functions.

%% Inter/Intra cluster distance measure
Once the proper metric is identified, the following huge influencing factors
are the mathematical definitions of ``intra-cluster'' and ``inter-cluster''
distances. They vary a lot in the different implementation leading to
completely different clustering configurations.
For example, three methods are widely used when performing agglomerative
clustering to define the distance between two clusters:
the average, the minimum, and the maximum distance.
The average approach uses the barycenters, while the minimum (maximum)
relies upon the minimal (maximal) distance between any two points
belonging to a different clusters.

\subsection*{Choosing the $k$ parameter}
The first main issue of the k-means described in Section \ref{related} is the choice
of the number of clusters the dataset has to be divided into.

The original k-means does not address this problem at all. Thus, some heuristic has to
be applied. One possibility is to have a deep knowledge of the structure underlying the
dataset, having an idea of the target number of groups. On the other hand, especially
in exploratory data analysis, this value cannot be known in advance.
Hence, the most common solution is to run the algorithm with increasing values of $k$ and
finally keeping the parameter that produces the best-quality clusters.

Choosing the right value for this parameter is crucial, since a wrong one may
instruct the algorithm to collapse entities that are actually very far from each other
(or vice versa).

\subsection*{Positioning the initial centroids}
Given that an optimal value for $k$ is found, the other big problem is how to position the $k$
initial centroids.

Given enough time, the k-means algorithm will always converge. However it may be to a local
minimum. This is highly dependent on the initialization of the centroids. The boostrapping
phase can be addressed in various ways. For example, the
\emph{scikit-learn}\footnote{http://scikit-learn.org} Python library
uses an approach that positions the initial centroids to be (generally) distant
from each other by default. This procedure provides provably better results than using a random
one~\cite{arthur2007k}.

Using a proper and efficient boostrapping heuristic is very important, since misplacing
the initial centroids does not allow the algorithm to find the real clusters underlying
the data.


%!TEX root=report.tex
\subsection{The design matrix}
In OLS regression, a linear combination of the columns in the design matrix $X$, is used to predict $Y$.
As a starting point, the only values in $X$ are a column of ones (to fit an intercept) and a column containing the observed time $\mathbf{t}$

\begin{equation}
X = \left[\begin{matrix} \mathbf{1} & \mathbf{t} \end{matrix}\right].
\end{equation}
In the above case one would predict $\hat{Y}$ as $\beta_1 + \beta_2 t$ where $\beta_1$ will correspond to the intercept and $\beta_2$ will correspond to the speed of mass gain.
From this it follows naturally to include the acceleration by adding a column

\begin{equation}
X = \left[\begin{matrix} \mathbf{1} & \mathbf{t} & \frac{1}{2} \mathbf{t}^2 \end{matrix}\right]
\end{equation}
From Fourier analysis it is known that functions can be approximated by infinite sums of the trigonometric functions sines and cosines.
Trigonometric functions are periodic in their behavior and thus are able to model periodic signals in the function that one wishes to approximate.

In the GRACE dataset, however, the measurements are recorded with a frequency of $\frac{1}{10}$ days.
The Nyquist-Shannon sampling theorem states that one cannot find periodic signals above the Nyquist frequency which is given as half of the sample frequency.
Therefore an infinite sum is not suited for approximating the function.
Instead it is used, that the maximal frequency in the data has a period of a year ($T = 365.242$ days) while the minimal frequency will have 18 periods per year since  $\sfrac{365.242}{18} \approx 20.29 \text{ days}$. Hence it follows that the final design matrix will be given as
\begin{equation*}
\resizebox{\textwidth}{!}{$
X = \left[\begin{matrix}
	\mathbf{1} &
	\mathbf{t} &
	\frac{1}{2} \mathbf{t}^2 &
	\cos\left( \dfrac{2 \pi}{\frac{365.242}{1}} \mathbf{t} \right) &
	\sin\left( \dfrac{2 \pi}{\frac{365.242}{1}} \mathbf{t} \right) &
	\cdots &
	\cos\left( \dfrac{2 \pi}{\frac{365.242}{18}} \mathbf{t} \right) &
	\sin\left( \dfrac{2 \pi}{\frac{365.242}{18}} \mathbf{t} \right)
\end{matrix}\right].
$}
\end{equation*}

From this it's seen that the angular frequency is $\omega_i =  \frac{2 \pi}{\sfrac{365.242}{i}} = \frac{2 \pi i}{T}$


\subsection{Surface area}
Let $S$ be a surface parameterized by $x$ and $y$.
That is, let $S=(x,y,z)$ where $z=f(x,y)$.
The tangent lines at a point on $S$ form a tiny parallelogram.
The area $a$ of the parallelogram is given by the magnitude of the cross product.
$$a=\left|\frac{\partial S}{\partial x}\times\frac{\partial S}{\partial y}\right|$$
By summing over all the parallelograms we obtain the total surface area $A$.
Hence
$$A=\int\!\!\!\int dA=\int\!\!\!\int a\,dx\,dy$$
The following example computes the surface area of a unit disk
parallel to the $xy$ plane.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
M1 = ((0,0,0),(0,0,-1),(0,1,0))
M2 = ((0,0,1),(0,0,0),(-1,0,0))
M3 = ((0,-1,0),(1,0,0),(0,0,0))
M = (M1,M2,M3)
cross(u,v) = dot(u,M,v)
z = 2
S = (x,y,z)
a = abs(cross(d(S,x),d(S,y)))
defint(a,y,-sqrt(1 - x^2),sqrt(1 - x^2),x,-1,1)
\end{Verbatim}

\noindent
$\displaystyle \pi$

\bigskip
\noindent
The result is $\pi$, the area of a unit circle, which is what we expect.
The following example computes the surface area of $z=x^2+2y$ over
a unit square.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
M1 = ((0,0,0),(0,0,-1),(0,1,0))
M2 = ((0,0,1),(0,0,0),(-1,0,0))
M3 = ((0,-1,0),(1,0,0),(0,0,0))
M = (M1,M2,M3)
cross(u,v) = dot(u,M,v)
z = x^2 + 2y
S = (x,y,z)
a = abs(cross(d(S,x),d(S,y)))
defint(a,x,0,1,y,0,1)
\end{Verbatim}

\noindent
$\displaystyle \tfrac{5}{8}\log(5)+\tfrac{3}{2}$

\bigskip
\noindent
The following exercise is from
{\it Multivariable Mathematics} by Williamson and Trotter, p. 598.
Find the area of the spiral ramp defined by
$$S=\begin{bmatrix}u\cos v\\\ u\sin v\\ v\end{bmatrix},\qquad 0\le u\le1,\qquad 0\le v\le3\pi$$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
M1 = ((0,0,0),(0,0,-1),(0,1,0))
M2 = ((0,0,1),(0,0,0),(-1,0,0))
M3 = ((0,-1,0),(1,0,0),(0,0,0))
M = (M1,M2,M3)
cross(u,v) = dot(u,M,v)
x = u cos(v)
y = u sin(v)
z = v
S = (x,y,z)
a = circexp(abs(cross(d(S,u),d(S,v))))
defint(a,u,0,1,v,0,3pi)
\end{Verbatim}

\noindent
$\displaystyle \tfrac{3}{2}\pi\log(1+2^{1/2})+\frac{3\pi}{2^{1/2}}$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
float
\end{Verbatim}

\noindent
$\displaystyle 10.8177$


\subsection{Probabilistic language models}

\subsubsection{Introduction}

Probabilistic language models can predict future words given a history of words.

This can be used for predictive text. For example if a user types "Did you call your" we may want to estimate the probability that the next word is "child".

We can state this problem:

\(P(child|did\ you\ call\ your)\)

By definition this is:

\(P(child|did\ you\ call\ your)= \dfrac{P(did\ you\ call\ your\ child)}{P(did\ you\ call\ your)}\)

We can estimate each of these:

\(P(did\ you\ call\ your\ child)=\dfrac{|did\ you\ call\ your\ child|}{|5\ word\ sentences|}\)

\(P(did\ you\ call\ your)=\dfrac{|did\ you\ call\ your|}{|4\ word\ sentences|}\)

\subsubsection{Data requirements}

This needs a large corpus, which may not be practical.

Additionallly, the words must be indexed, and not simply stored as a bag of words.

\subsubsection{Decomposition}

We can decompose the probabilities using the chain rule.

\(P(did\ you\ call\ your\ child)=P(did)P(you|did)...P(child|did\ you\ call\ your)\)

\(P(w_1,...,w_k)= \prod_k p(w_k|w_1,...,w_{k-1})\)

\subsubsection{N-grams}

We can simplify the decomposition using the Markov assumption:

\(P(w_k|w_1,...,w_{k-1})=P(w_k|w_{k-1})\)

This is a \(1\)-gram.

We can do this for \(n\) words back. This is an \(n\)-gram.

\subsubsection{Smoothing}

We can use smoothing to address small corpuses.

\(P(did\ you\ call\ your\ child)=\dfrac{|did\ you\ call\ your\ child|+1}{|5\ word\ sentences|+V}\)

\(P(did\ you\ call\ your)=\dfrac{|did\ you\ call\ your|+1}{|4\ word\ sentences|+V}\)

For some value \(V\).

\subsubsection{Perplexity}

We can compare probabilistic language models using perplexity.

We can then choose the model with the lowest perplexity.

\(perplexity(w_1, w_2, ..., w_n)=P(w_1, w_2, ..., w_n)^{-\dfrac{1}{n}}\)

We can expand this:

\(perplexity(w_1, w_2, ..., w_n)=\prod_i P(w_i| w_1, ..., w_{i-1})^{-\dfrac{1}{n}}\)

Depending on which n-gram we use we can then simplify this.


\subsection{Avogadro's constant}
There is a proposal to define Avogadro's constant as exactly
84446886 to the third power.
(Fox, Ronald and Theodore Hill.
``An Exact Value for Avogadro's Number.''
{\it American Scientist} 95 (2007): 104--107.)
The proposed number in the article is actually $(84446888)^3$.
In a subsequent addendum the authors reduced it to $84446886^3$ to make the
number divisible by 12. (See {\tt www.physorg.com/news109595312.html}.)
This number corresponds to an ideal cube of atoms with 84,446,886
atoms along each edge.
Let us check the difference between the proposed value and the measured value
of $(6.0221415\pm0.0000010)\times10^{23}$ atoms.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = 84446886^3
B = 6.0221415*10^23
A-B
\end{Verbatim}

$\displaystyle -5.17173\times10^{16}$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
0.0000010*10^23
\end{Verbatim}

$\displaystyle 1\times10^{17}$

We see that the proposed value is within the experimental error.
Just for the fun of it, let us factor the proposed value.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
factor(A)
\end{Verbatim}

$\displaystyle 2^3\times3^3\times1667^3\times8443^3$