notes

Preliminaries It is usually the case that we have a dataset $\mathcal{D} = {x_1, \cdots, x_N}$ and a parametrized family of distributions $p_\theta (x)$. We would like to find the parameters that best describe the data. This is typically done using [[MLE and MAP|maximum likelihood estimation (MLE)]]. In this method, the optimal parameters are those that maximize the log likelihood of the data. Mathematically speaking, $$ \hat{\theta}_\mathrm{MLE} = \arg\max_\theta \frac{1}{N}\sum_{i=1}^{N}\log p_{\theta}(x_i)....

MAP-Elites is an elegant algorithm for solving general optimization problems. To be more accurate, it is an illumination algorithm that tries to find high-performing and diverse solutions in a search space. At its core, it is a simple algorithm, both conceptually and to implement. Here, I briefly introduce the main idea behind the algorithm and its components. I will also discuss its merits and demerits compared to other approaches. This note is based on Illuminating Search Spaces by Mapping Elites....

(Based on a lecture by professor Coralia Cartis, University of Oxford) (I don’t currently plan to extend it, but may expand and add more details to some of the later chapters in the future. I also like to eventually add some useful resources (books, talks, notes, etc.) about optimization) This brief note is about optimization problems. Though the main focus is on the general non-convex optimization problem, a lot of the methods borrow some ideas from convex optimization, so there are a lot of similarities....

Some of the notes I wrote when I took an information theory course. It contains basic definitions and theorems. It doesn’t get into much details and is sort of an intuitive cheat-sheet. The content aligns with the first few chapters of the book “Elements of Information Theory” by Cover and Thomas, so you can find the proofs there (it is a very well-written and easy-to-read book, I definitely recommend it!)....

My handwritten notes/summaries for an undergraduate stochastic processes course that I took in my 8th semester can be found HERE. I examined a lot of books, but I didn’t like the order in which they introduced different topics. These notes are organized in a way that I thought was most intuitive and logical. So if you can read them (and that’s a big if😊) you shouldn’t have any problem following the arguments and the chain of thought....