高斯分布

本文为机器学习(白板推导系列)P3-P8的学习笔记。

符号约定

$$X_{N \times p}=(x_1,x_2,\cdots,x_N)^{T},\quad x_{i}=(x_{i1},x_{i2},\cdots,x_{ip})^{T}$$ $$x_i \stackrel{i.i.d}{\sim}N(\mu,\sigma^2)$$

极大似然估计

令，，则

$$p(x)=\frac{1}{\sqrt{2\pi}\sigma}exp[-\frac{(x-\mu)^2}{2\sigma^2}]$$ $$\theta_{MLE}=\underset{\theta}{argmax}\ p(X|\theta)$$

对数似然函数为

$$\begin{aligned} lnp(X|\theta)&=ln\prod_{i=1}^Np(x_i|\theta)=\sum_{i=1}^N lnp(x_i|\theta)\\ &=\sum_{i=1}^N ln \frac{1}{\sqrt{2\pi}\sigma}exp[-\frac{(x_i-\mu)^2}{2\sigma^2}]\\ &=\sum_{i=1}^N[-ln\sqrt{2\pi}-ln\sigma-\frac{(x_i-\mu)^2}{2\sigma^2}] \end{aligned}$$

$$\begin{aligned} \mu_{MLE}&=\underset{\mu}{argmax}\ lnp(X|\theta)\\ &=\underset{\mu}{argmax} \sum_{i=1}^N -\frac{(x_i-\mu)^2}{2\sigma^2}\\ &=\underset{\mu}{argmin} \sum_{i=1}^N(x_i-\mu)^2 \end{aligned}$$

对求偏导

$$\begin{aligned} \frac{\partial}{\partial \mu}\sum_{i=1}^N (x_i-\mu)^2&=0\\ \sum_{i=1}^N -2(x_i-\mu)&=0\\ \end{aligned}$$

得

$$\mu_{MLE}=\frac{1}{N}\sum_{i=1}^N x_i$$

因为

$$E(\mu_{MLE})=\frac{1}{N}\sum_{i=1}^N E(x_i)=\mu$$

故为的无偏估计。

$$\sigma^2_{MLE}=\underset{\sigma^2}{argmax}\sum_{i=1}^N[-ln\sigma-\frac{(x_i-\mu)^2}{2\sigma^2}]$$

对求偏导

$$\begin{aligned} \frac{\partial}{\partial \sigma}\sum_{i=1}^N[-ln\sigma-\frac{(x_i-\mu)^2}{2\sigma^2}]&=0\\ \sum_{i=1}^N[-\frac{1}{\sigma}+(x_i-\mu)^2 \sigma^3]&=0\\ \end{aligned}$$

得

$$\sigma^2_{MLE}=\frac{1}{N}\sum_{i=1}^N (x_i-\mu_{MLE})^2$$

因为

$$\begin{aligned} E(\sigma^2_{MLE})&=E[\frac{1}{N}\sum_{i=1}^N (x_i-\mu_{MLE})^2]\\ &=E(\frac{1}{N}\sum_{i=1}^N x_i^2-\mu_{MLE}^2)\\ &=\frac{1}{N}\sum_{i=1}^N E(x_i^2)-\mu^2-E(\mu_{MLE}^2)+\mu^2\\ &=\frac{1}{N}\sum_{i=1}^N [E(x_i^2)-(E(x_i))^2]-[E(\mu_{MLE}^2)-(E(\mu_{MLE}))^2]\\ &=\frac{1}{N}\sum_{i=1}^N Var(x_i)-Var(\mu_{MLE})\\ &=\sigma^2-Var(\frac{1}{N}\sum_{i=1}^N x_i)\\ &=\sigma^2-(\frac{1}{N})^2 \sum_{i=1}^N Var(x_i)\\ &=\frac{N-1}{N} \sigma^2 \end{aligned}$$

故为的有偏估计，的无偏估计为

$$\hat \sigma^2=\frac{1}{N-1} \sum_{i=1}^N (x_i-\mu_{MLE})^2$$