对于二维数据 ( x i , y i ) (x_{i},y_{i}) (xi​,yi​)进行建模，通过回归方程 y i = β 0 + β 1 x i + u y_{i}=\beta_{0}+\beta_{1}x_{i}+u yi​=β0​+β1​xi​+u描述其数据关系。求该方程需要获取未知参数 β 0 ^ \hat{\beta_0} β0​^​和 β 1 ^ \hat{\beta_1} β1​^​

矩估计方法依赖于零条件均值假设： E ( u ∣ x ) = 0 E(u|x)=0 E(u∣x)=0该假设的意思是给定 x x x，通过回归方程所得的 y ^ \hat{y} y^​与实际 y y y的误差，平均值为0。也就是说因果关系上， y y y只受到 β 0 \beta_0 β0​和 β 1 \beta_1 β1​的影响。

根据零条件均值可以推出两个公式。 E ( u ) = E ( E ( u ∣ x ) ) = 0 c o v ( x , u ) = E ( x u ) − E ( x ) E ( u ) = E ( x u ) = E ( E ( x u ∣ x ) ) = E ( x E ( u ∣ x ) ) = 0 \begin{equation*} \begin{aligned} E(u)&=E(E(u|x))=0\\ \end{aligned} \end{equation*}\\ \begin{equation*} \begin{aligned} cov(x,u)&=E(xu)-E(x)E(u)\\ &=E(xu)\\ &=E(E(xu|x))\\ &=E(xE(u|x))\\ &=0 \end{aligned}\end{equation*} E(u)​=E(E(u∣x))=0​​cov(x,u)​=E(xu)−E(x)E(u)=E(xu)=E(E(xu∣x))=E(xE(u∣x))=0​​ 这是求解未知参数的关键。

另外已知 y i = β 0 + β 1 x + u    = = = = =    y ˉ = β 0 + β 1 x ˉ + u ˉ    = = = = = ;    \begin{equation*} \begin{split} y_i&=\beta_0+\beta_1x+u\phantom{\;=====\;}\tag{1}\\ \end{split} \end{equation*} \\\begin{equation*} \begin{split} \bar{y}&=\beta_0+\beta_1\bar{x}+\bar{u}\phantom{\;=====;\;}\tag{2} \end{split} \end{equation*} yi​​=β0​+β1​x+u=====​​(1)​yˉ​​=β0​+β1​xˉ+uˉ=====;​​(2)​ 由 ( 1 ) − ( 2 ) (1)-(2) (1)−(2)得 y i − y ˉ = β 1 ( x − x ˉ ) + ( u − u ˉ ) ( x − x ˉ ) ( y i − y ˉ ) = β 1 ( x − x ˉ ) 2 + ( u − u ˉ ) ( x − x ˉ ) \begin{equation*} \begin{split} y_i-\bar{y}=\beta_1(x-\bar{x})+(u-\bar{u}) \end{split} \end{equation*}\\ \begin{equation*} \begin{split} (x-\bar{x})(y_i-\bar{y})=\beta_1(x-\bar{x})^2+(u-\bar{u})(x-\bar{x}) \end{split} \end{equation*} yi​−yˉ​=β1​(x−xˉ)+(u−uˉ)​​(x−xˉ)(yi​−yˉ​)=β1​(x−xˉ)2+(u−uˉ)(x−xˉ)​​ 遍历所有的 i = 1 , 2 , 3.... i=1,2,3.... i=1,2,3....，并求和 1 N ∑ i = 1 N ( x − x ˉ ) ( y i − y ˉ ) = 1 N ∑ i = 1 N β 1 ( x − x ˉ ) 2 + 1 N ∑ i = 1 N ( u − u ˉ ) ( x − x ˉ ) \begin{equation*} \begin{split} \frac{1}{N}\sum_{i=1}^N(x-\bar{x})(y_i-\bar{y})=\frac{1}{N}\sum_{i=1}^N\beta_1(x-\bar{x})^2+\frac{1}{N}\sum_{i=1}^N(u-\bar{u})(x-\bar{x}) \end{split} \end{equation*} N1​i=1∑N​(x−xˉ)(yi​−yˉ​)=N1​i=1∑N​β1​(x−xˉ)2+N1​i=1∑N​(u−uˉ)(x−xˉ)​​由于 c o v ( x , u ) = 0 cov(x,u)=0 cov(x,u)=0，因此 1 N ∑ i = 1 N ( u − u ˉ ) ( x − x ˉ ) = 0 \frac{1}{N}\sum_{i=1}^N(u-\bar{u})(x-\bar{x})=0 N1​∑i=1N​(u−uˉ)(x−xˉ)=0。从而得到 ∑ i = 1 N ( x − x ˉ ) ( y i − y ˉ ) = ∑ i = 1 N β 1 ( x − x ˉ ) 2 β 1 ^ = ∑ i = 1 N ( x − x ˉ ) ( y i − y ˉ ) ∑ i = 1 N ( x − x ˉ ) 2 = c o v ( x , y ) v a r ( x ) \begin{equation*} \begin{split} \sum_{i=1}^N(x-\bar{x})(y_i-\bar{y})&=\sum_{i=1}^N\beta_1(x-\bar{x})^2 \\ \hat{\beta_1}&=\frac{\sum_{i=1}^N(x-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^N(x-\bar{x})^2} \\&=\frac{cov(x,y)}{var(x)} \end{split} \end{equation*} i=1∑N​(x−xˉ)(yi​−yˉ​)β1​^​​=i=1∑N​β1​(x−xˉ)2=∑i=1N​(x−xˉ)2∑i=1N​(x−xˉ)(yi​−yˉ​)​=var(x)cov(x,y)​​​再由式子 ( 1 ) (1) (1)得 β 0 ^ = y ˉ − β 1 ^ x ˉ \hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x} β0​^​=yˉ​−β1​^​xˉ

简单的说，我们的任务就是找到使均方误差最小的 β 0 \beta_0 β0​和 β 1 \beta_1 β1​。数学表达式如下： m i n β 1 ^ , β 0 ^ 1 N ∑ i = 1 N ( u ^ i − u ˉ ) 2 = m i n β 1 ^ , β 0 ^ 1 N ∑ i = 1 N ( u ^ i ) 2 = m i n β 1 ^ , β 0 ^ 1 N ∑ i = 1 N ( y i − β 0 ^ − β 1 ^ x i ) 2 \begin{equation*} \begin{split} \mathop{min}\limits_{\hat{\beta_1},\hat{\beta_0}}\frac{1}{N}\sum_{i=1}^N(\hat{u}_i-\bar{u})^2 &=\mathop{min}\limits_{\hat{\beta_1},\hat{\beta_0}}\frac{1}{N}\sum_{i=1}^N(\hat{u}_i)^2 \\&=\mathop{min}\limits_{\hat{\beta_1},\hat{\beta_0}}\frac{1}{N}\sum_{i=1}^N(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)^2 \tag{3} \end{split} \end{equation*} β1​^​,β0​^​min​N1​i=1∑N​(u^i​−uˉ)2​=β1​^​,β0​^​min​N1​i=1∑N​(u^i​)2=β1​^​,β0​^​min​N1​i=1∑N​(yi​−β0​^​−β1​^​xi​)2​​(3)​对 ( 3 ) (3) (3)尾式分别对 β 1 、 β 2 \beta_1、\beta_2 β1​、β2​微分得到 − 2 × 1 N ∑ i = 1 N ( y i − β 0 ^ − β 1 ^ x i ) = 0 − 2 × 1 N ∑ i = 1 N ( x i ( y i − β 0 ^ − β 1 ^ x i ) ) = 0 \begin{equation*} \begin{split} -2\times\frac{1}{N}\sum_{i=1}^N(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)=0\\ -2\times\frac{1}{N}\sum_{i=1}^N(x_i(y_i-\hat{\beta_0}-\hat{\beta_1}x_i))=0 \end{split} \end{equation*} −2×N1​i=1∑N​(yi​−β0​^​−β1​^​xi​)=0−2×N1​i=1∑N​(xi​(yi​−β0​^​−β1​^​xi​))=0​​这等价于样本矩条件，连列两个方程，可以求得与1.1证明相同的结果。

分别定义一下指标以评价方程对数据的拟合情况。

他们的关系为SST=SSE+SSR，以下将给出证明。

S S T = ∑ i = 1 N ( y i − y ˉ ) 2 = ∑ i = 1 N ( y i − y i ^ + y i ^ − y ˉ ) 2 = ∑ i = 1 N ( u i ^ + y i ^ − y ˉ ) 2 = ∑ i = 1 N ( u i ^ ) 2 + ∑ i = 1 N ( y i ^ − y ˉ ) 2 + 2 ∑ i = 1 N u i ^ ( y i ^ − y i ˉ ) = S S R + S S E + 2 ∑ i = 1 N u i ^ ( y i ^ − y i ˉ ) \begin{equation*} \begin{aligned} SST&=\sum_{i=1}^N(y_i-\bar{y})^2 \\&=\sum_{i=1}^N(y_i-\hat{y_i}+\hat{y_i}-\bar{y})^2 \\&=\sum_{i=1}^N(\hat{u_i}+\hat{y_i}-\bar{y})^2 \\&=\sum_{i=1}^N(\hat{u_i})^2+\sum_{i=1}^N(\hat{y_i}-\bar{y})^2+2\sum_{i=1}^N\hat{u_i}(\hat{y_i}-\bar{y_i}) \tag{4} \\&=SSR+SSE+2\sum_{i=1}^N\hat{u_i}(\hat{y_i}-\bar{y_i}) \end{aligned} \end{equation*} SST​=i=1∑N​(yi​−yˉ​)2=i=1∑N​(yi​−yi​^​+yi​^​−yˉ​)2=i=1∑N​(ui​^​+yi​^​−yˉ​)2=i=1∑N​(ui​^​)2+i=1∑N​(yi​^​−yˉ​)2+2i=1∑N​ui​^​(yi​^​−yi​ˉ​)=SSR+SSE+2i=1∑N​ui​^​(yi​^​−yi​ˉ​)​​(4)​另外结合零条件均值假设，考察 ∑ i = 1 N u i ^ ( y i ^ − y i ˉ ) \sum_{i=1}^N\hat{u_i}(\hat{y_i}-\bar{y_i}) ∑i=1N​ui​^​(yi​^​−yi​ˉ​) ∑ i = 1 N u i ^ ( y i ^ − y i ˉ ) = ∑ i = 1 N u i ^ y i ^ − y ˉ ∑ i = 1 N u i ^ = ∑ i = 1 N u i ^ ( β 0 ^ − β 1 ^ x i ) − 0 = β 0 ^ ∑ i = 1 N u i ^ − β 1 ^ ∑ i = 1 N u i ^ x i = 0 s o   t h a t   w i t h ( 4 ) , S S T = S S R + S S E \begin{equation*} \begin{aligned} \sum_{i=1}^N\hat{u_i}(\hat{y_i}-\bar{y_i})&=\sum_{i=1}^N\hat{u_i}\hat{y_i}-\bar{y}\sum_{i=1}^N\hat{u_i} \\&=\sum_{i=1}^N\hat{u_i}(\hat{\beta_0}-\hat{\beta_1}x_i)-0 \\&=\hat{\beta_0}\sum_{i=1}^N\hat{u_i}- \hat{\beta_1}\sum_{i=1}^N\hat{u_i}x_i \\&=0 \\ \\so \ that \ with (4)&,SST=SSR+SSE \end{aligned} \end{equation*} i=1∑N​ui​^​(yi​^​−yi​ˉ​)so that with(4)​=i=1∑N​ui​^​yi​^​−yˉ​i=1∑N​ui​^​=i=1∑N​ui​^​(β0​^​−β1​^​xi​)−0=β0​^​i=1∑N​ui​^​−β1​^​i=1∑N​ui​^​xi​=0,SST=SSR+SSE​​ 我们定义拟合优度为 R 2 = S S E S S T = 1 − S S R S S T R^2=\frac{SSE}{SST}=1-\frac{SSR}{SST} R2=SSTSSE​=1−SSTSSR​，同时拟合优度亦可通过相关系数进行计算 R 2 = c o r r ( y , y ^ ) = c o r r ( x , y ) R^2=corr(y,\hat{y})=corr(x,y) R2=corr(y,y^​)=corr(x,y)。

显而易见 c o r r ( y , y ^ ) = c o r r ( y , β 1 ^ x + β 0 ^ ) = c o r r ( x , y ) corr(y,\hat{y})=corr(y,\hat{\beta_1}x+\hat{\beta_0})=corr(x,y) corr(y,y^​)=corr(y,β1​^​x+β0​^​)=corr(x,y)（对某一数据线性变换不影响相关性） d e f i n e d   b y   R 2 = S S E S S T = ∑ i = 1 N ( y ^ i − y ˉ ) 2 ∑ i = 1 N ( y i − y ˉ ) 2 = ∑ i = 1 N ( β 1 ^ x + β 0 ^ − β 1 ^ x ˉ − β 0 ^ ) 2 ∑ i = 1 N ( y i − y ˉ ) 2 = ∑ i = 1 N ( β 1 ^ x − β 1 ^ x ˉ ) 2 ∑ i = 1 N ( y i − y ˉ ) 2 = v a r 2 ( β 1 ^ x ) v a r 2 ( y ) = β 1 ^ 2 v a r ( x ) v a r ( y ) = c o v 2 ( x , y ) v a r 2 ( x ) × v a r ( x ) v a r ( y ) = [ c o v ( x , y ) v a r ( x ) v a r ( y ) ] 2 = c o r r 2 ( x , y )   s o   c e r t i f i e d   t h a t   R 2 = c o r r 2 ( y , y ^ ) = c o r r 2 ( x , y ) \begin{equation*} \begin{aligned} defined\ by\ R^2&=\frac{SSE}{SST}=\frac{\sum_{i=1}^{N}(\hat{y}_i-\bar{y})^2}{\sum_{i=1}^{N}(y_i-\bar{y})^2} \\&=\frac{\sum_{i=1}^{N}(\hat{\beta_1}x+\hat{\beta_0}-\hat{\beta_1}\bar{x}-\hat{\beta_0})^2}{\sum_{i=1}^{N}(y_i-\bar{y})^2} \\&=\frac{\sum_{i=1}^{N}(\hat{\beta_1}x-\hat{\beta_1}\bar{x})^2}{\sum_{i=1}^{N}(y_i-\bar{y})^2} \\&=\frac{var^2(\hat{\beta_1}x)}{var^2(y)} \\&=\hat{\beta_1}^2\frac{var(x)}{var(y)} \\&=\frac{cov^2(x,y)}{var^2(x)}\times\frac{var(x)}{var(y)} \\&=[\frac{cov(x,y)}{\sqrt{var(x)var(y)}}]^2 \\&=corr^2(x,y) \\ \ so\ certified \ that \ R^2&=corr^2(y,\hat{y})=corr^2(x,y) \end{aligned} \end{equation*} defined by R2 so certified that R2​=SSTSSE​=∑i=1N​(yi​−yˉ​)2∑i=1N​(y^​i​−yˉ​)2​=∑i=1N​(yi​−yˉ​)2∑i=1N​(β1​^​x+β0​^​−β1​^​xˉ−β0​^​)2​=∑i=1N​(yi​−yˉ​)2∑i=1N​(β1​^​x−β1​^​xˉ)2​=var2(y)var2(β1​^​x)​=β1​^​2var(y)var(x)​=var2(x)cov2(x,y)​×var(y)var(x)​=[var(x)var(y) ​cov(x,y)​]2=corr2(x,y)=corr2(y,y^​)=corr2(x,y)​​

我们所求参数 β 1 ^ \hat{\beta_1} β1​^​与 β 0 ^ \hat{\beta_0} β0​^​具有无偏性。以下将给出证明。

k n o w n    t h a t     β ^ 1 = ∑ i = 1 N ( x − x ˉ ) ( y i − y ˉ ) ∑ i = 1 N ( x − x ˉ ) 2     a n d    y i − y ˉ = β 1 ( x i − x ˉ ) + ( u i − u ˉ ) \begin{equation*} \begin{aligned} known \ \ that \ \ \ \hat{\beta}_1=\frac{\sum_{i=1}^N(x-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^N(x-\bar{x})^2} \tag{5} \ \ \ and \ \ y_i-\bar{y}=\beta_1(x_i-\bar{x})+(u_i-\bar{u}) \end{aligned} \end{equation*} known  that   β^​1​=∑i=1N​(x−xˉ)2∑i=1N​(x−xˉ)(yi​−yˉ​)​   and  yi​−yˉ​=β1​(xi​−xˉ)+(ui​−uˉ)​​(5)​ 待更新，有问题请评论区说明