神经网络—BP算法(反向传播)手算推导
宥名_
编辑于 2023年12月29日 01:56

下面的计算过程如有错误，还请指正

定义一个三层全连接网络，输入层(第0层)，隐藏层(第1层)，输出层(第2层)。无偏置,激活函数为Sigmoid

初始化神经网络如下：

%5Cbegin%7Balign*%7D%0AX%20%26%3D%20Z_0%20%3D%20%5Cbegin%7Bbmatrix%7D%200.35%20%5C%5C%200.9%20%5Cend%7Bbmatrix%7D%20%5C%5C%5B0.5cm%5D%0Ay_%7B%5Ctext%7Bout%7D%7D%20%26%3D%200.5%20%5C%5C%5B0.5cm%5D%0AW_0%20%26%3D%20%5Cbegin%7Bbmatrix%7D%20w_%7B31%7D%20%26%20w_%7B32%7D%20%5C%5C%20w_%7B41%7D%20%26%20w_%7B42%7D%20%5Cend%7Bbmatrix%7D%20%3D%20%5Cbegin%7Bbmatrix%7D%200.1%20%26%200.8%20%5C%5C%200.4%20%26%200.6%20%5Cend%7Bbmatrix%7D%20%5C%5C%5B0.5cm%5D%0AW_1%20%26%3D%20%5Cbegin%7Bbmatrix%7D%20w_%7B53%7D%20%26%20w_%7B54%7D%5Cend%7Bbmatrix%7D%3D%20%5Cbegin%7Bbmatrix%7D%200.3%20%26%200.9%5Cend%7Bbmatrix%7D%0A%5Cend%7Balign*%7D

输入层：对应矩阵X和Z_0；输入到神经元1中的值为0.35，输入到神经元2中的值为0.9

隐藏层：神经元3，神经元4

输出层：神经元5 y_%7B%5Ctext%7Bout%7D%7D%20%3D%200.5

W_0：前向传播时输入层和隐藏层之间的权重参数。w_%7B31%7D为神经元1和神经元3之间的权重，w_%7B32%7D为神经元2和神经元3之间的权重，w_%7B41%7D为神经元1和神经元4之间的权重，w_%7B42%7D为神经元2和神经元4之间的权重

W_1：前向传播时隐藏层和输出层之间的权重参数，w_%7B53%7D为神经元3和神经元5之间的权重，w_%7B54%7D为神经元4和神经元5之间的权重。

单个神经元的结构如下：

Z是该神经元的输入(前一层的输出)与对应权重的线性组合；f是激活函数，这里是Sigmoid函数，对应公式为：f(x)%20%3D%20%5Cfrac%7B1%7D%7B1%20%2B%20e%5E%7B-x%7D%7D ；a是把Z经过激活函数处理后的值，如果没有偏置参数b的话即为该神经元的输出值，有偏置项的话神经元的输出值如 y 所示；

神经网络参数计算和更新流程如下：

1. 前向传播过程计算

正向计算1，从输入层到隐藏层

%5Cbegin%7Balign*%7D%0AZ_1%20%26%20%3D%20%5Cbegin%7Bbmatrix%7D%20z_3%5C%5C%20z_4%5Cend%7Bbmatrix%7D%20%3D%20W_0%20*%20X%20%0A%20%3D%20%5Cbegin%7Bbmatrix%7D%20w_%7B31%7D%20%26%20w_%7B32%7D%5C%5C%20w_%7B41%7D%20%26%20w_%7B42%7D%5Cend%7Bbmatrix%7D%20*%20%5Cbegin%7Bbmatrix%7D%20x_1%5C%5C%20x_2%5Cend%7Bbmatrix%7D%20%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%20w_%7B31%7D*x_1%20%2B%20w_%7B32%7D*x_2%5C%5C%20w_%7B41%7D*x_1%20%2B%20w_%7B42%7D*x_2%5Cend%7Bbmatrix%7D%20%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.1*0.35%20%2B%200.8*0.9%5C%5C%200.4*0.35%20%2B%200.6*0.9%5Cend%7Bbmatrix%7D%20%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.755%5C%5C%200.68%5Cend%7Bbmatrix%7D%0A%5Cend%7Balign*%7D

%5Cbegin%7Balign*%7D%0AY_1%20%26%20%3D%20%5Cbegin%7Bbmatrix%7D%20y_3%5C%5C%20y_4%5Cend%7Bbmatrix%7D%20%3Df(Z_1)%20%3D%20f(W_0%20*%20X)%20%0A%20%3D%20f%5Cleft(%5Cbegin%7Bbmatrix%7D%20w_%7B31%7D%20%26%20w_%7B32%7D%5C%5C%20w_%7B41%7D%20%26%20w_%7B42%7D%5Cend%7Bbmatrix%7D%20*%20%5Cbegin%7Bbmatrix%7D%20x_1%5C%5C%20x_2%5Cend%7Bbmatrix%7D%5Cright)%20%5C%5C%0A%26%20%3D%20f%5Cleft(%5Cbegin%7Bbmatrix%7D%20w_%7B31%7D*x_1%20%2B%20w_%7B32%7D*x_2%5C%5C%20w_%7B41%7D*x_1%20%2B%20w_%7B42%7D*x_2%5Cend%7Bbmatrix%7D%5Cright)%20%5C%5C%0A%26%20%3D%20f%5Cleft(%5Cbegin%7Bbmatrix%7D%200.755%5C%5C%200.68%5Cend%7Bbmatrix%7D%5Cright)%20%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.680%5C%5C%200.663%5Cend%7Bbmatrix%7D%0A%5Cend%7Balign*%7D%0A

正向计算2，从隐藏层到输出层

%5Cbegin%7Balign*%7D%0Az_2%20%26%20%3D%20W_1%20*%20Y_1%20%0A%3D%20%5Cbegin%7Bbmatrix%7D%20w_%7B53%7D%20%26%20w_%7B54%7D%20%5Cend%7Bbmatrix%7D%20*%20%5Cbegin%7Bbmatrix%7D%20y_3%20%5C%5C%20y_4%20%5Cend%7Bbmatrix%7D%20%5C%5C%0A%26%20%3D%20%20w_%7B53%7D*y_3%20%2B%20w_%7B54%7D*y_4%20%20%5C%5C%0A%26%20%3D%200.801%20%0A%5Cend%7Balign*%7D%0A

%5Cbegin%7Balign*%7D%0Ay_2%20%26%20%3D%20f(Z_2)%20%3D%20f(W_1%20*%20Y_1)%20%0A%20%3D%20f%5Cleft(%20%5Cbegin%7Bbmatrix%7D%20w_%7B53%7D%20%26%20w_%7B54%7D%20%5Cend%7Bbmatrix%7D%20*%20%5Cbegin%7Bbmatrix%7D%20y_3%20%5C%5C%20y_4%20%5Cend%7Bbmatrix%7D%5Cright)%20%5C%5C%0A%26%20%3D%20f(%20w_%7B53%7D*y_3%20%2B%20w_%7B54%7D*y_4%20)%20%5C%5C%0A%26%20%3D%20f(0.801)%20%5C%5C%0A%26%20%3D%200.690%0A%5Cend%7Balign*%7D

则经过第一轮前向传播之后的输出值为0.690

2. 构建损失函数

定义损失函数为：C%20%3D%20%5Cfrac%7B1%7D%7B2%7D%20%5Csum_%7Bj%3D1%7D%5E%7Bn_L%7D%20(a_j%20-%20y_j)%5E2

由于输出层的真实值为0.5，所以第一轮前向传播计算后的损失为C%20%3D%20%5Cfrac%7B1%7D%7B2%7D%20(0.690%20-%200.5)%5E2%20%3D%200.01805

3. 反向传播过程计算

反向传播计算1，从输出层到隐藏层

已知Sigmoid函数的导数 f%26%2339%3B(x)%20%3D(%20%5Cfrac%7B1%7D%7B1%20%2B%20e%5E%7B-x%7D%7D)%26%2339%3B%3Df(x)*(1-f(x))

%5Cbegin%7Balign*%7D%0A%5Cleft%5C%7B%0A%5Cbegin%7Baligned%7D%0AC%20%26%20%3D%20%5Cfrac%7B1%7D%7B2%7D%20(y_2%20-%20y_%7Bout%7D)%5E2%20%5C%5C%0Ay_2%20%26%20%3D%20f(z_2)%20%5C%5C%0Az_2%20%26%20%3D%20(w_%7B53%7D%20*%20y_3%20%2B%20w_%7B54%7D%20*%20y_4)%0A%5Cend%7Baligned%7D%0A%5Cright.%0A%5Cend%7Balign*%7D%0A

求损失函数C对最后一层W中w_%7B53%7D的偏导，根据链式法则:

%5Cbegin%7Balign*%7D%0A%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B53%7D%7D%20%26%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y_2%7D%20*%20%5Cfrac%7B%5Cpartial%20y_2%7D%7B%5Cpartial%20z_2%7D%20*%20%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20w_%7B53%7D%7D%20%5C%5C%0A%26%20%3D%20(y_2%20-%20y_%7Bout%7D)%20*%20f(z_2)%20*%20(1%20-%20f(z_2))%20*%20y_3%20%5C%5C%0A%26%20%3D%20(0.69%20-%200.5)%20*%20(0.69)%20*%20(1%20-%200.69)%20*%200.68%20%5C%5C%0A%26%20%3D%200.02763%0A%5Cend%7Balign*%7D%0A

求损失函数C对最后一层W_1中w_%7B54%7D的偏导，根据链式法则:

%5Cbegin%7Balign*%7D%0A%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B54%7D%7D%20%26%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y_2%7D%20*%20%5Cfrac%7B%5Cpartial%20y_2%7D%7B%5Cpartial%20z_2%7D%20*%20%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20w_%7B53%7D%7D%20%5C%5C%0A%26%20%3D%20(y_2%20-%20y_%7Bout%7D)%20*%20f(z_2)%20*%20(1%20-%20f(z_2))%20*%20y_4%20%5C%5C%0A%26%20%3D%20(0.69%20-%200.5)%20*%200.69%20*%20(1%20-%200.69)%20*%200.663%20%5C%5C%0A%26%20%3D%200.02694%0A%5Cend%7Balign*%7D%0A

反向传播计算2，从隐藏层(神经元3)到输入层

已知%5Cbegin%7Balign*%7D%0A%5Cleft%5C%7B%0A%5Cbegin%7Baligned%7D%0AC%20%26%20%3D%20%5Cfrac%7B1%7D%7B2%7D%20(y_2%20-%20y_%7B%5Ctext%7Bout%7D%7D)%5E2%20%5C%5C%0Ay_2%20%26%20%3D%20f(z_2)%20%5C%5C%0Az_2%20%26%20%3D%20(w_%7B53%7D%20*%20y_3%20%2B%20w_%7B54%7D%20*%20y_4)%20%5C%5C%0Ay_3%20%26%20%3D%20f(z_3)%20%5C%5C%0Az_3%20%26%20%3D%20w_%7B31%7D%20*%20x_1%20%2B%20w_%7B32%7D%20*%20x_2%0A%5Cend%7Baligned%7D%0A%5Cright.%0A%5Cend%7Balign*%7D%0A

求损失函数C对第一层W_0中w_%7B31%7D的偏导，根据链式法则

%5Cbegin%7Balign*%7D%0A%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B31%7D%7D%20%26%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y_2%7D%20*%20%5Cfrac%7B%5Cpartial%20y_2%7D%7B%5Cpartial%20z_2%7D%20*%20%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20y_3%7D%20*%20%5Cfrac%7B%5Cpartial%20y_3%7D%7B%5Cpartial%20z_3%7D%20*%20%5Cfrac%7B%5Cpartial%20z_3%7D%7B%5Cpartial%20w_%7B31%7D%7D%20%5C%5C%0A%26%20%3D%20(y_2%20-%20y_%7Bout%7D)%20*%20f(z_2)%20*%20(1%20-%20f(z_2))%20*%20w_%7B53%7D%20*%20f(z_3)%20*%20(1%20-%20f(z_3))%20*%20x_1%20%5C%5C%0A%26%20%3D(0.69-0.5)*0.69*(1-0.69)*0.3*0.680*(1-0.680)*0.35%20%5C%5C%0A%26%20%3D0.0009286%0A%5Cend%7Balign*%7D%0A

求损失函数C对第一层W_0中w_%7B32%7D的偏导，根据链式法则

%5Cbegin%7Balign*%7D%0A%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B32%7D%7D%20%26%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y_2%7D%20*%20%5Cfrac%7B%5Cpartial%20y_2%7D%7B%5Cpartial%20z_2%7D%20*%20%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20y_3%7D%20*%20%5Cfrac%7B%5Cpartial%20y_3%7D%7B%5Cpartial%20z_3%7D%20*%20%5Cfrac%7B%5Cpartial%20z_3%7D%7B%5Cpartial%20w_%7B32%7D%7D%20%5C%5C%0A%26%20%3D%20(y_2%20-%20y_%7Bout%7D)%20*%20f(z_2)%20*%20(1%20-%20f(z_2))%20*%20w_%7B53%7D%20*%20f(z_3)%20*%20(1%20-%20f(z_3))%20*%20x_2%20%5C%5C%0A%26%20%3D(0.69-0.5)*0.69*(1-0.69)*0.3*0.680*(1-0.680)*0.9%20%5C%5C%0A%26%20%3D0.002388%0A%5Cend%7Balign*%7D%0A

反向传播计算3，从隐藏层(神经元4)到输入层

已知%5Cbegin%7Balign*%7D%0A%5Cleft%5C%7B%0A%5Cbegin%7Baligned%7D%0AC%20%26%20%3D%20%5Cfrac%7B1%7D%7B2%7D%20(y_2%20-%20y_%7B%5Ctext%7Bout%7D%7D)%5E2%20%5C%5C%0Ay_2%20%26%20%3D%20f(z_2)%20%5C%5C%0Az_2%20%26%20%3D%20(w_%7B53%7D%20*%20y_3%20%2B%20w_%7B54%7D%20*%20y_4)%20%5C%5C%0Ay_4%20%26%20%3D%20f(z_4)%20%5C%5C%0Az_4%20%26%20%3D%20w_%7B41%7D%20*%20x_1%20%2B%20w_%7B42%7D%20*%20x_2%0A%5Cend%7Baligned%7D%0A%5Cright.%0A%5Cend%7Balign*%7D%0A

求损失函数C对第一层W中w_%7B41%7D的偏导，根据链式法则

%5Cbegin%7Balign*%7D%0A%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B41%7D%7D%20%26%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y_2%7D%20*%20%5Cfrac%7B%5Cpartial%20y_2%7D%7B%5Cpartial%20z_2%7D%20*%20%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20y_4%7D%20*%20%5Cfrac%7B%5Cpartial%20y_4%7D%7B%5Cpartial%20z_4%7D%20*%20%5Cfrac%7B%5Cpartial%20z_4%7D%7B%5Cpartial%20w_%7B41%7D%7D%20%5C%5C%0A%26%20%3D%20(y_2%20-%20y_%7Bout%7D)%20*%20f(z_2)%20*%20(1%20-%20f(z_2))%20*%20w_%7B54%7D%20*%20f(z_4)%20*%20(1%20-%20f(z_4))%20*%20x_1%20%5C%5C%0A%26%20%3D(0.69-0.5)*0.69*(1-0.69)*0.9*0.663*(1-0.663)*0.35%20%5C%5C%0A%26%20%3D0.002860%0A%5Cend%7Balign*%7D%0A

求损失函数C对第一层W中w_%7B42%7D的偏导，根据链式法则

%5Cbegin%7Balign*%7D%0A%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B42%7D%7D%20%26%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y_2%7D%20*%20%5Cfrac%7B%5Cpartial%20y_2%7D%7B%5Cpartial%20z_2%7D%20*%20%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20y_4%7D%20*%20%5Cfrac%7B%5Cpartial%20y_4%7D%7B%5Cpartial%20z_4%7D%20*%20%5Cfrac%7B%5Cpartial%20z_4%7D%7B%5Cpartial%20w_%7B42%7D%7D%20%5C%5C%0A%26%20%3D%20(y_2%20-%20y_%7Bout%7D)%20*%20f(z_2)%20*%20(1%20-%20f(z_2))%20*%20w_%7B54%7D%20*%20f(z_4)%20*%20(1%20-%20f(z_4))%20*%20x_2%20%5C%5C%0A%26%20%3D(0.69-0.5)*0.69*(1-0.69)*0.9*0.663*(1-0.663)*0.9%20%5C%5C%0A%26%20%3D0.007355%0A%5Cend%7Balign*%7D%0A

4、权重更新

假设学习率 lr%3D0.1，则

(1) 反向传播后最后一层权重参数W_1对应的梯度矩阵G_1= %5Cbegin%7Bbmatrix%7D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B53%7D%7D%20%26%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B54%7D%7D%5Cend%7Bbmatrix%7D

更新W_1：    %5Cbegin%7Balign*%7D%0AW_1%20%26%20%3D%20W_1%20-lr*%20G_1%20%0A%3D%20W_1-0.1*%5Cbegin%7Bbmatrix%7D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B53%7D%7D%20%26%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B54%7D%7D%5Cend%7Bbmatrix%7D%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.3%20%26%200.9%5Cend%7Bbmatrix%7D%20-0.1*%5Cbegin%7Bbmatrix%7D%200.2763%20%26%200.2694%5Cend%7Bbmatrix%7D%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.27237%20%26%200.87306%5Cend%7Bbmatrix%7D%0A%5Cend%7Balign*%7D%0A

(2) 反向传播后第一层权重参数W_0对应的梯度矩阵G_0= %5Cbegin%7Bbmatrix%7D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B31%7D%7D%20%26%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B32%7D%7D%5C%5C%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B41%7D%7D%20%26%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B42%7D%7D%5Cend%7Bbmatrix%7D

更新W_0：    %5Cbegin%7Balign*%7D%0AW_0%20%26%20%3D%20W_0%20-lr*%20G_0%20%0A%3D%20W_0-0.1*%5Cbegin%7Bbmatrix%7D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B31%7D%7D%20%26%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B32%7D%7D%5C%5C%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B41%7D%7D%20%26%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w_%7B42%7D%7D%5Cend%7Bbmatrix%7D%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.1%20%26%200.8%20%5C%5C0.4%20%26%200.6%20%5Cend%7Bbmatrix%7D%20-0.1*%5Cbegin%7Bbmatrix%7D%200.0009286%20%26%200.002388%20%5C%5C%200.002860%20%26%200.007355%5Cend%7Bbmatrix%7D%5C%5C%0A%26%20%3D%20%5Cbegin%7Bbmatrix%7D%200.09990714%20%26%200.7997612%20%5C%5C%200.399714%20%26%200.5992645%5Cend%7Bbmatrix%7D%0A%5Cend%7Balign*%7D%0A

人工智能 神经网络 深度学习 全连接神经网络 BP算法 反向传播
cv28845645