关于神经网络训练不收敛的问题

[happydezhangning]

最近用 python+numpy 写了一个识别 CIFAR10 的程序，没用框架，但是训练不收敛，训练次数越多，输出每个类别的概率越相同，前向和反向传播逻辑和算法上都没找出什么问题，找了各种办法也没解决，快要被搞得自闭了。想问问有没有遇到过类似问题的大佬，如何解决。。

[baiye23333]

代码都不放你让我们怎么说？

[happydezhangning]

@baiye23333
import math
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
np.set_printoptions(threshold=np.inf)
#open file
def unpickle(file):
import pickle
with open(file, 'rb') as fo:
dict = pickle.load(fo,encoding='bytes')
return dict

dict1 = unpickle('cifar-10-batches-py/data_batch_1')
dict2 = unpickle('cifar-10-batches-py/data_batch_2')
dict3 = unpickle('cifar-10-batches-py/data_batch_3')
dict4 = unpickle('cifar-10-batches-py/data_batch_4')
dict5 = unpickle('cifar-10-batches-py/data_batch_5')
test_dict = unpickle("cifar-10-batches-py/test_batch")

data1_4d = np.reshape(dict1[b'data'],(10000, 32, 32, 3), order = 'F')
data1_4d = np.rot90(data1_4d, k = 3, axes = (1,2))
data2_4d = np.reshape(dict2[b'data'],(10000, 32, 32, 3), order = 'F')
data2_4d = np.rot90(data2_4d, k = 3, axes = (1,2))
data3_4d = np.reshape(dict3[b'data'], (10000, 32, 32, 3), order = 'F')
data3_4d = np.rot90(data3_4d, k = 3, axes = (1,2))
data4_4d = np.reshape(dict4[b'data'], (10000, 32, 32, 3), order = 'F')
data4_4d = np.rot90(data4_4d, k = 3, axes = (1,2))
data5_4d = np.reshape(dict5[b'data'], (10000, 32, 32, 3), order = 'F')
data5_4d = np.rot90(data5_4d, k = 3, axes = (1,2))
test_data = np.reshape(test_dict[b'data'], (10000, 32, 32, 3), order = 'F')
test_data = np.rot90(test_data, k = 3, axes = (1,2))

label1 = dict1[b'labels']
label2 = dict2[b'labels']
label3 = dict3[b'labels']
label4 = dict4[b'labels']
label5 = dict5[b'labels']
test_label = test_dict[b'labels']

def softmax(x):
#减去最大值
x-=np.max(x)
x = np.exp(x)/np.sum(np.exp(x))
return x
#权值参数初始化
weight = np.random.normal(loc = 0,scale = 0.01,size = (3,3,3,16))
bias = np.zeros([16],dtype = np.float64)
conv_out = np.zeros([30,30,16],dtype = np.float64)
Maxpool_out = np.zeros([15,15,16],dtype = np.float64)
weight_of_fc = np.random.uniform(0,0.1,size = (3600,10))
fc_in = np.zeros([1,3600],dtype = np.float64)
softmax_out = np.zeros([1,10],dtype = np.float64)
Relu_out = np.zeros([30,30,16],dtype = np.float64)
dl_div_weight = np.zeros([3,3,3,16],dtype = np.float64)
dl_div_bias = np.zeros([16],dtype = np.float64)


def fc_forward(in_pic):
global conv_out, weight, Maxpool_out, bias, Relu_out ,softmax_out
global weight_of_fc, fc_in, dl_div_weight,dl_div_bias
#卷积操作，Convolutional layer Apply 16 flters with size 3 × 3 × 3,
#stride 1 and padding 0，Layer input 32 × 32 × 3, output 30 × 30 × 16.
for i in range (16):
for j in range(30):
for k in range(30):
conv_out[j][k][i] = (in_pic[j:j+3,k:k+3,0] * weight[:,:,0,i]).sum()+ \
(in_pic[j:j+3,k:k+3,1] * weight[:,:,1,i]).sum()+ \
(in_pic[j:j+3,k:k+3,2] * weight[:,:,2,i]).sum()
conv_out += bias
Relu_out = np.choose(conv_out < 0 ,(conv_out,0))#激活函数
for i in range(16):#池化层
for j in range(15):
for k in range(15):
Maxpool_out[j][k][i] = np.max(Relu_out[j*2:j*2+2,k*2:k*2+2,i])
fc_in = np.reshape(Maxpool_out,(1,3600))
fc_out = np.dot(fc_in,weight_of_fc)
softmax_out = softmax(fc_out)
return (np.argmax(fc_out))
#损失函数，交叉熵
#loss =y*np.logp(标签索引对应的)
def back_forward(inputs,label):#优化卷积层和池化层的参数
global conv_out, weight, Maxpool_out, bias, Relu_out ,softmax_out
global weight_of_fc, fc_in, dl_div_weight,dl_div_bias
for index,input_picture in enumerate(inputs):
num_predict = fc_forward(input_picture)
print("softmax_out : ", softmax_out)
print("预测结果: ",num_predict,"真实值: ",label[index])
#loss 对全连接层输出的偏导 p-y,此时 softmax_out 为 dl_div_dz，z 为全连接最后输出
softmax_out[0][label[index]] -= 1

dw_fc = np.dot(np.transpose(fc_in),softmax_out)
#将 fc_in 转为 3600*1，softmax_out 为 1*10，dw_fc 为 3600*10
dl_div_dfc3600 = np.dot(softmax_out,np.transpose(weight_of_fc))
#weight_of_fc 为 3600*10，dl/dz=softmax_out 为 1*10，dl_div_dfc3600：1*3600·
dl_div_dMaxpool_out = np.reshape(dl_div_dfc3600,(15,15,16))

#求对激活层输出(池化层输入)的偏导:
dl_div_dRelu_out = np.zeros([30,30,16],dtype = np.float64)
for i in range(16):
for j in range(15):
for k in range(15):
if Maxpool_out[j][k][i] == Relu_out[j*2][k*2][i]:
dl_div_dRelu_out[j*2][k*2][i] = dl_div_dMaxpool_out[j][k][i]
elif Maxpool_out[j][k][i] == Relu_out[j*2+1][k*2][i]:
dl_div_dRelu_out[j*2+1][k*2][i] = dl_div_dMaxpool_out[j][k][i]
elif Maxpool_out[j][k][i] == Relu_out[j*2][k*2+1][i]:
dl_div_dRelu_out[j*2][k*2+1][i] = dl_div_dMaxpool_out[j][k][i]
else:
dl_div_dRelu_out[j*2+1][k*2+1][i] = dl_div_dMaxpool_out[j][k][i]
#loss 对 relu(input)即 conv_out 的偏导
dReluout_div_convout = np.choose(conv_out >= 0,(0,1))#reluout 对 reluin 的偏导
dl_div_convout = dReluout_div_convout * dl_div_dRelu_out #30*30*16

#loss 对卷积层 w 和 bias 的偏导
for i in range(16):
for j in range(3):
for k in range(3):
for m in range(3):
dl_div_weight[k,m,j,i] = \
(input_picture[k:k+30,m:m+30,j] * dl_div_convout[:,:,i]).sum()
dl_div_bias[i] = dl_div_convout[:,:,i].sum()
weight_of_fc =weight_of_fc - 0.001 * dw_fc
weight = weight - 0.001 * dl_div_weight
bias =bias - 0.001 * dl_div_bias
def train():
back_forward(data1_4d,label1)
back_forward(data2_4d,label2)
back_forward(data3_4d,label3)
back_forward(data4_4d,label4)
back_forward(data5_4d,label5)
train()

[happydezhangning]

@baiye23333 模型要求
1.2 Network Architecture
Implement a neural network with layers in the following order:
Input Image size 32 × 32 × 3.
Convolutional layer Apply 16 flters with size 3 × 3 × 3, stride 1 and padding 0.
Layer input 32 × 32 × 3, output 30 × 30 × 16.
ReLU layer Apply ReLU activation function on each component.
Layer input 30 × 30 × 16, output 30 × 30 × 16.
Pooling layer Max-pooling with size 2 × 2 and stride 2.
Layer input 30 × 30 × 16, output 15 × 15 × 16.
Fully-connected layer Reshape the data to a vector with length 3600 and fully connect to 10 output nodes.
Layer input 15 × 15 × 16 = 3600, output size 10.
Softmax layer Apply softmax function to get the fnal output, indicating the probability in each category.
Layer input size 10, output size 10.
Here you should calculate the forward and backward propagation by yourself.

[honist]

dpaste.de

[happydezhangning]

https://dpaste.de/yNoH

[baiye23333]

你可视化每一层的梯度试试，高斯初始化只适用层数较少的情况，你看看每层的梯度是不是快接近 0 了。

[baiye23333]

然后你再对数据进行归一化，我个人认为是梯度消失了。

但我觉得你的代码写的好烂。。。

[ypw]

https://dpaste.de/O6JB

目测 batch_size=1，你可以看看这个知乎问题，batch_size=1 的时候是不收敛的： https://www.zhihu.com/question/32673260/answer/71137399

https://pic1.zhimg.com/d6fb7abbaeef80e739d824582a0fa384_r.jpg

另外你可以先用 pytorch 跑通整个流程，https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html，然后再用 numpy 去写细节。

[yemenchun1]

weight 0.001 --> 0.00001

[honist]

lz 你这粘完了和没粘一样啊

参考 8 楼和 9 楼吧 单次只有一个样本 很难确定梯度方向；学习率太大了，也会导致权值变化过大

增大 batch 和 学习率；
增大 batch 就是使用多个样本计算 loss 来更新一次 weight

[ipwx]

Mini-batch 的本质是，通过有限个样本计算 x 上的期望。这叫做蒙特卡洛积分。

https://en.wikipedia.org/wiki/Monte_Carlo_method

真正的目标函数是 E[log p(y|x)]，而你的 log p(y|x) = log Bernoulli-likelihood(y|x)。

而 1-mini batch 会让这个期望估计地非常不准，方差极大，以至于大部分情况下估计的期望所求出的梯度，连方向都是错的，自然就根本不可能收敛。

这就是 mini-batch 不能太小的数学原理。

- - - -

另外，我看你好像没用 log Bernoulli-likelihood(y|x)，而是用了 Bernoulli-likelihood(y|x)（没开 log ）。

这是不妥当的，会带来数值问题，容易出 NaN。

你应该用 log_softmax(t) = t - t_max - log(sum(exp(t - t_max)))

[ipwx]

错了，不是 Bernoulli-likelihood，是 Categorical likelihood。

https://en.wikipedia.org/wiki/Categorical_distribution

[zzj0311]

能不能先把缩进给加上。。cifar 10 的学习率一般要在 1e-3 -4 这种量级，batch size32/64 会好一点 搞个三层的基本就有个 60 -70%的正确率了（你现在导出的这一坨我是看不懂

[EscYezi]

歪个楼，代码片段建议使用 gist