CupCnn的最近的一些更新情况

Posted 阳光玻璃杯

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了CupCnn的最近的一些更新情况相关的知识,希望对你有一定的参考价值。

CupCnn是一个用java写的卷积神经网络。

支持L1、L2正则化

正则化的理论非常复杂,推导过程也比较繁琐,但是实现确实异常的容易,主要体现在权重的衰减。通俗的讲,就是我们每次在更新权重w的时候,可以的让他比应该的大小减小一点。

	// TODO Auto-generated method stub
			float[] wData = w.getData();
			float[] gradData = gradient.getData();
			if(mode==GMode.L2) 
				for(int j=0;j<w.getSize();j++)
					//添加l2衰减
					wData[j] = (1.0f-lr*lamda)*wData[j]  - lr*gradData[j];
				
			else if(mode==GMode.L1)
				for(int j=0;j<w.getSize();j++)
					//添加l1衰减
					if(wData[j]>=0) 
						wData[j] = wData[j] - lr*lamda - lr*gradData[j];
					else 
						wData[j] = wData[j] + lr*lamda - lr*gradData[j];
					
								
			else 
				for(int j=0;j<w.getSize();j++)
					wData[j] -= lr*gradData[j];
				
			

这里lamda是一个很小的数,你可以自己根据实际情况设置,设置是在创建Optimizer的时候指定的,比如在SGDOptimizer的构造函数中指定:

public SGDOptimizer(float lr,Optimizer.GMode mode,float lamda)
		super(lr,mode,lamda);
	

参数解释如下:

  1. lr 学习速率
  2. mode L1或者L2
  3. 衰减因子

也可以不使用正则化,直接用如下构造函数即可:

public SGDOptimizer(float lr)
		super(lr);
	

该构造函数只需传入学习速度。

实现标准卷积

之前实现的卷积其实不能称之为是标准卷积,它更像是深度可分离卷积。因此,我重新实现了标准卷积并将原来的卷积改为了深度可分离卷积。分别命名为Conv2dLayer和DeepWiseConv2dLayer。

重构了各个层的输入参数

以前,使用BlobParams最为传入参数,很不自由:

InputLayer layer1 = new InputLayer(network,new BlobParams(network.getBatch(),1,28,28));
		network.addLayer(layer1);
		
		ConvolutionLayer conv1 = new ConvolutionLayer(network,new BlobParams(network.getBatch(),6,28,28),new BlobParams(1,6,3,3));
		conv1.setActivationFunc(new ReluActivationFunc());
		network.addLayer(conv1);
		
		PoolMaxLayer pool1 = new PoolMaxLayer(network,new BlobParams(network.getBatch(),6,14,14),new BlobParams(1,6,2,2),2,2);
		network.addLayer(pool1);
		
		ConvolutionLayer conv2 = new ConvolutionLayer(network,new BlobParams(network.getBatch(),12,14,14),new BlobParams(1,12,3,3));
		conv2.setActivationFunc(new ReluActivationFunc());
		network.addLayer(conv2);

重构后:

InputLayer layer1 =  new InputLayer(network,28,28,1);
		network.addLayer(layer1);
		
		Conv2dLayer conv1 = new Conv2dLayer(network,28,28,1,6,3,1);
		conv1.setActivationFunc(new ReluActivationFunc());
		network.addLayer(conv1);
		
		PoolMaxLayer pool1 = new PoolMaxLayer(network,28,28,6,2,2);
		network.addLayer(pool1);
		
		//这里尝试使用深度可分离卷积神经网络
		DeepWiseConv2dLayer dwconv1 = new DeepWiseConv2dLayer(network,14,14,6,6,3,1);
		network.addLayer(dwconv1);
		
		Conv2dLayer conv2 = new Conv2dLayer(network,14,14,6,6,1,1);
		conv2.setActivationFunc(new ReluActivationFunc());
		network.addLayer(conv2);
		
		PoolMeanLayer pool2 = new PoolMeanLayer(network,14,14,6,2,2);
		network.addLayer(pool2);
		
		
		FullConnectionLayer fc1 = new FullConnectionLayer(network,7*7*6,256);
		fc1.setActivationFunc(new ReluActivationFunc());
		network.addLayer(fc1);
		
		FullConnectionLayer fc2 = new FullConnectionLayer(network,256,10);
		fc2.setActivationFunc(new ReluActivationFunc());
		network.addLayer(fc2);
		
		SoftMaxLayer sflayer = new SoftMaxLayer(network,10);
		network.addLayer(sflayer);

如上构建一个深度可分离卷积,参数更加简单易懂。

新增cifar10的例子

cifar10的例子验证了CupCnn在处理彩色图像的时候也表现良好。

新增MSELoss

MSELoss是最简单的二次误差损失函数,我发现它比LogLikeLoss表现好,所以新增了它。

public class MSELoss extends Loss

	@Override
	public float loss(Blob label, Blob output) 
		// TODO Auto-generated method stub
		float[] labelData = label.getData();
		float[] outputData = output.getData();
		float loss = 0.0f;
	    for (int i = 0; i < label.getSize(); ++i) 
	    	loss += (labelData[i] - outputData[i]) * (labelData[i] - outputData[i]);
	     

	    return loss /label.getHeight();
	

	@Override
	public void diff(Blob label, Blob output, Blob diff) 
		// TODO Auto-generated method stub
		float[] labelData = label.getData();
		float[] outputData = output.getData();
		float[] diffData = diff.getData();
		int width = label.getWidth();
		int height = label.getHeight();
		float factor = 2;
		diff.fillValue(0.0f);
		for(int n=0;n<height;n++)
			for(int os=0;os<width;os++)
				diffData[n*width+os] += factor*(outputData[n*width+os]-labelData[n*width+os]);
			
		
	

新增线程池并发加速

没有实现多线程之前,确实训练很慢。所以还是给CupCnn增加了线程池并发加速,使用方式极其简单,只需要在常见Network后,设置线程池的数目即可:

	network = new Network();
	network.setThreadNum(6);

新增SGDMOptimizer

SGDMOptimizer就是使用了动量思想的SGDOptimizer。所谓动量就是这次改变参数,不光要和当前的梯度相关,还要和上一次的梯度相关。相关性主要通过momentum参数决定,momentum越接近1,相关性却大,越接近0,相关性越小。

package cupcnn.optimizer;

import java.util.HashMap;
import java.util.List;

import cupcnn.data.Blob;
import cupcnn.optimizer.Optimizer.GMode;
/*
 * SGD with momentum
 */
public class SGDMOptimizer extends Optimizer 
	
	private float momentum = 0.9f;
	private HashMap<Blob,Blob> privMap = new HashMap();
	
	public SGDMOptimizer(float lr,float mententum)
		super(lr);
		this.momentum = mententum;
	

	/*
	 * lamda是衰减权重,是一个很小的数字
	 * */
	
	public SGDMOptimizer(float lr,Optimizer.GMode mode,float lamda,float mententum)
		super(lr,mode,lamda);
		this.momentum = mententum;
	

	@Override
	public void updateB(Blob b, Blob gradient) 
		// TODO Auto-generated method stub
		Blob priv = privMap.get(b);
		if(priv == null) 
			priv = new Blob(b,false);
			privMap.put(b, priv);
		
		
		float[] privData = priv.getData();
		float[] bData = b.getData();
		float[] gradData = gradient.getData();
		for(int j=0;j<b.getSize();j++)
			float V = momentum*privData[j]-lr*gradData[j];
			bData[j] += V;
			privData[j] = V;
		
	
	@Override
	public void updateW(Blob w, Blob gradient) 
		// TODO Auto-generated method stub
		Blob priv = privMap.get(w);
		if(priv == null) 
			priv = new Blob(w,false);
			privMap.put(w, priv);
		
		float[] privData = priv.getData();
		float[] wData = w.getData();
		float[] gradData = gradient.getData();
		if(mode==GMode.L2) 
			for(int j=0;j<w.getSize();j++)
				//添加l2衰减
				float V = momentum*privData[j]-lr*lamda*wData[j]  - lr*gradData[j];
				wData[j] += V;
				privData[j] = V;
			
		else if(mode==GMode.L1)
			for(int j=0;j<w.getSize();j++)
				//添加l1衰减
				float V = 0;
				if(wData[j]>=0) 
					V = momentum*privData[j] - lr*lamda  - lr*gradData[j];
				else 
					V = momentum*privData[j] + lr*lamda - lr*gradData[j];
				
				wData[j] += V;
				privData[j] = V;
							
		else 
			for(int j=0;j<w.getSize();j++)
				float V = momentum*privData[j]-lr*gradData[j];
				wData[j] += V;
				privData[j] = V;
			
		
	


所有的double改为了float

double改为float应该能让神经网络运行速度更快一点。

修改了训练时测试的逻辑

之前是没隔一段时间,将刚才拿来训练的那一批数据(一个batch)拿来测试,测试速度很快,而且往往准确度很高。但是训练完后在测试集上一测试发现准确率一般,所以,干脆改为了没训练一个epoe,就在测试集上进行一次完整的测试,这样能更好的观察训练的情况,也更容易发现过拟合的情况。

最后

CupCnn在github中上传了mnist和cifar10的数据集,因此clone的时候会比较慢,但是好处是,clone后即可以训练与测试。
CupCnn在mnist上能轻松达到98%的准确率,可能只需要4~5个epoe就能达到。如果你有耐心的话,99%的准确率也是没问题的。比如,以下是一个达到99%的模型:

	private void buildConvNetwork()
		InputLayer layer1 =  new InputLayer(network,28,28,1);
		network.addLayer(layer1);
		
		Conv2dLayer conv1 = new Conv2dLayer(network,28,28,1,10,3,1);
		conv1.setActivationFunc(new ReluActivationFunc());
		network.addLayer(conv1);
		
		PoolMaxLayer pool1 = new PoolMaxLayer(network,28,28,10,2,2);
		network.addLayer(pool1);
		
		Conv2dLayer conv2 = new Conv2dLayer(network,14,14,10,10,3,1);
		conv2.setActivationFunc(new ReluActivationFunc());
		network.addLayer(conv2);
	
		PoolMeanLayer pool2 = new PoolMeanLayer(network,14,14,10,2,2);
		network.addLayer(pool2);
	
		FullConnectionLayer fc1 = new FullConnectionLayer(network,7*7*10,256);
		fc1.setActivationFunc(new ReluActivationFunc());
		network.addLayer(fc1);
		
		FullConnectionLayer fc2 = new FullConnectionLayer(network,256,10);
		fc2.setActivationFunc(new ReluActivationFunc());
		network.addLayer(fc2);
		
		SoftMaxLayer sflayer = new SoftMaxLayer(network,10);
		network.addLayer(sflayer);
		
	
	public void buildNetwork(int numOfTrainData)
		//首先构建神经网络对象,并设置参数
		network = new Network();
		network.setThreadNum(8);
		network.setBatch(20);
		network.setLrAttenuation(0.9f);
		//network.setLoss(new LogLikeHoodLoss());
		//network.setLoss(new CrossEntropyLoss());
		network.setLoss(new MSELoss());
		optimizer = new SGDOptimizer(0.1f);
		network.setOptimizer(optimizer);
		
		//buildFcNetwork();
		buildConvNetwork();

		network.prepare();
	

训练过程中神经网络的输出如下:

training...... please wait for a moment!
............................................................
training...... epoe: 0 lossValue: 0.15509754   lr: 0.1   cost 117179
testing...... please wait for a moment!
test accuracy is 0.8785 correctCount 8785 allCount 10000
............................................................
training...... epoe: 1 lossValue: 0.08268655   lr: 0.089999996   cost 115667
testing...... please wait for a moment!
test accuracy is 0.9761 correctCount 9761 allCount 10000
............................................................
training...... epoe: 2 lossValue: 0.06630586   lr: 0.08099999   cost 114282
testing...... please wait for a moment!
test accuracy is 0.9818 correctCount 9818 allCount 10000
............................................................
training...... epoe: 3 lossValue: 0.015823135   lr: 0.07289999   cost 114677
testing...... please wait for a moment!
test accuracy is 0.986 correctCount 9860 allCount 10000
............................................................
training...... epoe: 4 lossValue: 0.006677883   lr: 0.06560999   cost 115048
testing...... please wait for a moment!
test accuracy is 0.9843 correctCount 9843 allCount 10000
............................................................
training...... epoe: 5 lossValue: 7.9412933E-4   lr: 0.05904899   cost 114657
testing...... please wait for a moment!
test accuracy is 0.9858 correctCount 9858 allCount 10000
............................................................
training...... epoe: 6 lossValue: 0.010126321   lr: 0.05314409   cost 114791
testing...... please wait for a moment!
test accuracy is 0.9863 correctCount 9863 allCount 10000
............................................................
training...... epoe: 7 lossValue: 0.11289799   lr: 0.04782968   cost 116797
testing...... please wait for a moment!
test accuracy is 0.9873 correctCount 9873 allCount 10000
............................................................
training...... epoe: 8 lossValue: 0.007667846   lr: 0.04304671   cost 116323
testing...... please wait for a moment!
test accuracy is 0.9889 correctCount 9889 allCount 10000
............................................................
training...... epoe: 9 lossValue: 0.0069320253   lr: 0.038742036   cost 114736
testing...... please wait for a moment!
test accuracy is 0.9886 correctCount 9886 allCount 10000
............................................................
training...... epoe: 10 lossValue: 0.011851874   lr: 0.03486783   cost 116032
testing...... please wait for a moment!
test accuracy is 0.9893 correctCount 9893 allCount 10000
............................................................
training...... epoe: 11 lossValue: 0.0129155135   lr: 0.03138105   cost 115604
testing...... please wait for a moment!
test accuracy is 0.9892 correctCount 9892 allCount 10000
............................................................
training...... epoe: 12 lossValue: 0.0018233052   lr: 0.028242942   cost 119216
testing...... please wait for a moment!
test accuracy is 0.99 correctCount 9900 allCount 10000
............................................................
training...... epoe: 13 lossValue: 0.0027558927   lr: 0.025418647   cost 119067
testing...... please wait for a moment!
test accuracy is 0.9891 correctCount 9891 allCount 10000
............................................................
training...... epoe: 14 lossValue: 0.0024366616   lr: 0.02287678   cost 115927
testing...... please wait for a moment!
test accuracy is 0.9893 correctCount 9893 allCount 10000
............................................................
training...... epoe: 15 lossValue: 0.014906692   lr: 0.020589102   cost 118666
testing...... please wait for a moment!
test accuracy is 0.9903 correctCount 9903 allCount 10000
............................................................
training...... epoe: 16 lossValue: 0.012857122   lr: 0.018530192   cost 116381
testing...... please wait for a moment!
test accuracy is 0.9892 correctCount 9892 allCount 10000
............................................................
training...... epoe: 17 lossValue: 0.0077277897   lr: 0.016677173   cost 115545
testing...... please wait for a moment!
test accuracy is 0.9894 correctCount 9894 allCount 10000
............................................................
training...... epoe: 18 lossValue: 0.008005977   lr: 0.015009455   cost 115614
testing...... please wait for a moment!
test accuracy is 0.9895 correctCount 9895 allCount 10000
............................................................
training...... epoe: 19 lossValue: 0.005667642   lr: 0.01350851   cost 116124
testing...... please wait for a moment!
test accuracy is 0.9903 correctCount 9903 allCount 10000
............................................................
training...... epoe: 20 lossValue: 0.0015369358   lr: 0.012157658   cost 115027
testing...... please wait for a moment!
test accuracy is 0.9903 correctCount 9903 allCount 10000
............................................................
training...... epoe: 21 lossValue: 0.0027716388   lr: 0.010941892   cost 114647
testing...... please wait for a moment!
test accuracy is 0.9903 correctCount 9903 allCount 10000
............................................................
training...... epoe: 22 lossValue: 3.0824103E-4   lr: 0.009847702   cost 114195
testing...... please wait for a moment!
test accuracy is 0.99 correctCount 9900 allCount 10000
............................................................
training...... epoe: 23 lossValue: 0.013186147   lr: 0.008862932   cost 114099
testing...... please wait for a moment!
test accuracy is 0.9905 correctCount 9905 allCount 10000
............................................................
training...... epoe: 24 lossValue: 3.2178203E-5   lr: 0.007976639   cost 114108
testing...... please wait for a moment!
test accuracy is 0.9903 correctCount 9903 allCount 10000
............................................................
training...... epoe: 25 lossValue: 0.0058679665   lr: 0.007178975   cost 116286
testing...... please wait for a moment!
test accuracy is 0.9902 correctCount 9902 allCount 10000
............................................................
training...... epoe: 26 lossValue: 0.0012587834   lr: 0.0064610774   cost 115560
testing...... please wait for a moment!
test accuracy is 0.9904 correctCount 9904 allCount 10000
............................................................
training...... epoe: 27 lossValue: 0.011831607   lr: 0.0058149695   cost 115372
testing...... please wait for a moment!
test accuracy is 0.9904 correctCount 9904 allCount 10000
............................................................
training...... epoe: 28 lossValue: 1.4093271E-4   lr: 0.0052334727   cost 116815
testing...... please wait for a moment!
test accuracy is 0.9904 correctCount 9904 allCount 10000
............................................................
training...... epoe: 29 lossValue: 8.6647685E-4   lr: 0.0047101253   cost 116404
testing...... please wait for a moment!
test accuracy is 0.9904 correctCount 9904 allCount 10000
begin save model
save model finished
begin load model
load model finished
testing...... please wait for a moment!
test accuracy is 0.9904 correctCount 9904 allCount 10000

交流

机器学习 QQ交流群:704153141

以上是关于CupCnn的最近的一些更新情况的主要内容,如果未能解决你的问题,请参考以下文章

使用卷积神经网络(CupCnn)训练人脸检测模型

java写卷积神经网络---卷积神经网络(CupCnn)的数据结构

深度学习优化算法之动量法[公式推导](MXNet)

CupCnn 添加rnn的实现

CupCnn 添加rnn的实现

java写卷积神经网络---CupCnn简介