对Kaldi nnet3进行奇异值分解（SVD）以减小模型大小

Posted 2022-01-29 jarvanwang

用处

基于SVD实现模型压缩以适配低功耗平台

? ?

根据nnet3bin/nnet3-copy，nnet3-copy或nnet3-am-copy的"--edits-config"参数中，新支持了以下选项：

apply-svd name=<name-pattern> bottleneck-dim=<dim>

查找所有名字与<name-pattern>匹配的组件，类型需要是AffineComponent或其子类。如果<dim>小于组件的输入或输出维数，则对组件参数进行奇异值分解，只保留最大<dim>奇异值，将这些组件替换为两个组件：LinearComponent和NaturalGradientAffineComponent（的序列）。又见‘reduce-rank‘。

? ?

示例cd

dir=`mktemp -d`

nnet3-am-copy --edits=‘apply-svd name=*.affine bottleneck-dim=64‘ $dir/final.mdl $dir/final_svd.mdl

? ?

vimdiff <(nnet3-info --print-args=false $dir/final.raw 2>&1|sort) <(nnet3-info --print-args=false $dir/final.raw 2>&1|sort)

? ?

component-node name=tdnn1.affine component=tdnn1.affine input=lda input-dim=195 output-dim=1024

component name=tdnn1.affine type=NaturalGradientAffineComponent, input-dim=195, output-dim=1024, learning-rate=0.00136, max-change=0.75, linear-params-rms=0.4864, linear-params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.92,1.0,1.1,1.2 1.3,1.6,6.4,8.8,10 11,13,14,17), mean=5.79, stddev=3.56], linear-params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(5.0,5.5,8.3,12 13,14,15,17,18 19,20,20,24), mean=15.4, stddev=2.46], bias-mean,stddev=-0.06099,0.2027, rank-in=20, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

component-node name=tdnn1.affine_a component=tdnn1.affine_a input=lda input-dim=195 output-dim=64

component-node name=tdnn1.affine_b component=tdnn1.affine_b input=tdnn1.affine_a input-dim=64 output-dim=1024

component name=tdnn1.affine_a type=LinearComponent, input-dim=195, output-dim=64, learning-rate=0.00136, max-change=0.75, params-rms=0.3461, params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(4.0,4.0,4.0,4.1 4.2,4.3,4.8,5.2,5.5 5.6,5.8,5.8,6.0), mean=4.81, stddev=0.496], params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.36,0.45,0.75,1.9 2.0,2.3,2.7,3.2,3.4 3.6,3.7,3.9,4.7), mean=2.7, stddev=0.618], use-natural-gradient=true, rank-in=40, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

component name=tdnn1.affine_b type=NaturalGradientAffineComponent, input-dim=64, output-dim=1024, learning-rate=0.00136, max-change=0.75, linear-params-rms=0.151, linear-params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.12,0.13,0.14,0.16 0.19,0.23,1.1,1.6,1.8 2.1,2.5,2.6,3.8), mean=1, stddev=0.676], linear-params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(4.0,4.0,4.0,4.1 4.2,4.3,4.8,5.2,5.5 5.6,5.8,5.8,6.0), mean=4.81, stddev=0.496], bias-mean,stddev=-0.06099,0.2027, rank-in=20, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

结果

经过解码测试，SVD后的模型识别率极差，完全无法使用。

需要再对模型进行retrain。

? ?

使用SVD实现模型压缩后，再进行几轮迭代

在已有训练样本的情况在，假设总iteration=2000，，将final.mdl进行SVD得到final_svd.mdl，再链接为0.mdl，运行一个epoch：local/chain/run_tdnn.sh --stage 16 --num_epochs 1

在之前的epoch的基础上，再训几个epochs

local/chain/run_tdnn.sh --stage 16 --num_epochs 2 --train_stage 1155