使用javacv录像，同时进行讯飞声纹认证

Posted 2021-01-12 骑猪撞交警

由于最近的demo中需要在活体检测的同时进行音视频录制 ,  尝试使用MediaRecord和camera来录制视频 , 然而Camera.onPreviewFrame 不能与 MediaRecord同时调用。活体检测的原理其实是把camera的预览回调onPreviewFrame(byte[] data, Camera camera) 中的图片数据data作为参数传递到活体检测引擎中去拿返回的检测结果码,由于种种原因 , 不能使用Camera2实现 , 于是通过谷歌了解到javacv这个库可以录制视频 , 下了几个demo , 感觉不仅满足需求 , 录制的视频质量也还可以。使用javacv中的FrameRecorder进行录像，录像的时候，调用record方法写帧数据和音频数据，这时候我们有一个需求，录像的同时，要把声音实时拿过来进行声纹认证。由此产生了2个问题：

问题1：

语音识别用的是讯飞的SDK，要求声音采样率8k或16k。而设置FrameRecorder.setSampleRate(8000)后，再FrameRecorder.start()会报错，报错如下：

avcodec_encode_audio2() error 2: Could not encode audio packet.

 

问题2：

javacv官方录制demo中，从AudioRecord中read到的是ShortBuffer，而讯飞SDK方法要求传入byte，他的方法如下：

public void writeAudio(byte[] data, int start, int length) 

百度谷歌无果，只好自己研究。

• 使用javacv进行录像

下面是使用javacv进行录像的示例代码：

1. 初始化 ffmpeg_recorder

public void initRecorder() {
String ffmpeg_link = parentPath + "/" + "video.mp4";
Log.w(LOG_TAG, "init recorder");

if (yuvIplimage == null) {
yuvIplimage = IplImage.create(cameraManager.getDefaultSize().width,
cameraManager.getDefaultSize().height, IPL_DEPTH_8U, 2);
Log.i(LOG_TAG, "create yuvIplimage");
}

Log.i(LOG_TAG, "ffmpeg_url: " + ffmpeg_link);
recorder = new FFmpegFrameRecorder(ffmpeg_link,
cameraManager.getDefaultSize().width,
cameraManager.getDefaultSize().height, 1);
recorder.setFormat("mp4");
recorder.setSampleRate(sampleAudioRateInHz);
// Set in the surface changed method
recorder.setFrameRate(frameRate);

Log.i(LOG_TAG, "recorder initialize success");

audioRecordRunnable = new AudioRecordRunnable();
audioThread = new Thread(audioRecordRunnable);
try {
recorder.start();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
audioThread.start();
}

 

2. 捕捉摄像头视频数据：

public void onPreviewFrame(byte[] data, Camera camera) {
int during = checkIfMax(new Date().getTime());
/* get video data */
if (yuvIplimage != null && isStart) {
yuvIplimage.getByteBuffer().put(data);
//yuvIplimage = rotateImage(yuvIplimage.asCvMat(), 90).asIplImage();
Log.v(LOG_TAG, "Writing Frame");
try {
System.out.println(System.currentTimeMillis() - videoStartTime);
if (during < 6000) {
recorder.setTimestamp(1000 * during);
recorder.record(yuvIplimage);
}
} catch (FFmpegFrameRecorder.Exception e) {
Log.v(LOG_TAG, e.getMessage());
e.printStackTrace();
}
}
}

3. 捕捉声音数据：

class AudioRecordRunnable implements Runnable {

@Override
public void run() {
android.os.Process
.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);

// Audio
int bufferSize;
short[] audioData;
int bufferReadResult;

bufferSize = AudioRecord.getMinBufferSize(sampleAudioRateInHz,
AudioFormat.CHANNEL_CONFIGURATION_MONO,
AudioFormat.ENCODING_PCM_16BIT);
audioRecord = new AudioRecord(MediaRecorder.Audiosource.MIC,
sampleAudioRateInHz,
AudioFormat.CHANNEL_CONFIGURATION_MONO,
AudioFormat.ENCODING_PCM_16BIT, bufferSize);

audioData = new short[bufferSize];

Log.d(LOG_TAG, "audioRecord.startRecording()");
audioRecord.startRecording();

/* ffmpeg_audio encoding loop */
while (!isFinished) {
// Log.v(LOG_TAG,"recording? " + recording);
bufferReadResult = audioRecord.read(audioData, 0,
audioData.length);
if (bufferReadResult > 0) {
// Log.v(LOG_TAG, "bufferReadResult: " + bufferReadResult);
// If "recording" isn‘t true when start this thread, it
// never get‘s set according to this if statement...!!!
// Why? Good question...
if (isStart) {
try {
Buffer[] barray = new Buffer[1];
barray[0] = ShortBuffer.wrap(audioData, 0,
bufferReadResult);
recorder.record(barray);
// Log.v(LOG_TAG,"recording " + 1024*i + " to " +
// 1024*i+1024);
} catch (FFmpegFrameRecorder.Exception e) {
Log.v(LOG_TAG, e.getMessage());
e.printStackTrace();
}
}
}
}
Log.v(LOG_TAG, "AudioThread Finished, release audioRecord");

/* encoding finish, release recorder */
if (audioRecord != null) {
audioRecord.stop();
audioRecord.release();
audioRecord = null;
Log.v(LOG_TAG, "audioRecord released");
}
}
}

 

解决问题1：

demo中默认设置FrameRecorder.setSampleRate(44100)没问题，想到一个办法，这个地方设置44100，在语音采集的地方设置8000，最后成功了。不过这个计算时间的方法要修改：

public static int getTimeStampInNsFromSampleCounted(int paramInt) {
// return (int) (paramInt / 0.0441D);
return (int) (paramInt / 0.0080D);
}

 

 

解决问题2：

short数组转byte数组，注意数组长度变为原来的2倍

public static byte[] short2byte(short[] sData) {
int shortArrsize = sData.length;
byte[] bytes = new byte[shortArrsize * 2];

for (int i = 0; i < shortArrsize; i++) {
bytes[i * 2] = (byte) (sData[i] & 0x00FF);
bytes[(i * 2) + 1] = (byte) (sData[i] >> 8);
sData[i] = 0;
}
return bytes;

}

 

录制音频源码：

 /**
     * 录制音频的线程
     */
    class AudioRecordRunnable implements Runnable {
        short[] audioData;
        private final AudioRecord audioRecord;
        private int mCount = 0;
        int sampleRate = Constants.AUDIO_SAMPLING_RATE;
 
 
        private AudioRecordRunnable() {
            int bufferSize = AudioRecord.getMinBufferSize(sampleRate,
                    AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
            audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, sampleRate,
                    AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, bufferSize);
            audioData = new short[bufferSize];
 
 
        }
 
 
        /**
         * 包含了音频的数据和起始位置
         *
         * @param buffer
         */
        private void record(Buffer buffer) {
            synchronized (mAudioRecordLock) {
                this.mCount += buffer.limit();
                if (!mIsPause) {
                    try {
                        if (mRecorder != null) {
                            mRecorder.record(sampleRate, new Buffer[]{buffer});
                        }
                    } catch (FrameRecorder.Exception e) {
                        e.printStackTrace();
                    }
                }
            }
        }
 
 
        /**
         * 更新音频的时间戳
         */
        private void updateTimestamp() {
            int i = Util.getTimeStampInNsFromSampleCounted(this.mCount);
            if (mAudioTimestamp != i) {
                mAudioTimestamp = i;
                mAudioTimeRecorded = System.nanoTime();
            }
        }
 
 
        public void run() {
            android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);
            if (audioRecord != null) {
                //判断音频录制是否被初始化
                while (this.audioRecord.getState() == 0) {
                    try {
                        Thread.sleep(100L);
                    } catch (InterruptedException localInterruptedException) {
                    }
                }
                this.audioRecord.startRecording();
                while ((runAudioThread)) {
                    updateTimestamp();
                    int bufferReadResult = this.audioRecord.read(audioData, 0, audioData.length);
                    if (bufferReadResult > 0) {
                        if (recording || (mVideoTimestamp > mAudioTimestamp)) {
                            record(ShortBuffer.wrap(audioData, 0, bufferReadResult));
                        }
                        if (SpeechManager.getInstance().isListening()) {
                            SpeechManager.getInstance().writeAudio(Util.short2byte(audioData), 0, bufferReadResult * 2);
                        }
                    }
                }
                SpeechManager.getInstance().stopListener();
                this.audioRecord.stop();
                this.audioRecord.release();
            }
        }
    }