Android端WebRTC本地音视频采集流程源码分析
Posted 冬天的毛毛雨
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Android端WebRTC本地音视频采集流程源码分析相关的知识，希望对你有一定的参考价值。
WebRTC源码版本为：org.webrtc:google-webrtc:1.0.32006 本文仅分析Java层源码，在分析之前，先说明一下一些重要类的基本概念。
- MediaSource：WebRTC媒体资源数据来源，它有两个子类：Audiosource（音频资源）、VideoSource（视频资源）；
- MediaStreamTrack：媒体资源轨，一个MediaStreamTrack对应一个MediaSource，创建媒体轨需要MediaSource，同样，它也有两个子类：AudioTrack（音频轨）对应AudioSource、VideoTrack（视频轨）对应VideoSource；
- MediaStream：媒体流，一个媒体流可以添加多条AudioTrack和VideoTrack，一般来说我们对应分别只添加一条。
在使用WebRTC进行音视频通话时，需要构建一个PeerConnectionFactory，这是一个连接工厂，创建本地LocalMediaStream流及客户端PeerConnection等，都需要使用。构建大致代码如下：
//在创建PeerConnectionFactory之前，必须至少调用一次。不得在 PeerConnectionFactory 处于活动状态时调用。
PeerConnectionFactory.initialize(
PeerConnectionFactory.InitializationOptions
.builder(applicationContext).createInitializationOptions()
)
val eglBaseContext = EglBase.create().eglBaseContext
//视频编码工厂
val encoderFactory= DefaultVideoEncoderFactory(eglBaseContext, true, true)
//视频解码工厂
val decoderFactory = DefaultVideoDecoderFactory(eglBaseContext)
val audioDeviceModule = JavaAudioDeviceModule.builder(this)
.setSamplesReadyCallback audioSamples ->
//麦克风输入数据，即通话时LocalMediaStream的Audio的数据，pcm格式，通常用于录音。
.createAudioDeviceModule()
val peerConnectionFactory = PeerConnectionFactory.builder()
.setVideoEncoderFactory(encoderFactory)
.setVideoDecoderFactory(decoderFactory)
.setAudioDeviceModule(audioDeviceModule)
.createPeerConnectionFactory()
若你有需求WebRTC视频编码要启用H.264，请参考Android端WebRTC启用H264编码。
MediaSource
WebRTC媒体资源数据来源基类。
AudioSource（音频源）
通过 peerConnectionFactory.createAudioSource(MediaConstraints) 创建，参数为媒体约束，大致如下：
//音频源
val audioSource = peerConnectionFactory.createAudioSource(createAudioConstraints())
private fun createAudioConstraints(): MediaConstraints
val audioConstraints = MediaConstraints()
//回声消除
audioConstraints.mandatory.add(
MediaConstraints.KeyValuePair(
"googEchoCancellation",
"true"
)
)
//自动增益
audioConstraints.mandatory.add(MediaConstraints.KeyValuePair("googAutoGainControl", "true"))
//高音过滤
audioConstraints.mandatory.add(MediaConstraints.KeyValuePair("googHighpassFilter", "true"))
//噪音处理
audioConstraints.mandatory.add(
MediaConstraints.KeyValuePair(
"googNoiseSuppression",
"true"
)
)
return audioConstraints
事实上音频的输入、输出具体处理在JavaAudioDeviceModule：
package org.webrtc.audio;
/**
* AudioDeviceModule implemented using android.media.AudioRecord as input and
* android.media.AudioTrack as output.
*/
public class JavaAudioDeviceModule implements AudioDeviceModule
...
/**
* 本地麦克风采集的输入数据，android.media.AudioRecord
*/
private final WebRtcAudioRecord audioInput;
/**
* 用于播放通话时对方的音频数据，android.media.AudioTrack
*/
private final WebRtcAudioTrack audioOutput;
...
记录本地麦克风采集的音频数据WebRtcAudioRecord：
package org.webrtc.audio;
import android.media.AudioRecord;
class WebRtcAudioRecord
...
private AudioRecord audioRecord;
/**
* 读取线程
*/
private AudioRecordThread audioThread;
/**
* Audio thread which keeps calling ByteBuffer.read() waiting for audio
* to be recorded. Feeds recorded data to the native counterpart as a
* periodic sequence of callbacks using DataIsRecorded().
* This thread uses a Process.THREAD_PRIORITY_URGENT_AUDIO priority.
*/
private class AudioRecordThread extends Thread
private volatile boolean keepAlive = true;
public AudioRecordThread(String name)
super(name);
public void run()
Process.setThreadPriority(Process.THREAD_PRIORITY_URGENT_AUDIO);
//确保在采集数据状态
assertTrue(audioRecord.getRecordingState() == AudioRecord.RECORDSTATE_RECORDING);
//向外回调开始状态
doAudioRecordStateCallback(AUDIO_RECORD_START);
//存活状态
while(keepAlive)
int bytesRead = audioRecord.read(byteBuffer, byteBuffer.capacity());
if (bytesRead == byteBuffer.capacity())
if (microphoneMute)
//如果麦克风静音，则清空数据
byteBuffer.clear();
byteBuffer.put(WebRtcAudioRecord.this.emptyBytes);
if (keepAlive)
//调用native方法发送数据
nativeDataIsRecorded(nativeAudioRecord, bytesRead);
//向外回调音频数据，pcm格式
if (audioSamplesReadyCallback != null)
byte[] data = Arrays.copyOfRange(.byteBuffer.array(), byteBuffer.arrayOffset(), byteBuffer.capacity() + byteBuffer.arrayOffset());
audioSamplesReadyCallback.onWebRtcAudioRecordSamplesReady(new AudioSamples(audioRecord.getAudioFormat(), audioRecord.getChannelCount(), audioRecord.getSampleRate(), data));
else
String errorMessage = "AudioRecord.read failed: " + bytesRead;
Logging.e("WebRtcAudioRecordExternal", errorMessage);
if (bytesRead ==AudioRecord.ERROR_INVALID_OPERATION)
keepAlive = false;
reportWebRtcAudioRecordError(errorMessage);
try
if (audioRecord != null)
audioRecord.stop();
//向外回调停止状态
doAudioRecordStateCallback(AUDIO_RECORD_STOP);
catch (IllegalStateException e)
Logging.e("WebRtcAudioRecordExternal", "AudioRecord.stop failed: " + e.getMessage());
/*
* 结束读取
*/
public void stopThread()
Logging.d("WebRtcAudioRecordExternal", "stopThread");
this.keepAlive = false;
...
播放接收的音频数据WebRtcAudioTrack（这里不属于采集流程，提了一嘴）：
package org.webrtc.audio;
import android.media.AudioTrack;
class WebRtcAudioTrack
...
private AudioTrack audioTrack;
/**
* 读取线程
*/
private AudioTrackThread audioThread;
/**
* Audio thread which keeps calling AudioTrack.write() to stream audio.
* Data is periodically acquired from the native WebRTC layer using the
* nativeGetPlayoutData callback function.
* This thread uses a Process.THREAD_PRIORITY_URGENT_AUDIO priority.
*/
private class AudioTrackThread extends Thread
private volatile boolean keepAlive = true;
public AudioTrackThread(String name)
super(name);
public void run()
Process.setThreadPriority(Process.THREAD_PRIORITY_URGENT_AUDIO);
//确认状态
assertTrue(audioTrack.getPlayState() == AudioTrack.PLAYSTATE_PLAYING);
//向外回调开始状态
doAudioTrackStateCallback(AUDIO_TRACK_START);
for(int sizeInBytes = byteBuffer.capacity(); keepAlive; byteBuffer.rewind())
//调用native方法将播放数据写入到byteBuffer中
nativeGetPlayoutData(nativeAudioTrack, sizeInBytes);
assertTrue(sizeInBytes <= byteBuffer.remaining());
if (speakerMute)
//扬声器静音，清空数据
byteBuffer.clear();
byteBuffer.put(emptyBytes);
byteBuffer.position(0);
//写入到AudioTrack中，进行播放，pcm格式数据
int bytesWritten = writeBytes(audioTrack, byteBuffer, sizeInBytes);
if (bytesWritten != sizeInBytes)
Logging.e("WebRtcAudioTrackExternal", "AudioTrack.write played invalid number of bytes: " + bytesWritten);
if (bytesWritten < 0)
keepAlive = false;
reportWebRtcAudioTrackError("AudioTrack.write failed: " + bytesWritten);
private int writeBytes(AudioTrack audioTrack, ByteBuffer byteBuffer, int sizeInBytes)
return VERSION.SDK_INT >= 21 ? audioTrack.write(byteBuffer, sizeInBytes, 0) : audioTrack.write(byteBuffer.array(), byteBuffer.arrayOffset(), sizeInBytes);
public void stopThread()
Logging.d("WebRtcAudioTrackExternal", "stopThread");
keepAlive = false;
...
上述的WebRtcAudioRecord和WebRtcAudioTrack两个类都不是public，在WebRTC中只对外暴露出了本地麦克风采集数据，未暴露音频输出数据，如果你有获取WebRTC通话接收到的音频数据需求，请参考Android端WebRTC音视频通话录音-获取音频输出数据。
VideoSource（视频源）
VideoSource是一个视频源，用于二次处理视频数据（VideoProcessor）和调整大小、格式和发送视频数据（具体实现是jni的包装类NativeAndroidVideoTrackSource）。 VideoSource视频的获取需要通过CapturerObserver,视频捕捉器可以通过videoSource.getCapturerObserver()获取捕捉器观察者，将数据传给VideoSource。 先看一下VideoSource创建使用流程：
//视频捕捉器
val videoCapturer = createVideoCapture(context)
videoCapturer?.let capturer->
val videoSource = peerConnectionFactory.createVideoSource(capture.isScreencast)
//使用 SurfaceTexture 创建 WebRTC VideoFrames 的帮助类。为了创建 WebRTC VideoFrames，渲染到 SurfaceTexture。帧将传递给侦听器。
val surfaceTextureHelper =
SurfaceTextureHelper.create("surface_texture_thread", eglBaseContext)
//将surfaceTextureHelper和CapturerObserver传给VideoCapturer
capture.initialize(surfaceTextureHelper, context, videoSource.capturerObserver)
/**
* 创建相机视频捕捉器
*/
private fun createVideoCapture(context: Context): CameraVideoCapturer?
val enumerator: CameraEnumerator = if (Camera2Enumerator.isSupported(context))
Camera2Enumerator(context)
else
Camera1Enumerator()
for (name in enumerator.deviceNames)
if (enumerator.isFrontFacing(name))
return enumerator.createCapturer(name, null)
for (name in enumerator.deviceNames)
if (enumerator.isBackFacing(name))
return enumerator.createCapturer(name, null)
return null
我们来看一下VideoSource关键代码：
package org.webrtc;
public class VideoSource extends MediaSource
private final NativeAndroidVideoTrackSource nativeAndroidVideoTrackSource;
/**
* 视频处理器，可自行对图像数据进行二次处理；
* 如：添加水印、视频旋转、剪裁等。
*/
@Nullable
private VideoProcessor videoProcessor;
private final Object videoProcessorLock = new Object();
private final CapturerObserver capturerObserver = new CapturerObserver()
/**
* 开始捕捉
*/
@Override
public void onCapturerStarted(boolean success)
nativeAndroidVideoTrackSource.setState(success);
synchronized(videoProcessorLock)
isCapturerRunning = success;
if (videoProcessor != null)
videoProcessor.onCapturerStarted(success);
/**
* 停止捕捉
*/
@Override
public void onCapturerStopped()
nativeAndroidVideoTrackSource.setState(false);
synchronized(videoProcessorLock)
isCapturerRunning = false;
if (videoProcessor != null)
videoProcessor.onCapturerStopped();
/**
* 捕捉的视频数据
* @param frame 视频数据
*/
@Override
public void onFrameCaptured(VideoFrame frame)
//应在传送任何帧之前调用此函数，以确定是否应丢弃该帧或裁剪、缩放参数是什么。如果FrameAdaptationParameters#drop为true，则应丢弃该帧，否则应在调用 onFrameCaptured() 之前根据帧适配参数对帧进行适配。
FrameAdaptationParameters parameters = nativeAndroidVideoTrackSource.adaptFrame(frame);
synchronized(videoProcessorLock)
if (videoProcessor != null)
//如果添加了视频处理器，自行处理，直接return。
videoProcessor.onFrameCaptured(frame, parameters);
return;
//根据参数调整帧数据，此步骤会调用VideoFrame.getBuffer().cropAndScale()进行裁剪缩放。
VideoFrame adaptedFrame = VideoProcessor.applyFrameAdaptationParameters(frame, parameters);
if (adaptedFrame != null)
//如果不为空，则通过native方法将帧数据传递过去
nativeAndroidVideoTrackSource.onFrameCaptured(adaptedFrame);
adaptedFrame.release();
;
/**
* 调整输出格式
*/
public void adaptOutputFormat(int width, int height, int fps)
int maxSide = Math.max(width, height);
int minSide = Math.min(width, height);
this.adaptOutputFormat(maxSide, minSide, minSide, maxSide, fps);
public void adaptOutputFormat(int landscapeWidth, int landscapeHeight, int portraitWidth, int portraitHeight, int fps)
this.adaptOutputFormat(new VideoSource.AspectRatio(landscapeWidth, landscapeHeight), landscapeWidth * landscapeHeight, new VideoSource.AspectRatio(portraitWidth, portraitHeight), portraitWidth * portraitHeight, fps);
public void adaptOutputFormat(VideoSource.AspectRatio targetLandscapeAspectRatio, @Nullable Integer maxLandscapePixelCount, VideoSource.AspectRatio targetPortraitAspectRatio, @Nullable Integer maxPortraitPixelCount, @Nullable Integer maxFps)
nativeAndroidVideoTrackSource.adaptOutputFormat(targetLandscapeAspectRatio, maxLandscapePixelCount, targetPortraitAspectRatio, maxPortraitPixelCount, maxFps);
/**
* 设置视频处理器
*/
public void setVideoProcessor(@Nullable VideoProcessor newVideoProcessor)
synchronized(videoProcessorLock)
//判断之前是否设置过
if (videoProcessor != null)
videoProcessor.setSink(null);
if (isCapturerRunning)
videoProcessor.onCapturerStopped();
videoProcessor = newVideoProcessor;
if (newVideoProcessor != null)
newVideoProcessor.setSink(new VideoSink()
@Override
public void onFrame(VideoFrame frame)
//这里返回的是自行处理好的数据，再通过native方法将帧数据传递过去，和上面一样。
runWithReference(() ->
nativeAndroidVideoTrackSource.onFrameCaptured(frame);
);
);
if (isCapturerRunning)
newVideoProcessor.onCapturerStarted(true);
/**
* 获取捕捉器观察者
*/
public CapturerObserver getCapturerObserver()
以上是关于Android端WebRTC本地音视频采集流程源码分析的主要内容，如果未能解决你的问题，请参考以下文章
WebRTC中Android Demo中的摄像头从采集到预览流程
安卓mediasoup webrtc h264 编解码相关源码分析
安卓mediasoup webrtc h264 编解码相关源码分析
安卓mediasoup webrtc h264 编解码相关源码分析