webrtc音视频解析流程分析

Posted 2020-09-02

webrtc音视频解析流程包括多个线程：

1. rtp网络流接收线程（rtp stream reciever thread）

2. 音视频解码线程（decode thread）

3. 渲染线程（render thread）

 

rtp网络流接收线程（rtp stream reciever thread）：

接收网络rtp包，解析rtp包，得到音视频数据包。将解析出的rtp包，加入到RtpStreamReceiver::frame_buffer_中或最终加入VCMReceiver::jitter_buffer_，解码线程从frame_buffer_或jitter_buffer_中取出帧进行解码。

涉及类：RtpStreamReceiver(rtp_stream_receiver.cc)， VideoReceiveStream（video_receive_stream.cc）

函数调用流程： 

VideoReceiveStream::DeliverRtp() =>

rtp_stream_receiver_.DeliverRtp (rtp_stream_receiver.cc) =>

RtpStreamReceiver::ReceivePacket(rtp_stream_reciver.cc) =>

rtp_receiver_->IncomingRtpPacket(rtp_receiver_impl.cc) =>

rtp_media_receiver_->ParseRtpPacket(rtp_receiver_video.cc) =>

1. depacketizer->Parse(rtp_format_h264.cc) 解析出payload_type, 如sps和pps等。
2. data_callback_->OnReceivedPayloadData(rtp_stream_reciver.cc) =>

　　　  a)  if(h264) InsertSpsPpsIntoTracker

    　　　　packet_buffer_->InsertPacket

　　　  b) video_receiver_->IncomingPacket(video_receiver.cc) => 

　　　　　_receiver.InsertPacket (receiver.cc) =>

上面a)和b)步骤是或关系，

a)步骤：

packet_buffer_->InsertPacket（packet_buffer.cc） =>

received_frame_callback_->OnReceivedFrame(rtp_stream_receiver.cc) =>

reference_finder->ManageFrame(rtp_frame_reference_finder.cc) =>

RtpFrameReferenceFinder::ManageFrame =>

a) RtpFrameReferenceFinder::ManageFrameGeneric =>

b) RtpFrameReferenceFinder::ManageFrameV8=>RtpFrameReferenceFinder::CompletedFrameV8 =>

c) RtpFrameReferenceFinder::ManageFrameV9=>RtpFrameReferenceFinder::CompletedFrameV9 => 

frame_callback_->OnCompleteFrame =>

complete_frame_callback_->OnCompleteFrame =>

frame_buffer_->InsertFrame

 

b)步骤：

 _receiver.InsertPacket(receiver.cc) =>

jitter_buffer_.InsertPacket(jitter_buffer.cc) =>

decodable_frames_->InsertFrame(jitter_buffer.cc)

 

成员说明：

rtp_receiver_： RtpReceiver, 子类RtpReceiverImpl。

rtp_media_receiver_：RTPReceiverStrategy指针，子类是RTPReceiverAudio和RTPReceiverVideo。

data_callback： RtpData, 子类RtpStreamReceiver。

video_receiver_：VideoReceiver。

packet_buffer_: PacketBuffer。

_receiver： VCMReceiver。

received_frame_callback_: 类RtpStreamReceiver，实现video_coding::OnReceivedFrameCallback。

reference_finder_: 类RtpFrameReferenceFinder。

frame_callback_: 类RtpStreamReceiver，实现video_coding::OnCompleteFrameCallback。

complete_frame_callback_：类VideoReceiveStream， 实现video_coding::OnCompleteFrameCallback。

 

视频解码线程（video decode thread）：

从RtpStreamReceiver::frame_buffer_中读取每一帧进行解码。

涉及类：VideoReceiveStream(video_receive_stream.cc)

函数调用流程：

VideoReceiveStream::Decode(video_receive_stream.cc) =>

　　video_receiver_->Decode(video_receiver.cc) =>

　　　　1. _codecDataBase.GetDecoder(frame, _decodedFrameCallback) =>

　　　　　　ptr_decoder_->RegisterDecodeCompleteCallback(_decodedFrameCallback)

　　　　2. if (frame_buffer_->NextFrame) video_receiver_.Decode(frame) (从frame_buffer_取帧)

　　　　　 else video_receiver_.Decode(kMaxDecodeWaitTimeMs) (从jitter_buffer_取帧) =>

　　　　　　　　_receiver.FrameForDecoding(取帧) =>

　　　　　　　　jitter_buffer_.NextCompleteFrame

  　　　　  _deocoder->Decode() (_decoder是具体的decoder，如h264)=>　　　　　　

　 　　　　decoded_image_callback_->Decoded(generic_decoder.cc) =>

　 　　　　_receiveCallback->FrameToRender(video_stream_decoder.cc) =>

　 　　　　incoming_video_stream_->OnFrame(incoming_video_stream.cc) =>

　　　　　 render_buffers_->AddFrame(incoming_video_stream.cc)

　　rtp_stream_receiver_->FrameDecoded(统计)

成员说明：

 video_receiver_：VideoReceiver。

_decodedFrameCallback: 类VCMDecodedFrameCallback。

_receiveCallback: 类VCMReceiveCallback，子类VideoStreamDecoder。

incoming_video_stream_:  rtc::VideoSinkInterface<VideoFrame>*,子类IncomingVideoStream, WebRtcVideoReceiveStream。

 

渲染线程（render thread）：

渲染线程从IncomingVideoStream::render_buffers_ 中读取帧，发送出去。

涉及类：IncomingVideoStream

IncomingVideoStream::IncomingVideoStreamProcess(incoming_video_stream.cc) =>

1. render_buffers_->FrameToRender() =>

2. external_callback_->OnFrame (在VideoReceiveStream::Start 中设置为VideoReceiveStream) =>

config_.renderer->OnFrame (video_receive_stream.cc, WebRtcVideoReceiveStream构造函数中设置config_.render 为自己, 最终传递给VideoReceiveStream::config_)=>

sink_->OnFrame

 

sink设置：

PeerConnection::CreateVideoReceiver(peerconnection.cc) =>

VideoRtpReceiver::VideoRtpReceiver(rtpreceiver.cc) => 

channel_->SetSink(broadcaster_) =>

WebRtcVideoChannel2::SetSink(ssrc, sink) =>

receive_streams_[ssrc]->SetSink(sink)(最终设置WebRtcVideoReceiveStream::sink_)

 

成员说明：

render_buffers_：std::list<VideoRenderFrames>

external_callback_: rtc::VideoSinkInterface<VideoFrame>*。

sink_: rtc::VideoSinkInterface<webrtc::VideoFrame>*, 通过WebRtcVideoReceiveStream::SetSink设置。

broadcaster_: rtc::VideoBroadcaster，继承VideoSinkInterface<webrtc::VideoFrame>，broadcast video frames to sinks 。

receive_streams_: std::map<uint32_t, WebRtcVideoReceiveStream*>。