让 WebRTC 使用外部的音视频编解码器

Posted 2020-09-12 foruok

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了让 WebRTC 使用外部的音视频编解码器相关的知识，希望对你有一定的参考价值。

WebRTC 支持使用自己的编解码器（限 native 开发），音频，视频都可以。这里以视频编码为例来分析下 WebRTC 中相应的源码。

CreatePeerConnectionFactory

在 webrtc/api/peerconnectioninterface.h 中有个方法 CreatePeerConnectionFactory，原型如下：

inline rtc::scoped_refptr<PeerConnectionFactoryInterface> 
CreatePeerConnectionFactory(
    rtc::Thread* worker_and_network_thread,
    rtc::Thread* signaling_thread,
    AudioDeviceModule* default_adm,
    rtc::scoped_refptr<AudioEncoderFactory> audio_encoder_factory,
    rtc::scoped_refptr<AudioDecoderFactory> audio_decoder_factory,
    cricket::WebRtcVideoEncoderFactory* video_encoder_factory,
    cricket::WebRtcVideoDecoderFactory* video_decoder_factory);

可以看到， CreatePeerConnectionFactory 最后四个参数，允许我们提供自己的编解码工厂。这样我们就可以实现自己的 factory ，在 factory 中创建自己的 encoder（decoder），WebRTC 内部就会使用我们的 encoder（decoder）。

代码片段类似下面：

    rtc::scoped_refptr<webrtc::PeerConnectionFactoryInterface> factory = webrtc::CreatePeerConnectionFactory(
            current_thread, current_thread, 
            your_adm, 
            your_audio_encoder_factory, 
            your_audio_decoder_factory,
            your_video_encoder_factory,
            your_video_decoder_factory);
    rtc::scoped_refptr<PeerConnectionInterface> peer_connection = factory->CreatePeerConnection(
            config, constraints, NULL, NULL, your_observer);

WebRtcVideoEncoderFactory

这个接口在 webrtcvideoencoderfactory.h（webrtc/media/engine）中定义：

class WebRtcVideoEncoderFactory {
 public:
  // This VideoCodec class is deprecated. Use cricket::VideoCodec directly
  // instead and the corresponding factory function. See
  // http://crbug/webrtc/6402 for more info.
  struct VideoCodec {
    webrtc::VideoCodecType type;
    std::string name;

    VideoCodec(webrtc::VideoCodecType t, const std::string& nm)
        : type(t), name(nm) {}

    VideoCodec(webrtc::VideoCodecType t,
               const std::string& nm,
               int w,
               int h,
               int fr)
        : type(t), name(nm) {}
  };

  virtual ~WebRtcVideoEncoderFactory() {}

  // TODO(magjed): Make these functions pure virtual when every external client
  // implements it. See http://crbug/webrtc/6402 for more info.
  // Caller takes the ownership of the returned object and it should be released
  // by calling DestroyVideoEncoder().
  virtual webrtc::VideoEncoder* CreateVideoEncoder(
      const cricket::VideoCodec& codec);

  // Returns a list of supported codecs in order of preference.
  virtual const std::vector<cricket::VideoCodec>& supported_codecs() const;

  // Caller takes the ownership of the returned object and it should be released
  // by calling DestroyVideoEncoder().
  // Deprecated: Use cricket::VideoCodec as argument instead. See
  // http://crbug/webrtc/6402 for more info.
  virtual webrtc::VideoEncoder* CreateVideoEncoder(webrtc::VideoCodecType type);

  // Returns a list of supported codecs in order of preference.
  // Deprecated: Return cricket::VideoCodecs instead. See
  // http://crbug/webrtc/6402 for more info.
  virtual const std::vector<VideoCodec>& codecs() const;

  // Returns true if encoders created by this factory of the given codec type
  // will use internal camera sources, meaning that they don't require/expect
  // frames to be delivered via webrtc::VideoEncoder::Encode. This flag is used
  // as the internal_source parameter to
  // webrtc::ViEExternalCodec::RegisterExternalSendCodec.
  virtual bool EncoderTypeHasInternalSource(webrtc::VideoCodecType type) const {
    return false;
  }

  virtual void DestroyVideoEncoder(webrtc::VideoEncoder* encoder) = 0;

 private:
  // TODO(magjed): Remove these. They are necessary in order to return a const
  // reference to a std::vector in the default implementations of codecs() and
  // supported_codecs(). See http://crbug/webrtc/6402 for more info.
  mutable std::vector<VideoCodec> encoder_codecs_;
  mutable std::vector<cricket::VideoCodec> codecs_;
};

实现 WebRtcVideoEncoderFactory 接口，传递给 CreatePeerConnectionFactory 即可。可以参考 webrtc/media/engine 目录下的 internalencoderfactory.h 和 internalencoderfactory.cpp 。

外部编解码器与内部的关系

以视频编码为例， webrtc/media/engine/webrtcvideoengine2.cc 中有段代码可以说明外部视频编码器和内部视频编码器之间的关系：

WebRtcVideoChannel2::WebRtcVideoSendStream::AllocatedEncoder
WebRtcVideoChannel2::WebRtcVideoSendStream::CreateVideoEncoder(
    const VideoCodec& codec) {
  RTC_DCHECK_RUN_ON(&thread_checker_);
  // Do not re-create encoders of the same type.
  if (codec == allocated_encoder_.codec &&
      allocated_encoder_.encoder != nullptr) {
    return allocated_encoder_;
  }

  // Try creating external encoder.
  if (external_encoder_factory_ != nullptr &&
      FindMatchingCodec(external_encoder_factory_->supported_codecs(), codec)) {
    webrtc::VideoEncoder* encoder =
        external_encoder_factory_->CreateVideoEncoder(codec);
    if (encoder != nullptr)
      return AllocatedEncoder(encoder, codec, true /* is_external */);
  }

  // Try creating internal encoder.
  if (FindMatchingCodec(internal_encoder_factory_->supported_codecs(), codec)) {
    if (parameters_.encoder_config.content_type ==
            webrtc::VideoEncoderConfig::ContentType::kScreen &&
        parameters_.conference_mode && UseSimulcastScreenshare()) {
      // TODO(sprang): Remove this adapter once libvpx supports simulcast with
      // same-resolution substreams.
      WebRtcSimulcastEncoderFactory adapter_factory(
          internal_encoder_factory_.get());
      return AllocatedEncoder(adapter_factory.CreateVideoEncoder(codec), codec,
                              false /* is_external */);
    }
    return AllocatedEncoder(
        internal_encoder_factory_->CreateVideoEncoder(codec), codec,
        false /* is_external */);
  }

  // This shouldn't happen, we should not be trying to create something we don't
  // support.
  RTC_NOTREACHED();
  return AllocatedEncoder(NULL, cricket::VideoCodec(), false);
}

可以看到，这里会先尝试调用 external_encoder_factory_ 创建外部的编码器，如果创建成功，用外部的，如果失败，再找内部的。

这里的 external_encoder_factory_ ，就是我们调用 CreatePeerConnectionFactory 时传入的那个 encoder factory 实例。