Android - 在使用WebRTC发送到Wowza Streaming Engine之前旋转视频帧
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Android - 在使用WebRTC发送到Wowza Streaming Engine之前旋转视频帧相关的知识,希望对你有一定的参考价值。
我想使用WebRTC将视频从android摄像头流式传输到Wowza Streaming Engine(WSE)。当设备处于横向模式时,一切运行良好。然后我尝试通过将设备置于纵向模式来进行流式传输。
我在WSE播放器中注意到的第一件事是视频流已逆时针旋转90度。我发现WebRTC在发送到WSE之前不会旋转从onPreviewFrame API发出的每个视频帧,不幸的是WSE不支持至少到目前为止在他们身边旋转视频帧的任何机制。
所以我检查了WebRTC android原生源代码并修改它以在发送到WSE之前旋转每个视频帧。现在我可以在WSE播放器中以纵向模式看到视频流。
但它有一个问题,有时视频流看起来很奇怪。请参阅以下图片。
我把相机放到固定位置。 WSE播放器第一次显示第一个,但有时第二个显示。
这里是我改变的WebRTC源代码中的文件。 〜/的WebRTC / src目录/ SDK /安卓/ src目录/ JNI / androidvideotracksource.cc
void AndroidVideoTrackSource::OnByteBufferFrameCaptured(const void* frame_data,
int length,
int width,
int height,
VideoRotation rotation,
int64_t timestamp_ns) {
RTC_DCHECK(camera_thread_checker_.CalledOnValidThread());
int64_t camera_time_us = timestamp_ns / rtc::kNumNanosecsPerMicrosec;
int64_t translated_camera_time_us =
timestamp_aligner_.TranslateTimestamp(camera_time_us, rtc::TimeMicros());
int adapted_width;
int adapted_height;
int crop_width;
int crop_height;
int crop_x;
int crop_y;
if (!AdaptFrame(width, height, camera_time_us, &adapted_width,
&adapted_height, &crop_width, &crop_height, &crop_x,
&crop_y)) {
return;
}
const uint8_t* y_plane = static_cast<const uint8_t*>(frame_data);
const uint8_t* uv_plane = y_plane + width * height;
const int uv_width = (width + 1) / 2;
RTC_CHECK_GE(length, width * height + 2 * uv_width * ((height + 1) / 2));
// Can only crop at even pixels.
crop_x &= ~1;
crop_y &= ~1;
// Crop just by modifying pointers.
y_plane += width * crop_y + crop_x;
uv_plane += uv_width * crop_y + crop_x;
rtc::scoped_refptr<I420Buffer> buffer =
buffer_pool_.CreateBuffer(adapted_width, adapted_height);
nv12toi420_scaler_.NV12ToI420Scale(
y_plane, width, uv_plane, uv_width * 2, crop_width, crop_height,
buffer->MutableDataY(), buffer->StrideY(),
// Swap U and V, since we have NV21, not NV12.
buffer->MutableDataV(), buffer->StrideV(), buffer->MutableDataU(),
buffer->StrideU(), buffer->width(), buffer->height());
// TODO: Rotate I420 frame 90 degrees clockwise.
rtc::scoped_refptr<I420Buffer> rotated_buffer =
I420Buffer::Rotate(*buffer, kVideoRotation_90);
OnFrame(VideoFrame(rotated_buffer, rotation, translated_camera_time_us));
}
我添加了这行代码,顺时针旋转I420框架90度。
// TODO: Rotate I420 frame 90 degrees clockwise.
rtc::scoped_refptr<I420Buffer> rotated_buffer =
I420Buffer::Rotate(*buffer, kVideoRotation_90);
我将不胜感激任何帮助!
最后,我找到了解决这个问题的解决方案。以下是我的步骤:
第1步:确保将流媒体活动锁定为纵向
第2步:在WebRTC android本机源代码中更改此方法〜/ webrtc / src / sdk / android / src / jni / androidvideotracksource.cc
原始版本:
void AndroidVideoTrackSource::OnByteBufferFrameCaptured(const void* frame_data,
int length,
int width,
int height,
VideoRotation rotation,
int64_t timestamp_ns) {
RTC_DCHECK(camera_thread_checker_.CalledOnValidThread());
int64_t camera_time_us = timestamp_ns / rtc::kNumNanosecsPerMicrosec;
int64_t translated_camera_time_us =
timestamp_aligner_.TranslateTimestamp(camera_time_us, rtc::TimeMicros());
int adapted_width;
int adapted_height;
int crop_width;
int crop_height;
int crop_x;
int crop_y;
if (!AdaptFrame(width, height, camera_time_us, &adapted_width,
&adapted_height, &crop_width, &crop_height, &crop_x,
&crop_y)) {
return;
}
const uint8_t* y_plane = static_cast<const uint8_t*>(frame_data);
const uint8_t* uv_plane = y_plane + width * height;
const int uv_width = (width + 1) / 2;
RTC_CHECK_GE(length, width * height + 2 * uv_width * ((height + 1) / 2));
// Can only crop at even pixels.
crop_x &= ~1;
crop_y &= ~1;
// Crop just by modifying pointers.
y_plane += width * crop_y + crop_x;
uv_plane += uv_width * crop_y + crop_x;
rtc::scoped_refptr<I420Buffer> buffer =
buffer_pool_.CreateBuffer(adapted_width, adapted_height);
nv12toi420_scaler_.NV12ToI420Scale(
y_plane, width, uv_plane, uv_width * 2, crop_width, crop_height,
buffer->MutableDataY(), buffer->StrideY(),
// Swap U and V, since we have NV21, not NV12.
buffer->MutableDataV(), buffer->StrideV(), buffer->MutableDataU(),
buffer->StrideU(), buffer->width(), buffer->height());
OnFrame(VideoFrame(rotated_buffer, rotation, translated_camera_time_us));
}
修改版本:
void AndroidVideoTrackSource::OnByteBufferFrameCaptured(const void* frame_data,
int length,
int width,
int height,
VideoRotation rotation,
int64_t timestamp_ns) {
RTC_DCHECK(camera_thread_checker_.CalledOnValidThread());
int64_t camera_time_us = timestamp_ns / rtc::kNumNanosecsPerMicrosec;
int64_t translated_camera_time_us =
timestamp_aligner_.TranslateTimestamp(camera_time_us, rtc::TimeMicros());
int adapted_width;
int adapted_height;
int crop_width;
int crop_height;
int crop_x;
int crop_y;
if (!AdaptFrame(width, height, camera_time_us, &adapted_width,
&adapted_height, &crop_width, &crop_height, &crop_x,
&crop_y)) {
return;
}
const uint8_t* y_plane = static_cast<const uint8_t*>(frame_data);
const uint8_t* uv_plane = y_plane + width * height;
const int uv_width = (width + 1) / 2;
RTC_CHECK_GE(length, width * height + 2 * uv_width * ((height + 1) / 2));
// Can only crop at even pixels.
crop_x &= ~1;
crop_y &= ~1;
// Crop just by modifying pointers.
y_plane += width * crop_y + crop_x;
uv_plane += uv_width * crop_y + crop_x;
rtc::scoped_refptr<I420Buffer> buffer =
buffer_pool_.CreateBuffer(adapted_width, adapted_height);
nv12toi420_scaler_.NV12ToI420Scale(
y_plane, width, uv_plane, uv_width * 2, crop_width, crop_height,
buffer->MutableDataY(), buffer->StrideY(),
// Swap U and V, since we have NV21, not NV12.
buffer->MutableDataV(), buffer->StrideV(), buffer->MutableDataU(),
buffer->StrideU(), buffer->width(), buffer->height());
// TODO: Comment out this line of code to apply custom code.
// OnFrame(VideoFrame(buffer, rotation, translated_camera_time_us));
// TODO: The custom code to rotate video frame before passing
// to next layers of WebRTC.
// Rotate I420 frame rotation degrees.
// Value of the rotation is 90 or 270 based on camera orientation.
rtc::scoped_refptr<I420Buffer> rotated_buffer =
I420Buffer::Rotate(*buffer, rotation);
// Make sure the I420 frame has valid side in portrait mode.
rtc::scoped_refptr<I420Buffer> final_buffer =
buffer_pool_.CreateBuffer(height, width);
final_buffer->ScaleFrom(*rotated_buffer);
// After rotating the I420 frame, set value of the rotation to 0.
// This mean we do not want to rotate the frame in next layers anymore.
rotation = kVideoRotation_0;
// Pass processed frame to the next layers.
OnFrame(VideoFrame(final_buffer, rotation, translated_camera_time_us));
}
现在我的流在Streaming Wowza Engine Player上完美显示。
以上是关于Android - 在使用WebRTC发送到Wowza Streaming Engine之前旋转视频帧的主要内容,如果未能解决你的问题,请参考以下文章
我的 webrtc 在我的 sdp 中发送 recvonly 方向