Qt+GStreamer：如何在播放实时视频流时拍摄快照
Posted
技术标签:
【中文标题】Qt+GStreamer：如何在播放实时视频流时拍摄快照【英文标题】：Qt+GStreamer: How to take a snapshot while playing live video stream 【发布时间】：2016-03-14 09:27:00 【问题描述】：我开发了一个基于 Qt 和 QtGstreamer 的视频播放器。它用于播放实时流 (RTSP)。我必须增加用户在播放实时流时拍摄快照而不干扰视频播放的可能性。
这是我制作的管道图：
-->queue-->autovideosink
uridecodebin-->videoflip-->tee--|
| -->queue->videoconvert-->pngenc-->filesink
|
|->audioconvert-->autoaudiosink
我使用来自uridecodebin 的pad-added 信号将我的元素动态添加和链接到管道，接收到的大写函数。
void Player::onPadAdded(const QGst::PadPtr &pad)
QGst::CapsPtr caps = pad->currentCaps();
if (caps->toString().startsWith("video/x-raw"))
qDebug("Received 'video/x-raw' caps");
handleNewVideoPad(pad);
else if (caps->toString().startsWith("audio/x-raw"))
qDebug("Received 'audio/x-raw' caps");
if (!m_audioEnabled)
qDebug("Audio is disabled in the player. Ignoring...");
return;
handleNewAudioPad(pad);
else
qWarning("Unsuported caps, arborting ...!");
return;
[...]
void Player::handleNewVideoPad(QGst::PadPtr pad)
m_player->videoTeeVideoSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Add video elements
m_player->pipeline->add(m_player->videoFlip);
m_player->pipeline->add(m_player->videoTee);
m_player->pipeline->add(m_player->videoQueue);
m_player->pipeline->add(m_player->videoSink);
// Add snap elements
m_player->pipeline->add(m_player->snapQueue);
m_player->pipeline->add(m_player->snapConverter);
m_player->pipeline->add(m_player->snapEncoder);
m_player->pipeline->add(m_player->snapSink);
// Link video elements
m_player->videoFlip->link(m_player->videoTee);
m_player->videoQueue->link(m_player->videoSink);
// Link snap elements
m_player->snapQueue->link(m_player->snapConverter);
m_player->snapConverter->link(m_player->snapEncoder);
m_player->snapEncoder->link(m_player->snapSink);
// Lock snap elements
m_player->snapQueue->setStateLocked(true);
m_player->snapConverter->setStateLocked(true);
m_player->snapEncoder->setStateLocked(true);
m_player->snapSink->setStateLocked(true);
m_player->videoFlip->setState(QGst::StatePlaying);
m_player->videoTee->setState(QGst::StatePlaying);
m_player->videoQueue->setState(QGst::StatePlaying);
m_player->videoSink->setState(QGst::StatePlaying);
// Link pads
m_player->videoTeeVideoSrcPad->link(m_player->videoQueue->getStaticPad("sink"));
pad->link(m_player->videoSinkPad);
m_player->videoLinked = true;
拍摄快照的方法：
void Player::takeSnapshot()
QDateTime dateTime = QDateTime::currentDateTime();
QString snapLocation = QString("/%1/snap_%2.png").arg(m_snapDir).arg(dateTime.toString(Qt::ISODate));
m_player->inSnapshotCaputre = true;
if (m_player->videoTeeSnapSrcPad)
m_player->videoTee->releaseRequestPad(m_player->videoTeeSnapSrcPad);
m_player->videoTeeSnapSrcPad.clear();
m_player->videoTeeSnapSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Stop the snapshot branch
m_player->snapQueue->setState(QGst::StateNull);
m_player->snapConverter->setState(QGst::StateNull);
m_player->snapEncoder->setState(QGst::StateNull);
m_player->snapSink->setState(QGst::StateNull);
// Link Tee src pad to snap queue sink pad
m_player->videoTeeSnapSrcPad->link(m_player->snapQueue->getStaticPad("sink"));
// Set the snapshot location property
m_player->snapSink->setProperty("location", snapLocation);
// Unlock snapshot branch
m_player->snapQueue->setStateLocked(false);
m_player->snapConverter->setStateLocked(false);
m_player->snapEncoder->setStateLocked(false);
m_player->snapSink->setStateLocked(false);
m_player->videoTeeSnapSrcPad->setActive(true);
// Synch snapshot branch state with parent
m_player->snapQueue->syncStateWithParent();
m_player->snapConverter->syncStateWithParent();
m_player->snapEncoder->syncStateWithParent();
m_player->snapSink->syncStateWithParent();
总线消息回调：
void Player::onBusMessage(const QGst::MessagePtr & message)
QGst::ElementPtr source = message->source().staticCast<QGst::Element>();
switch (message->type())
case QGst::MessageEos: //End of stream. We reached the end of the file.
qDebug("Message End Off Stream");
if (m_player->inSnapshotCaputre)
blockSignals(true);
pause();
play();
blockSignals(false);
m_player->inSnapshotCaputre = false;
else
m_eos = true;
stop();
break;
[...]
问题是：
当我将pngenc 元素的snapshot 属性设置为true 时，我收到了停止我的管道的EOS 事件，所以我需要重新启动它，这会冻结视频播放大约半第二，这在我的情况下是不可接受的。
当我将pngenc 元素的snapshot 属性设置为false 时，我没有管道扰动，但我的png 文件一直在增长，直到我再次调用Player::takeSnapshot() 方法。
我哪里错了？有更好的方法吗？
我尝试为我的snapshot 分支创建QGst::Bin 元素，但未成功。焊盘探针呢？
提前致谢
【问题讨论】：
您可以使用焊盘探针阻止文件增长；当您不想要快照时，也许在 videoconvert 之前丢弃所有样本。并且在快照的情况下只允许一个样本通过。我不知道生成的文件是否是正确的 .png 文件。通常，这些文件容器在 EOS 事件上具有最终确定步骤，这使得它们可以正确播放（以 MP4 为例）。 谢谢，我听说过焊盘探针，但从未尝试过。我去测试一下 【参考方案1】：您可以在任何接收器上获取 last-sample 属性，例如您的视频接收器。这包含一个 GstSample，它有一个缓冲区，其中包含最新的视频帧。您可以将其作为快照，例如使用 gst_video_convert_sample() 或其异步变体，将其转换为 PNG/JPG/其他格式。
见https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSink.html#GstBaseSink--last-sample和https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html#gst-video-convert-sample
或者，您必须在第一帧之后关闭文件接收器快照管道。例如，通过填充探针了解第一帧何时发生，然后注入 EOS 事件以防止将更多 PNG 帧附加到同一文件。
【讨论】：
我试过类似的东西：GstElement videoSinkElement = (GstElement)*m_videoSink; // m_videoSink 是 QGst::ElementPtr GstBaseSink *videoSink = NULL; videoSink = GST_BASE_SINK(&videoSinkElement);样本 = gst_base_sink_get_last_sample(videoSink);但它在gst_base_sink_get_last_sample(videoSink); 上崩溃。你有想法吗 ？对我来说复杂的事情是在QtGstreamer API 和“官方”API 之间切换。
这里的演员阵容是错误的：videoSink = GST_BASE_SINK(&videoSinkElement);尝试删除与号 (&)。【参考方案2】：
感谢@sebastian-droge 的回答，我找到了解决方案，使用gst_video_convert_sample 和我的视频接收器的last-sample 属性。
我实施的解决方案是：
void Player::takeSnapshot()
QDateTime currentDate = QDateTime::currentDateTime();
QString location = QString("%1/snap_%2.png").arg(QDir::homePath()).arg(currentDate.toString(Qt::ISODate));
QImage snapShot;
QImage::Format snapFormat;
QGlib::Value val = m_videoSink->property("last-sample");
GstSample *videoSample = (GstSample *)g_value_get_boxed(val);
QGst::SamplePtr sample = QGst::SamplePtr::wrap(videoSample);
QGst::SamplePtr convertedSample;
QGst::BufferPtr buffer;
QGst::CapsPtr caps = sample->caps();
QGst::MapInfo mapInfo;
GError *err = NULL;
GstCaps * capsTo = NULL;
const QGst::StructurePtr structure = caps->internalStructure(0);
int width, height;
width = structure.data()->value("width").get<int>();
height = structure.data()->value("height").get<int>();
qDebug() << "Sample caps:" << structure.data()->toString();
/*
* QImage::Format_RGBX8888, GST_VIDEO_FORMAT_RGBx ,
* QImage::Format_RGBA8888, GST_VIDEO_FORMAT_RGBA ,
* QImage::Format_RGB888 , GST_VIDEO_FORMAT_RGB ,
* QImage::Format_RGB16 , GST_VIDEO_FORMAT_RGB16
*/
snapFormat = QImage::Format_RGB888;
capsTo = gst_caps_new_simple("video/x-raw",
"format", G_TYPE_STRING, "RGB",
"width", G_TYPE_INT, width,
"height", G_TYPE_INT, height,
NULL);
convertedSample = QGst::SamplePtr::wrap(gst_video_convert_sample(videoSample, capsTo, GST_SECOND, &err));
if (convertedSample.isNull())
qWarning() << "gst_video_convert_sample Failed:" << err->message;
else
qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();
buffer = convertedSample->buffer();
buffer->map(mapInfo, QGst::MapRead);
snapShot = QImage((const uchar *)mapInfo.data(),
width,
height,
snapFormat);
qDebug() << "Saving snap to" << location;
snapShot.save(location);
buffer->unmap(mapInfo);
val.clear();
sample.clear();
convertedSample.clear();
buffer.clear();
caps.clear();
g_clear_error(&err);
if (capsTo)
gst_caps_unref(capsTo);
我创建了一个简单的测试应用程序，它实现了这个解决方案。该代码可在我的Github上获得
【讨论】：
以上是关于Qt+GStreamer：如何在播放实时视频流时拍摄快照的主要内容，如果未能解决你的问题，请参考以下文章