QAbstractVideoSurface 奇怪的行为

Posted 2023-02-16

【中文标题】QAbstractVideoSurface 奇怪的行为

【英文标题】：QAbstractVideoSurface Strange behaviour

【发布时间】：2019-01-05 10:10:47

【问题描述】：

我已经实现了 QAbstractVideoSurface，如下所示，我面临以下问题：

有时根本不调用 QAbstractVideosurface 的现有方法。也没有错误。我可以看到视频正在从音频播放，并且时间戳正在更新。

我在实施中缺少什么？

video_frame_grabber.cpp

#include "video_frame_grabber.h"
#include <QtWidgets>
#include <qabstractvideosurface.h>
#include <qvideosurfaceformat.h>

VideoFrameGrabber::VideoFrameGrabber(QGraphicsView *view, QGraphicsPixmapItem *pixmap, QObject *parent)
    : QAbstractVideoSurface(parent)
    , view(view)
    , pixmap(pixmap)
    , imageFormat(QImage::Format_Invalid)



QList<QVideoFrame::PixelFormat> VideoFrameGrabber::supportedPixelFormats(QAbstractVideoBuffer::HandleType handleType) const

    Q_UNUSED(handleType);
    return QList<QVideoFrame::PixelFormat>()
        << QVideoFrame::Format_ARGB32
        << QVideoFrame::Format_ARGB32_Premultiplied
        << QVideoFrame::Format_RGB32
        << QVideoFrame::Format_RGB24
        << QVideoFrame::Format_RGB565
        << QVideoFrame::Format_RGB555
        << QVideoFrame::Format_ARGB8565_Premultiplied
        << QVideoFrame::Format_BGRA32
        << QVideoFrame::Format_BGRA32_Premultiplied
        << QVideoFrame::Format_BGR32
        << QVideoFrame::Format_BGR24
        << QVideoFrame::Format_BGR565
        << QVideoFrame::Format_BGR555
        << QVideoFrame::Format_BGRA5658_Premultiplied
        << QVideoFrame::Format_AYUV444
        << QVideoFrame::Format_AYUV444_Premultiplied
        << QVideoFrame::Format_YUV444
        << QVideoFrame::Format_YUV420P
        << QVideoFrame::Format_YV12
        << QVideoFrame::Format_UYVY
        << QVideoFrame::Format_YUYV
        << QVideoFrame::Format_NV12
        << QVideoFrame::Format_NV21
        << QVideoFrame::Format_IMC1
        << QVideoFrame::Format_IMC2
        << QVideoFrame::Format_IMC3
        << QVideoFrame::Format_IMC4
        << QVideoFrame::Format_Y8
        << QVideoFrame::Format_Y16
        << QVideoFrame::Format_Jpeg
        << QVideoFrame::Format_CameraRaw
        << QVideoFrame::Format_AdobeDng;


bool VideoFrameGrabber::isFormatSupported(const QVideoSurfaceFormat &format) const

    const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
    const QSize size = format.frameSize();

    return imageFormat != QImage::Format_Invalid
            && !size.isEmpty()
            && format.handleType() == QAbstractVideoBuffer::NoHandle;


bool VideoFrameGrabber::start(const QVideoSurfaceFormat &format)

    const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
    const QSize size = format.frameSize();

    if (imageFormat != QImage::Format_Invalid && !size.isEmpty()) 
        this->imageFormat = imageFormat;
        QAbstractVideoSurface::start(format);
        return true;
     
    else
        return false;
    


void VideoFrameGrabber::stop()

    QAbstractVideoSurface::stop();

    view->update();


bool VideoFrameGrabber::present(const QVideoFrame &frame)

    if ((surfaceFormat().pixelFormat() != frame.pixelFormat()) || (surfaceFormat().frameSize() != frame.size())) 
        setError(IncorrectFormatError);
        stop();
        qDebug() << "error";
        return false;
     
    else 
        QImage img = VideoFrameToImage(frame);
        displayframe(img);
        emit frame_received();
        return true;
    


void VideoFrameGrabber::displayframe(QImage img)

    pixmap->setPixmap(QPixmap::fromImage(img));
    view->fitInView(QRectF(0,0,img.width(),img.height()),Qt::KeepAspectRatio);


    QImage VideoFrameGrabber::VideoFrameToImage(QVideoFrame frame)
    
        if (frame.map(QAbstractVideoBuffer::ReadOnly)) 
       return QImage(frame.bits(), frame.width(), frame.height(), frame.bytesPerLine(), imageFormat);
       
    return QImage();
    

video_frame_grabber.h

#include <QAbstractVideoSurface>
#include <QObject>
#include <QGraphicsView>
#include <QGraphicsScene>
#include <QPixmap>

class VideoFrameGrabber : public QAbstractVideoSurface

    Q_OBJECT
public:
    VideoFrameGrabber(QGraphicsView *view, QGraphicsPixmapItem *pixmap ,QObject *parent = nullptr);

    QList<QVideoFrame::PixelFormat> supportedPixelFormats(
            QAbstractVideoBuffer::HandleType handleType = QAbstractVideoBuffer::NoHandle) const;
    bool isFormatSupported(const QVideoSurfaceFormat &format) const;

    bool start(const QVideoSurfaceFormat &format);
    void stop();

    bool present(const QVideoFrame &frame);
    void displayframe(QImage img);
    QImage VideoFrameToImage(QVideoFrame frame);

private:
    QGraphicsView *view;
    QGraphicsPixmapItem *pixmap;
    QImage::Format imageFormat;
    int current_function;

signals:
    void frame_received();
;

还有mainwindow.cpp

m_MediaPlayer = new QMediaPlayer(this);
m_GraphicsScene = new QGraphicsScene();
pixmapItem = new QGraphicsPixmapItem();
m_GraphicsScene->addItem(pixmapItem);
ui->m_GraphicsView->setScene(m_GraphicsScene);
grabber = new VideoFrameGrabber(ui->m_GraphicsView, pixmapItem);
m_MediaPlayer->setVideoOutput(grabber);

m_MediaPlayer->setMedia(QUrl::fromLocalFile("1.mp4"));

提前致谢

【问题讨论】：

QVideoFrame copy(frame)。这是不必要的：它是浅拷贝，两者都将引用相同的数据。注意QImage(frame.bits(), frame.width(), frame.height(), frame.bytesPerLine(), imageFormat);也会引用原始数据，这很好。

【参考方案1】：

我最好的选择是你落后了，这是由于转换效率低下

 QVideoFrame -> QImage -> QPixmap

特别是QPixmap::fromImage(img) 是您管道中的瓶颈。

The QMediaPlayer media player already supports output to QGraphicsVideoItem。这意味着你应该能够做到：

m_MediaPlayer = new QMediaPlayer(this);
m_GraphicsScene = new QGraphicsScene();
videoItem = new QGraphicsVideoItem();
m_GraphicsScene->addItem(videoItem); 
m_MediaPlayer->setVideoOutput(videoItem);

【讨论】：

我已经实现了 QAbstractVideoSurface 在显示之前修改帧。但是，目前基本功能本身不起作用

你需要在框架上执行什么样的过滤器？

我希望将图像转换为 OpenCV。在显示之前根据组合框选择应用去噪、图像平滑、裁剪或旋转

为了实时做到这一点（低于 40 毫秒的延迟），非常具有挑战性。你有几个选择，但我建议工作队列。基本上你有一个引擎，它尽可能快地输出帧并且帧进入工作队列，然后你有一个用于显示的缓冲区。

您还可以做的是准备预过滤的视频，保存并重播。不幸的是，对视频进行编码需要很长时间。