使用Android NDK Camera2获取预览帧的正确方法是啥

Posted 2023-04-13

【中文标题】使用Android NDK Camera2获取预览帧的正确方法是啥

【英文标题】：What is the correct way to get Preview Frames using Android NDK Camera2使用Android NDK Camera2获取预览帧的正确方法是什么

【发布时间】：2022-01-14 00:47:44

【问题描述】：

基于 NDK 相机示例texture-view，我想创建一个ImageReader 来获取预览帧。

我做了什么

创建ImageReader 和相机会话：

yuvReader_ = new ImageReader(&compatibleCameraRes_, AIMAGE_FORMAT_YUV_420_888);
camera_->CreateSession(ANativeWindow_fromSurface(env_, surface), yuvReader_->GetNativeWindow());

void NDKCamera::CreateSession(ANativeWindow* previewWindow, ANativeWindow* yuvWindow) 
    // Create output from this app's ANativeWindow, and add into output container
    requests_[PREVIEW_REQUEST_IDX].outputNativeWindow_ = previewWindow;
    requests_[PREVIEW_REQUEST_IDX].template_ = TEMPLATE_PREVIEW;
    requests_[YUV_REQUEST_IDX].outputNativeWindow_ = yuvWindow;
    requests_[YUV_REQUEST_IDX].template_ = TEMPLATE_PREVIEW;

    CALL_CONTAINER(create(&outputContainer_));
    for (auto& req : requests_) 
        if (!req.outputNativeWindow_) continue;

        ANativeWindow_acquire(req.outputNativeWindow_);
        CALL_OUTPUT(create(req.outputNativeWindow_, &req.sessionOutput_));
        CALL_CONTAINER(add(outputContainer_, req.sessionOutput_));
        CALL_TARGET(create(req.outputNativeWindow_, &req.target_));
        CALL_DEV(createCaptureRequest(cameras_[activeCameraId_].device_,
                                      req.template_, &req.request_));
        CALL_REQUEST(addTarget(req.request_, req.target_));
    

    // Create a capture session for the given preview request
    captureSessionState_ = CaptureSessionState::READY;
    CALL_DEV(createCaptureSession(cameras_[activeCameraId_].device_,
                                  outputContainer_, GetSessionListener(),
                                  &captureSession_));

然后开始预览：

void NDKCamera::StartPreview(bool start) 
  if (start) 
    ACaptureRequest* requests[] =  requests_[PREVIEW_REQUEST_IDX].request_, requests_[YUV_REQUEST_IDX].request_;
    CALL_SESSION(setRepeatingRequest(captureSession_, nullptr, 2,
                                     requests,
                                     nullptr));
   else if (!start && captureSessionState_ == CaptureSessionState::ACTIVE) 
    ACameraCaptureSession_stopRepeating(captureSession_);
  

我在setRepeatingRequest 中设置了两个请求。一个用于TextureView 显示，另一个用于接收C++ 中的预览帧。

现在，问题是设置两个输出后，预览性能下降（看起来像在播放幻灯片），这在 Java 中不会发生：

mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()),
                    new CameraCaptureSession.StateCallback() 

                        @Override
                        public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) 
                            // The camera is already closed
                            if (null == mCameraDevice) 
                                return;
                            

                            mCaptureSession = cameraCaptureSession;
                            startPreview();
                        

                        @Override
                        public void onConfigureFailed(
                                @NonNull CameraCaptureSession cameraCaptureSession) 
                            showToast("Failed");
                        
                    , null
            );

我还尝试了一个带有两个输出目标的请求。但是代码导致屏幕冻结：

void NDKCamera::CreateSession(ANativeWindow* textureViewWindow, ANativeWindow* imgReaderWindow) 
    auto& req = requests_[PREVIEW_REQUEST_IDX];
    req.outputNativeWindow_ = textureViewWindow;
    req.yuvWindow = imgReaderWindow;
    req.template_ = TEMPLATE_PREVIEW;

    ACaptureSessionOutputContainer_create(&outputContainer_);
    CALL_DEV(createCaptureRequest(cameras_[activeCameraId_].device_,
                                          req.template_, &req.request_));
    
    // Add the texture view surface to the container
    ANativeWindow_acquire(req.outputNativeWindow_);
    CALL_OUTPUT(create(req.outputNativeWindow_, &req.sessionOutput_));
    CALL_CONTAINER(add(outputContainer_, req.sessionOutput_));
    CALL_TARGET(create(req.outputNativeWindow_, &req.target_));
    CALL_REQUEST(addTarget(req.request_, req.target_));

    // Add the image reader surface to the container
    ANativeWindow_acquire(req.yuvWindow);
    CALL_OUTPUT(create(req.yuvWindow, &req.yuvOutput));
    CALL_CONTAINER(add(outputContainer_, req.yuvOutput));
    CALL_TARGET(create(req.yuvWindow, &req.yuvTarget));
    CALL_REQUEST(addTarget(req.request_, req.yuvTarget));

    captureSessionState_ = CaptureSessionState::READY;
    ACameraDevice_createCaptureSession(cameras_[activeCameraId_].device_,
                                  outputContainer_, GetSessionListener(),
                                  &captureSession_);


void NDKCamera::StartPreview(bool start) 
  if (start) 
    ACaptureRequest* requests[] =  requests_[PREVIEW_REQUEST_IDX].request_;
    ACameraCaptureSession_setRepeatingRequest(captureSession_, nullptr, 1,
                                     requests,
                                     nullptr);
   else if (!start && captureSessionState_ == CaptureSessionState::ACTIVE) 
    ACameraCaptureSession_stopRepeating(captureSession_);
  

这是日志：

2021-12-14 08:42:20.316 24536-24556/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 13, request ID 0, subseq ID 0
2021-12-14 08:42:21.319 24536-24556/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 14, request ID 0, subseq ID 0
2021-12-14 08:42:22.321 24536-24584/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 15, request ID 0, subseq ID 0
2021-12-14 08:42:23.323 24536-24584/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 16, request ID 0, subseq ID 0
2021-12-14 08:42:24.325 24536-24556/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 17, request ID 0, subseq ID 0
2021-12-14 08:42:25.328 24536-24584/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 18, request ID 0, subseq ID 0
2021-12-14 08:42:26.330 24536-24584/com.sample.textureview D/ACameraDevice: Device error received, code 3, frame number 19, request ID 0, subseq ID 0

有人知道为什么吗？谢谢！

【参考方案1】：

相比之下，我不知道您是如何设置 Java 代码的，但是您在 NDK 代码中所做的事情会使您的帧速率降低一半。如果您想以 30fps 的速度将预览帧和帧都发送到本机 ImageReader，则需要在单个捕获请求中包含两个目标，而不是在每个目标仅针对一个输出的两个捕获请求之间交替。后者最多可以让您的每个输出达到 15fps。

所以只需创建一个请求，然后在预览和 YUV 窗口中调用 addTarget 两次。您可以向单个请求添加多少目标是有限制的，但通常这等于您可以在单个会话中配置的目标数量，这取决于设备的硬件能力和每个输出的分辨率。

但是，2 个流，一个预览和一个应用绑定的 YUV，应该始终有效。

【讨论】：

实际上我尝试了一个带有两个目标的请求。但是在接收到一个相机帧后屏幕就被冻结了。我收到了消息D/ACameraDevice: Device error received, code 3, frame number 13, request ID 0, subseq ID 0。

您要求的输出分辨率是多少？

分辨率为1664x768。