Camerax ImageCapture 传递给 Chaquopy 太慢了

Posted

技术标签:

【中文标题】Camerax ImageCapture 传递给 Chaquopy 太慢了【英文标题】:Camerax ImageCapture passing to Chaquopy too slow 【发布时间】:2021-06-23 17:30:34 【问题描述】:

我用 Chaquopy 制作了 Camerax 应用程序:

ImageCapture 用例捕获图像(jpg) 将其转换为位图 将位图转换为字符串 将字符串传递给 Python Python OpenCV 获取图像分辨率并将其返回给 android(如字符串) 在 Textview 上显示

应用程序正在运行,但问题是:

    应用程序太慢:15-20 秒(字符串转换?) 显示错误的分辨率(分辨率在 ImageCapture 配置中设置)
如何加速我的应用程序?? 除了base64字符串,还有其他方法可以将图像传递给python吗??

Android CameraAtivity 代码:

'''

public class CameraActivity extends AppCompatActivity 
    private PreviewView previewView;
    private ImageCapture imageCapture;
    private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;
    private TextView textView;
    private Button button;



    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) 
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        previewView = findViewById(R.id.previewView);
        cameraProviderFuture = ProcessCameraProvider.getInstance(this);
        textView = findViewById(R.id.length);
        button = findViewById(R.id.button);


        cameraProviderFuture.addListener(new Runnable() 
            @Override
            public void run() 
                try 
                    ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                    bindImageAnalysis(cameraProvider);
                 catch (ExecutionException | InterruptedException e) 
                    e.printStackTrace();
                
            
        , ContextCompat.getMainExecutor(this));
    


    private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) 

        Preview preview = new Preview.Builder().build();
        CameraSelector cameraSelector = new CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
        preview.setSurfaceProvider(previewView.createSurfaceProvider());

        Executor cameraexecutor = ContextCompat.getMainExecutor(this);
        ImageCapture imageCapture = new ImageCapture.Builder().setTargetResolution(new Size(1200, 1600)).build();
        button.setOnClickListener(new View.OnClickListener() 
            @Override
            public void onClick(View view) 
                imageCapture.takePicture(cameraexecutor, new ImageCapture.OnImageCapturedCallback() 
                    @Override
                    public void onCaptureSuccess(@NonNull ImageProxy image)
                        ByteBuffer byteBuffer = image.getPlanes()[0].getBuffer();

                        byte[] bytes = new byte[byteBuffer.capacity()];
                        byteBuffer.get(bytes);

                        Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
                        ByteArrayOutputStream baos = new ByteArrayOutputStream();
                        bitmap.compress(Bitmap.CompressFormat.PNG, 100, baos);
                        byte[] imageBytes = baos.toByteArray();
                        String imgString = Base64.encodeToString(imageBytes, Base64.DEFAULT);


                        PyObject obj = pythonn(imgString);

                        textView.setText(obj.toString());

                    

                    @Override
                    public void onError(@NonNull ImageCaptureException error) 
                        error.printStackTrace();
                    
                );
            

        );


        cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageCapture,
                preview);
    

    private PyObject pythonn(String imgString) 
        if(!Python.isStarted())
            Python.start(new AndroidPlatform(this ));
        Python py = Python.getInstance();

        final PyObject pyobj = py.getModule("shape");
        PyObject obj = pyobj.callAttr("main",imgString);
        return obj;
    

'''

Python 代码;

import cv2
import numpy as np
import base64

def main(imgString):

    decoded_data = base64.b64decode(imgString)
    np_data = np.fromstring(decoded_data,np.uint8)
    img = cv2.imdecode(np_data,cv2.IMREAD_UNCHANGED)

    if img.shape[0] > img.shape[1]:
        img = cv2.transpose(img)
    
    return str(str(img.shape[0]) + "_"  + str(img.shape[1])) 

所有代码都在 Github 上: https://github.com/kintipu/Camerax_ImageCapture_Chaquopy_OpenCV_ImageResolution/tree/master

2021 年 3 月 29 日更新: 我将尝试使用 bytearray,就像 mhsmith 建议的那样。 与此同时,我尝试了另一件事:

捕获图像并将其保存为 jpg

然后直接从python中读取图片 应用可以在模拟器上运行,但不能在真手机上运行 安卓代码: '''

公共类 CameraActivity 扩展 AppCompatActivity 私有 PreviewView 预览视图; 私有图像捕获图像捕获; 私有 ListenableFuture cameraProviderFuture; 私有文本视图文本视图; 私人按钮按钮; 私有上下文上下文;

  @Override
  protected void onCreate(@Nullable Bundle savedInstanceState) 
      super.onCreate(savedInstanceState);
      setContentView(R.layout.activity_camera);
      previewView = findViewById(R.id.previewView);
      cameraProviderFuture = ProcessCameraProvider.getInstance(this);
      textView = findViewById(R.id.length);
      button = findViewById(R.id.button);


      cameraProviderFuture.addListener(new Runnable() 
          @Override
          public void run() 
              try 
                  ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                  bindImageAnalysis(cameraProvider);
               catch (ExecutionException | InterruptedException e) 
                  e.printStackTrace();
              
          
      , ContextCompat.getMainExecutor(this));
  


  private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) 

      Preview preview = new Preview.Builder().build();
      CameraSelector cameraSelector = new CameraSelector.Builder()
              .requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
      preview.setSurfaceProvider(previewView.createSurfaceProvider());

      Executor cameraexecutor = ContextCompat.getMainExecutor(this);
      ImageCapture imageCapture = new ImageCapture.Builder().setTargetResolution(new Size(1200, 1600)).build();


      button.setOnClickListener(new View.OnClickListener() 
          @Override
          public void onClick(View v) 


              File file = new File(getBatchDirectoryName(), "photo"+ ".jpg");

              ImageCapture.OutputFileOptions outputFileOptions = new ImageCapture.OutputFileOptions.Builder(file).build();
              imageCapture.takePicture(outputFileOptions, cameraexecutor, new ImageCapture.OnImageSavedCallback () 
                  @Override
                  public void onImageSaved(@NonNull ImageCapture.OutputFileResults outputFileResults) 
                      new Handler().post(new Runnable() 
                          @Override
                          public void run() 
                              Toast.makeText(CameraActivity.this, "Image Saved successfully" + file.getAbsolutePath(), Toast.LENGTH_LONG).show();

                              PyObject obj = pythonn();
                              textView.setText(obj.toString());
                          


                      );

                  
                  @Override
                  public void onError(@NonNull ImageCaptureException error) 
                      error.printStackTrace();
                  
              );
          
      );


      cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageCapture,
              preview);
  




  public String getBatchDirectoryName() 

      String app_folder_path = "";
      app_folder_path = Environment.getExternalStorageDirectory().toString() + "/images";
      File dir = new File(app_folder_path);
      if (!dir.exists() && !dir.mkdirs()) 

      

      return app_folder_path;
  





  private PyObject pythonn() 
      if(!Python.isStarted())
          Python.start(new AndroidPlatform(this ));
      Python py = Python.getInstance();

      final PyObject pyobj = py.getModule("shape");
      PyObject obj = pyobj.callAttr("main");

      return obj;
  

'''

Python 代码:

'''

import cv2
import numpy as np
from android.os import Environment


path = str(str(Environment.getExternalStorageDirectory()) + "/images" + "/photo.jpg")


def main():
    

    img = cv2.imread(path)
    
    return str(str(img.shape[0]) + "_"  + str(img.shape[1])) 

'''

2021 年 4 月 4 日更新:

通过一些修改,我让它变得更快:

将 bytearray 之类的图像传递给 python(而不是字符串和 base64 转换) - 根据 Malcom Smith 的建议 位图压缩设置为 JPEG(而不是 PNG) Python 在 Create 时启动

安卓代码; '''

public class CameraActivity extends AppCompatActivity 
    private PreviewView previewView;
    private ImageCapture imageCapture;
    private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;
    private TextView textView;
    private Button button;



    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) 
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        previewView = findViewById(R.id.previewView);
        cameraProviderFuture = ProcessCameraProvider.getInstance(this);
        textView = findViewById(R.id.length);
        button = findViewById(R.id.button);
        
        if(!Python.isStarted())
              Python.start(new AndroidPlatform(this ));


        cameraProviderFuture.addListener(new Runnable() 
            @Override
            public void run() 
                try 
                    ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                    bindImageAnalysis(cameraProvider);
                 catch (ExecutionException | InterruptedException e) 
                    e.printStackTrace();
                
            
        , ContextCompat.getMainExecutor(this));
    


    private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) 

        Preview preview = new Preview.Builder().build();
        CameraSelector cameraSelector = new CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
        preview.setSurfaceProvider(previewView.createSurfaceProvider());

        Executor cameraexecutor = ContextCompat.getMainExecutor(this);
        ImageCapture imageCapture = new ImageCapture.Builder().setTargetResolution(new Size(1200, 1600)).build();
        button.setOnClickListener(new View.OnClickListener() 
            @Override
            public void onClick(View view) 
                imageCapture.takePicture(cameraexecutor, new ImageCapture.OnImageCapturedCallback() 
                    @Override
                    public void onCaptureSuccess(@NonNull ImageProxy image)
                        ByteBuffer byteBuffer = image.getPlanes()[0].getBuffer();

                        byte[] bytes = new byte[byteBuffer.capacity()];
                        byteBuffer.get(bytes);

                        Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
                        ByteArrayOutputStream baos = new ByteArrayOutputStream();
                        bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
                        byte[] imageBytes = baos.toByteArray();

//                        String length = String.valueOf((imgString.length()));
//                        textView.setText(length);

                        PyObject obj = pythonn(imageBytes);

                        textView.setText(obj.toString());

                    

                    @Override
                    public void onError(@NonNull ImageCaptureException error) 
                        error.printStackTrace();
                    
                );
            

        );


        cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageCapture,
                preview);
    

    private PyObject pythonn(byte[] imageBytes) 

        Python py = Python.getInstance();

        final PyObject pyobj = py.getModule("shape");
        PyObject obj = pyobj.callAttr("main",imageBytes);
        return obj;
    

Python: '''

import cv2
import numpy as np



def main(imageBytes):
         np_data = np.asarray(imageBytes,np.uint8)
         img = cv2.imdecode(np_data,cv2.IMREAD_UNCHANGED)
         if img.shape[0] > img.shape[1]:
                img = cv2.transpose(img)

         img = cv2.resize(img, (1600, 1200))
    
         return str(str(img.shape[0]) + "_"  + str(img.shape[1]))

更新了github:https://github.com/kintipu/Camerax_ImageCapture_Chaquopy_OpenCV_ImageResolution/blob/master/README.md

【问题讨论】:

【参考方案1】:

不需要使用base64,你可以发送PNG编码的byte[]数组,它会在Python中作为bytes-like对象接收。我不确定这个对象是否会被cv2.imdecode 直接接受,但它肯定会被np.arraynp.frombuffer 接受。 (不推荐将二进制数据传递给np.fromstring,正如其documentation page 所解释的那样。)

如果您不确定哪一行代码导致延迟,您可以在行之间添加Log 调用(或Python 中的print 调用),然后检查Logcat 中的时间戳

【讨论】:

谢谢mhsmith,我会试试的。

以上是关于Camerax ImageCapture 传递给 Chaquopy 太慢了的主要内容,如果未能解决你的问题,请参考以下文章

如何在camerax中锁定曝光?

Android CameraX - 录制视频时的面部检测

为 CameraX ImageAnalysis 进行 YUV 到 RGB 的转换

Android CameraX 仿一甜相机(录像拍照可调节尺寸聚焦照明网格线),最全的CameraX教程

CameraX + 华为ScanKit:二维码扫描的终极解决方案

Camerax 图像分析:将图像转换为 bytearray 或 ByteBuffer