Camerax 图像分析：将图像转换为 bytearray 或 ByteBuffer

Posted 2023-04-17

【中文标题】Camerax 图像分析：将图像转换为 bytearray 或 ByteBuffer

【英文标题】：Camerax image analysis: Convert image to bytearray or ByteBuffer

【发布时间】：2020-11-24 11:54:40

【问题描述】：

我做了很多阅读尝试了很多不同的方法。 CameraX 正在生成 yuv_420_888 格式的 Image 对象并将其提供给 ImageAnalysis。

但是，无法将其转换为字节缓冲区以进行缩放、转换为位图和运行检测操作。我尝试了以下和许多其他建议的技术。

Converting ImageProxy to Bitmap

所有这些都创建了灰度（即使在使用了所有 3 个平面之后）和一些叠加颜色阴影图像。它有时还会在帧之间产生故障输出，我无法找出原因。

获取简单字节数组以便稍后转换为位图的正确方法是什么？

还有如何引起cameraX作者的注意？

【问题讨论】：

你试过github.com/owahltinez/camerax-tflite/blob/master/app/src/main/…吗？

【参考方案1】：

您只需要使用imageProxy.image?.toBitmap() 转换imageProxy 然后将bitmap 转换为bytearray 如下：

这是一个例子：

private fun takePhoto() 

    camera_capture_button.isEnabled = false

    // Get a stable reference of the modifiable image capture use case
    val imageCapture = imageCapture ?: return

    imageCapture.takePicture(
        ContextCompat.getMainExecutor(this),
        object : ImageCapture.OnImageCapturedCallback() 

            @SuppressLint("UnsafeExperimentalUsageError")
            override fun onCaptureSuccess(imageProxy: ImageProxy) 
                val bitmapImage = imageProxy.image?.toBitmap()
                   
                val stream = ByteArrayOutputStream()
                bitmapImage.compress(Bitmap.CompressFormat.PNG, 90, stream)
                val image = stream.toByteArray()
                    
                
            

            override fun onError(exception: ImageCaptureException) 
                super.onError(exception)
            
        )

【参考方案2】：

 fun imageProxyToByteArray(image: ImageProxy): ByteArray 
            val yuvBytes = ByteArray(image.width * (image.height + image.height / 2))
            val yPlane = image.planes[0].buffer
            val uPlane = image.planes[1].buffer
            val vPlane = image.planes[2].buffer

            yPlane.get(yuvBytes, 0, image.width * image.height)

            val chromaRowStride = image.planes[1].rowStride
            val chromaRowPadding = chromaRowStride - image.width / 2

            var offset = image.width * image.height
            if (chromaRowPadding == 0) 

                uPlane.get(yuvBytes, offset, image.width * image.height / 4)
                offset += image.width * image.height / 4
                vPlane.get(yuvBytes, offset, image.width * image.height / 4)
             else 
                for (i in 0 until image.height / 2) 
                    uPlane.get(yuvBytes, offset, image.width / 2)
                    offset += image.width / 2
                    if (i < image.height / 2 - 2) 
                        uPlane.position(uPlane.position() + chromaRowPadding)
                    
                
                for (i in 0 until image.height / 2) 
                    vPlane.get(yuvBytes, offset, image.width / 2)
                    offset += image.width / 2
                    if (i < image.height / 2 - 1) 
                        vPlane.position(vPlane.position() + chromaRowPadding)
                    
                
            

            return yuvBytes
        

【讨论】：

社区鼓励在代码中添加解释，而不是纯粹基于代码的答案（参见here）。

【参考方案3】：

您可以使用从 Mlkit Pose Detection 中提取的此类。

Mlkit pose detection: BitmapUtils.java

object ImageProxyUtils 

fun getByteArray(image: ImageProxy): ByteArray? 
    image.image?.let 
        val nv21Buffer = yuv420ThreePlanesToNV21(
            it.planes, image.width, image.height
        )

        return ByteArray(nv21Buffer.remaining()).apply 
            nv21Buffer.get(this)
        
    

    return null


private fun yuv420ThreePlanesToNV21(
    yuv420888planes: Array<Plane>,
    width: Int,
    height: Int
): ByteBuffer 
    val imageSize = width * height
    val out = ByteArray(imageSize + 2 * (imageSize / 4))
    if (areUVPlanesNV21(yuv420888planes, width, height)) 

        yuv420888planes[0].buffer[out, 0, imageSize]
        val uBuffer = yuv420888planes[1].buffer
        val vBuffer = yuv420888planes[2].buffer
        vBuffer[out, imageSize, 1]
        uBuffer[out, imageSize + 1, 2 * imageSize / 4 - 1]
     else 
        unpackPlane(yuv420888planes[0], width, height, out, 0, 1)
        unpackPlane(yuv420888planes[1], width, height, out, imageSize + 1, 2)
        unpackPlane(yuv420888planes[2], width, height, out, imageSize, 2)
    
    return ByteBuffer.wrap(out)


private fun areUVPlanesNV21(planes: Array<Plane>, width: Int, height: Int): Boolean 
    val imageSize = width * height
    val uBuffer = planes[1].buffer
    val vBuffer = planes[2].buffer

    val vBufferPosition = vBuffer.position()
    val uBufferLimit = uBuffer.limit()

    vBuffer.position(vBufferPosition + 1)
    uBuffer.limit(uBufferLimit - 1)

    val areNV21 =
        vBuffer.remaining() == 2 * imageSize / 4 - 2 && vBuffer.compareTo(uBuffer) == 0

    vBuffer.position(vBufferPosition)
    uBuffer.limit(uBufferLimit)
    return areNV21


private fun unpackPlane(
    plane: Plane,
    width: Int,
    height: Int,
    out: ByteArray,
    offset: Int,
    pixelStride: Int
) 
    val buffer = plane.buffer
    buffer.rewind()
    val numRow = (buffer.limit() + plane.rowStride - 1) / plane.rowStride
    if (numRow == 0) 
        return
    
    val scaleFactor = height / numRow
    val numCol = width / scaleFactor

    var outputPos = offset
    var rowStart = 0
    for (row in 0 until numRow) 
        var inputPos = rowStart
        for (col in 0 until numCol) 
            out[outputPos] = buffer[inputPos]
            outputPos += pixelStride
            inputPos += plane.pixelStride
        
        rowStart += plane.rowStride