在运行时在 kotlin 中的相机上运行的不同 ML 模型之间切换

Posted 2023-03-16

技术标签:

【中文标题】在运行时在 kotlin 中的相机上运行的不同 ML 模型之间切换【英文标题】：Switch between different ML models that run on camera in kotlin on runtime 【发布时间】：2022-01-12 01:00:15 【问题描述】：

我正在尝试制作一个允许用户在不同的 ML 模型之间动态切换的应用程序。现在我有一个简单的应用程序，可以运行没有模型的相机和带有对象检测模型的相机。最终应用中将有五个这样的模型。

这是我的Home Activity

class HomeActivity : AppCompatActivity(), EasyPermissions.PermissionCallbacks 
private val rqSpeechRec = 102
private var tts: TextToSpeech? = null
private lateinit var binding: ActivityHomeBinding
private lateinit var objectDetector: ObjectDetector
private lateinit var cameraProviderFuture: ListenableFuture<ProcessCameraProvider>

override fun onCreate(savedInstanceState: Bundle?) 
    super.onCreate(savedInstanceState)

    // Binding ViewData
    binding = ActivityHomeBinding.inflate(layoutInflater)
    setContentView(binding.root)

    supportActionBar?.hide() // Hiding App bar

    requestCameraPermission()   // Requesting Camera Permission

这是我的Request Camera Permission 函数。

private fun requestCameraPermission() 
    speakOut("Allow Eynetic to access the camera to take photos or videos.")

    EasyPermissions.requestPermissions(
        this,
        "This app can not work without camera.",
        Constants.PERMISSION_CAMERA_REQUEST_CODE,
        permission.CAMERA
    )


override fun onRequestPermissionsResult(
    requestCode: Int,
    permissions: Array<out String>,
    grantResults: IntArray
) 
    super.onRequestPermissionsResult(requestCode, permissions, grantResults)

    EasyPermissions.onRequestPermissionsResult(requestCode, permissions, grantResults, this)


override fun onPermissionsDenied(requestCode: Int, perms: List<String>) 
    speakOut("Permissions Denied.")

    if (EasyPermissions.somePermissionDenied(this, perms.first()))
        SettingsDialog.Builder(this).build()
            .show()  // If permissions are permanently denied show settings.
    else 
        requestCameraPermission()


override fun onPermissionsGranted(requestCode: Int, perms: List<String>) 
    cameraProviderFuture = ProcessCameraProvider.getInstance(this)
    cameraProviderFuture.addListener(
        startCamera(cameraProviderFuture.get())
    , ContextCompat.getMainExecutor(this))

    // Introduction
    val intro =
        "Welcome to Eyenetic. Please activate any module through voice command.The nodules are obstacle detection, scene recognition, currency detection, human object interaction and human human interaction."
    speakOut(intro)

    allButtons()

只有在获得许可时，我才 addListener 到 cameraProviderFuture。当获得许可时，我在没有运行模型的情况下启动相机。请注意每次没有模型运行时应用打开的时间。

@SuppressLint("UnsafeOptInUsageError")
private fun startCamera(cameraProvider: ProcessCameraProvider) 
    val preview = Preview.Builder().build()

    val cameraSelector =
        CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

    preview.setSurfaceProvider(binding.previewView.surfaceProvider)

    cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, preview)

以及获取Object-Detection模型的代码。

private fun readyObjectDetectionModel()

    val localModel = LocalModel.Builder().setAbsoluteFilePath("object_detection.tflite").build()
    val customObjectDetectionOptions = CustomObjectDetectorOptions.Builder(localModel)
        .setDetectorMode(CustomObjectDetectorOptions.STREAM_MODE)
        .enableClassification()
        .setClassificationConfidenceThreshold(0.5f)
        .build()

    objectDetector = ObjectDetection.getClient(customObjectDetectionOptions)

Object-Detection 的代码

    @SuppressLint("UnsafeOptInUsageError")
private fun startObjectDetection(cameraProvider: ProcessCameraProvider) 
    val preview = Preview.Builder().build()

    val cameraSelector =
        CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

    preview.setSurfaceProvider(binding.previewView.surfaceProvider)
    
    val imageAnalysis = ImageAnalysis.Builder().setTargetResolution(Size(1280, 720))
        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build()

    imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this),
        imageProxy ->
            val rotationDegrees =imageProxy.imageInfo.rotationDegrees
            val image = imageProxy.image

            if (image != null) 
                val processImage = InputImage.fromMediaImage(image, rotationDegrees)

                objectDetector.process(processImage)
                    .addOnSuccessListener 
                        imageProxy.close()
                    .addOnFailureListener
                        imageProxy.close()
                    
            
    )
    cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, imageAnalysis, preview)

我可以通过一次调用每个代码并评论另一个来运行相机代码。但我制作了不同的按钮来触发不同的模型。在调用每个方法之前我应该做什么。

现在我能想到的唯一方法是取消绑定以前的生命周期并创建一个新的。但是像
这样的冗余代码呢？

val preview = Preview.Builder().build()

val cameraSelector =
    CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

preview.setSurfaceProvider(binding.previewView.surfaceProvider)