在 Swift 中设置 CoreML 模型 - 图像分类器

Posted

技术标签:

【中文标题】在 Swift 中设置 CoreML 模型 - 图像分类器【英文标题】:Setting up a CoreML Model in Swift - Image Classifier 【发布时间】:2020-06-30 19:54:55 【问题描述】:

我已经训练了一个模型来区分恶性和良性皮肤病变,以潜在地检测患者是否患有皮肤癌,并将我的 keras 模型转换为 coreML。现在,我正在尝试使用 swift(通过 Xcode)将我的模型应用于 ios 应用程序,而我完全没有经验(仍在通过反复试验学习)。

目前我正试图通过一个简单的应用程序让模型工作,该应用程序只从手机的摄像头获取图像以获取预测的标签作为输出,但我非常坚持让摄像头实际工作来做到这一点。

import UIKit
import CoreML
import Vision
import Social

@UIApplicationMain
class ViewControl: UIViewController, UIImagePickerControllerDelegate, UINavigationControllerDelegate, UIApplicationDelegate 

    @IBOutlet weak var imageView: UIImageView!
    var classificationResults : [VNClassificationObservation] = []

    let imagePicker = UIImagePickerController()

    override func viewDidLoad() 
        super.viewDidLoad()

        imagePicker.delegate = self

    

    func detect(image: CIImage) 

        // Load the ML model through its generated class
        guard let model = try? VNCoreMLModel(for: weights_skin_cancer().model) else 
            fatalError("can't load ML model")
        

        let request = VNCoreMLRequest(model: model)  request, error in
            guard let results = request.results as? [VNClassificationObservation],
                let topResult = results.first
                else 
                    fatalError("unexpected result type from VNCoreMLRequest")
                

                if topResult.identifier.contains("malignant") 
                    DispatchQueue.main.async 
                        self.navigationItem.title = "mal!"
                        self.navigationController?.navigationBar.barTintColor = UIColor.green
                        self.navigationController?.navigationBar.isTranslucent = false


                    
                
                else 
                    DispatchQueue.main.async 
                        self.navigationItem.title = "benign!"
                        self.navigationController?.navigationBar.barTintColor = UIColor.red
                        self.navigationController?.navigationBar.isTranslucent = false

                    
                


        

        let handler = VNImageRequestHandler(ciImage: image)

        do  try handler.perform([request]) 
        catch  print(error) 



    


    func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) 


        if let image = info[UIImagePickerController.InfoKey.originalImage] as? UIImage 

            imageView.image = image

            imagePicker.dismiss(animated: true, completion: nil)


            guard let ciImage = CIImage(image: image) else 
                fatalError("couldn't convert uiimage to CIImage")
            

            detect(image: ciImage)

        
    


    @IBAction func cameraTapped(_ sender: Any) 

        imagePicker.sourceType = .camera
        imagePicker.allowsEditing = false

        present(imagePicker, animated: true, completion: nil)
    


这也是用于将我的模型转换为 coreML 的代码以供参考:

import coremltools

output_labels = ['benign', 'malignant']
scale = 1/255.
coreml_model = coremltools.converters.keras.convert('/Users/Grampun/Desktop/ISIC-Archive-Downloader-master/trained_models/lr_0.00006-400_DS-20_epochs/weights.best.from_scratch.6.hdf5',
                                                    input_names='image',
                                                    image_input_names='image',
                                                    output_names='output',
                                                    class_labels=output_labels,
                                                    image_scale=scale)

coreml_model.author = 'Jack Bugeja'
coreml_model.short_description = 'Model used to identify between benign and malignant skin lesions'

coreml_model.input_description['image'] = 'Dermascopic image of skin lesion to evaluate'
coreml_model.input_description['output'] = 'Malignant/Benign'

coreml_model.save(
    '/Users/Grampun/Desktop/ISIC-Archive-Downloader-master/trained_models/model_for_ios/lr_0.00006-400_DS-20_epochs/weights_skin_cancer.mlmodel')

任何一般的帮助将不胜感激。 谢谢!

【问题讨论】:

你问的实际问题是什么? @MatthijsHollemans 抱歉,如果不清楚,我在问如何让相机工作 - 我的代码中的什么导致我的相机一旦按下就无法激活。 【参考方案1】:

    打开相机:

    @IBAction func cameraTapped(_ sender: Any) 
        let controller = UIImagePickerController()
        controller.sourceType = .camera
        controller.mediaTypes = ["public.image"]
        controller.allowsEditing = false
        controller.delegate = self
        present(controller, animated: true)
    
    

    YourModel.mlmodel 添加到您的项目中。

    didFinishPickingMediaWithInfo 中添加此代码:

    if let imageURL = info[.imageURL] as? URL 
        if let image = UIImage(contentsOfFile: imageURL.absoluteString) 
            self.getPrediction(image)
        
    
    

    添加这个以获得预测:

    func getPrediction(_ image: UIImage) 
        let model = YourModel()
    
        guard let pixelBuffer = buffer(from: image) else  return 
        guard let prediction = try? model.prediction(image: pixelBuffer) else  return 
    
        print(prediction.classLabel) // Most likely image category as string value
    
    

    使用此辅助函数从您的UIImage 生成一个CVPixelBuffer,您需要在getPrediction() 中使用它

    func buffer(from image: UIImage) -> CVPixelBuffer? 
        let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
        var pixelBuffer : CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
        guard (status == kCVReturnSuccess) else 
            return nil
        
    
        CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
    
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
    
        context?.translateBy(x: 0, y: image.size.height)
        context?.scaleBy(x: 1.0, y: -1.0)
    
        UIGraphicsPushContext(context!)
        image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    
        return pixelBuffer
    
    

【讨论】:

感谢您的回复。不幸的是,我目前无法打开相机。 @Grampun 我编辑了我的答案,看看是否能解决你的问题 您可能还需要为您的应用的 Info.plist 添加权限,否则将无法访问相机。在 Xcode 的调试输出窗格中应该有关于此的错误消息。 @MatthijsHollemans 你是对的。需要在Info.plist中添加Privacy - Camera Usage Description @MatthijsHollemans 将权限添加到 .plist 但相机仍然无法正常工作。将尝试更多地工作以希望它能够正常工作......

以上是关于在 Swift 中设置 CoreML 模型 - 图像分类器的主要内容,如果未能解决你的问题,请参考以下文章

如何将 CoreML 模型添加到 Swift 包中?

如何在 Swift 中设置半圆 UIImage 像这个截图

如何在 swift 4 中的 UserDefalts 中设置自定义类数组数据

在Swift中的导航栏中设置图像

Swift 在移动端使用CoreML

Swift中Vision / CoreML对象识别器的精度