在 swift3.2/4 中正确释放 MTAudioProcessingTapRef

Posted 2023-02-25

【中文标题】在 swift3.2/4 中正确释放 MTAudioProcessingTapRef

【英文标题】：Correctly deallocating MTAudioProcessingTapRef in swift3.2/4

【发布时间】：2017-10-23 13:20:15

【问题描述】：

我一直在我的项目中使用 MTAudioProcessingTapRef，以便在播放流式音频时实时分析缓冲区数据。问题是我无法让 Tap 处理器在需要时正确解除分配。

我有一个 AudioViewController swift 类，它引用了我的 AudioTapProcessor 目标 C 类，该 swift 类负责告诉处理器启动和停止 AVPlayerItem 的处理。处理器还有一个委托（在这种情况下为视图控制器）来通知处理时缓冲区的变化。

我的问题是，如果我将处理器委托声明为弱（应该如此），处理器将随机崩溃，试图通知已解除分配的委托，因为在停止处理后执行了几次点击处理器的处理方法称呼。 我发现解决此问题的唯一方法是将点击处理器委托声明为强属性，这显然会导致保留周期，并且我的 AudioViewControllers 将永远不会被释放。

下面，一些代码可以帮助你了解情况：

AudioTapProcessor.h

@interface AudioTapProcessor : NSObject

@property (nonatomic, strong) AVPlayerItem *item;
@property (nonatomic, strong) id<AudioProcessorDelegate> delegate;

- (instancetype)initWithDelegate:(id<AudioProcessorDelegate>)delegate 
    item:(AVPlayerItem *)item;
- (void)startProcessing;
- (void)stopProcessing;

@end

AudioTapProcessor.m

void init(MTAudioProcessingTapRef tap, void *clientInfo, void 
**tapStorageOut) 
    *tapStorageOut = clientInfo;


void finalize(MTAudioProcessingTapRef tap) 

void prepare(
         MTAudioProcessingTapRef tap,
         CMItemCount maxFrames,
         const AudiostreamBasicDescription *processingFormat
         ) 

void unprepare(MTAudioProcessingTapRef tap) 

void process(
         MTAudioProcessingTapRef tap,
         CMItemCount numberFrames,
         MTAudioProcessingTapFlags flags,
         AudioBufferList *bufferListInOut,
         CMItemCount *numberFramesOut,
         MTAudioProcessingTapFlags *flagsOut
         ) 
 //Random crashes here if I declare the delegate weak
 //Something like AUDeferredRenderer-0x7ff8f448ef (364): EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  AudioTapProcessor *processor = (__bridge AudioTapProcessor *)MTAudioProcessingTapGetStorage(tap);

  OSStatus err = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, NULL, numberFramesOut);

  AudioBuffer *pBuffer = &bufferListInOut->mBuffers[0];
  UInt32 frameLength = pBuffer->mDataByteSize / sizeof(float);
  float *pData = (float *)pBuffer->mData;

  if (err == noErr && processor) 
    if ([processor.delegate 
      respondsToSelector:@selector(updateWith:withSize:)]) 
      [processor.delegate updateWith:pData withSize:frameLength];
    
  
 

- (void)stopProcessing

  [self.item removeObserver:self forKeyPath:@"status"];
AVMutableAudioMixInputParameters *params =
(AVMutableAudioMixInputParameters *) _item.audioMix.inputParameters[0];
  MTAudioProcessingTapRef tap = params.audioTapProcessor;
  self.item.audioMix = nil;
  CFRelease(tap);
  //By doing this the tap processor does call its unprepare and finalize methods, so it is being deallocated fine.

然后在我的 AudioViewController.swift 我有：

var processor: AudioTapProcessor!

override func prepareForPlayback() 
  super.prepareForPlayback()
  if processor == nil 
    processor = AudioTapProcessor(delegate: self, item: item)
    processor.startProcessing()
  


override func viewWillDisappear(_ animated: Bool) 
   super.viewWillDisappear(animated)
   player.pause()


deinit 
  //I tried to do this early in the lifecycle(viewWillDissapear) and it is the same thing.
   processor.stopProcessing()

任何提示都将不胜感激，我对此感到疯狂。谢谢

【参考方案1】：

适用于所有 iOS 版本

⇨了解根本原因

1.AudioTapProcessor.m 初始化

callbacks.clientInfo 包含一个指向 self 的指针。 这不是一个弱或强引用，只是一个 C 指针。所以如果 self 被释放，我们 context->self 指向一个错误的内存地址

- (AVAudioMix *)audioMix 
    if (!_audioMix) 
        AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
        if (audioMix) 
            AVMutableAudioMixInputParameters *audioMixInputParameters = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:self.audioAssetTrack];
            if (audioMixInputParameters) 
                MTAudioProcessingTapCallbacks callbacks;

                ...
                callbacks.clientInfo = (__bridge void *)self;
                ...
            
        
     

2.AudioTapProcessor.m processCallback

每次调用 processCallback 时都会进行安全检查以查看 self 是否正在被释放，但请记住在步骤 1 中。即使 self 被释放 context->self 不是 nil 而是指向错误的内存地址导致EXC_BAD_ACCESS。

static void tap_ProcessCallback(MTAudioProcessingTapRef tap, CMItemCount numberFrames, MTAudioProcessingTapFlags flags, AudioBufferList *bufferListInOut, CMItemCount *numberFramesOut, MTAudioProcessingTapFlags *flagsOut) 
    ...
    MYAudioTapProcessor *self = ((__bridge MYAudioTapProcessor *)context->self);
    if (!self) 
      NSLog(@"AudioTapProcessor - processCallback CANCELLED (self is nil)");
      return;
    
    NSLog(@"AudioTapProcessor - processCallback PROCESSING");

⇨现在，如何解决这个问题？

1.ViewController.swift 或 audioTapProcessor 的所有者

deinit 
  print("ViewController - Dealloc")
  audioTapProcessor.stopProcessing()

2.AudioTapProcessor.m

我们需要一种方法来告诉我们的 audioTapProcessor 停止 processCallback。 最简单自然的方法是使用上面 processCallback 中已经存在的检查 if (!self) return;

所以停止 audioTapProcessor 只是正确地将 context->self 设置为 NULL。

- (void)stopProcessing 
    NSLog(@"AudioTapProcessor - stopProcessing");
    AVMutableAudioMixInputParameters *params = (AVMutableAudioMixInputParameters *)_audioMix.inputParameters[0];
    MTAudioProcessingTapRef audioProcessingTap = params.audioTapProcessor;
    AVAudioTapProcessorContext *context = (AVAudioTapProcessorContext *)MTAudioProcessingTapGetStorage(audioProcessingTap);

    // nils out the pointer so that we know in tapProcessorCallbacks that self will be dealloc'ed
    context->self = NULL;

⇨因此生命周期得到纠正

不是这个

ViewController - Dealloc AudioTapProcessor - 停止处理 AudioTapProcessor - processCallback 处理 EXC_BAD_ADDRESS

我们明白了

ViewController - Dealloc AudioTapProcessor - 停止处理 AudioTapProcessor - processCallback CANCELED (self is nil) AudioTapProcessor - unprepareCallback AudioTapProcessor - finalizeCallback AudioTapProcessor - Dealloc

【参考方案2】：

我最终通过使MTAudioProcessingTapRef 保留其AudioTapProcessor 父级来解决问题。这样它们就不会在生命周期的不同时刻被释放。

对原代码的改动：

1.首先，我们将委托设置为弱变量：

@property (nonatomic, weak) id<AudioProcessorDelegate> delegate;

2.然后，我们将 self(Our AudioTapProcessor) 的保留引用传递给创建的MTAudioProcessingTapRef：

 callbacks.clientInfo = CFRetain((__bridge void *)(self));

3.还创建了一个自定义上下文来通过水龙头传递数据：

typedef struct TapProcessorContext 
  void *self;
 TapProcessorContext;

void init(MTAudioProcessingTapRef tap, void *clientInfo, void **tapStorageOut) 
  TapProcessorContext *context = calloc(1, sizeof(TapProcessorContext));
  //Initialize TapProcessorContext context.
  context->self = clientInfo;

  *tapStorageOut = context;


void finalize(MTAudioProcessingTapRef tap) 
  TapProcessorContext *context = (TapProcessorContext 
*)MTAudioProcessingTapGetStorage(tap);
  // Clearing the context. THIS IS KEY TO DEALLOCATE THE AUDIOTAPPROCESSOR
  CFRelease(context->self);
  context->self = NULL;

  free(context);

4.最后，我对我们的 stopProcessing 方法应用了针对 iOS 中已知错误的解决方法：

- (void)stopProcessing

   if ( @available(iOS 11.0, *) ) 
    // Starting with iOS 11, it is not required to manually nil audioTapProcessor,
    // but we need to retain the audioMix for a bit to make sure the processing callback
    // will not be called after we release (this is due to a bug in iOS 11 which calls the release
    // callback but still calls the processing callback afterwards - it also releases internal data
    // on release, so simply checking for release in the processing block is not enough)
    // rdar://34977000
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(1 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^
      [self releaseTap];
    );
   else 
    // Prior to iOS 11 we need to manually nil the audioTapProcessor
    [self releaseTap];
  


-(void)releaseTap 
  AVMutableAudioMixInputParameters *params = (AVMutableAudioMixInputParameters *) _item.audioMix.inputParameters[0];
  params.audioTapProcessor = nil;
  _item.audioMix = nil;

【讨论】：

我在TapProcessorContext 结构中添加了一个布尔值invalid 标志，并在调用dispatch_after 之前设置了标志。如果在调用tap_ProcessCallback 时设置了标志，我也立即返回。