融云分析iOS 基于实时音视频 SDK 实现屏幕共享功能

Posted 2020-12-21

Replaykit 介绍

在之前的 ios 版本中，iOS 开发者只能拿到编码后的数据，拿不到原始的 PCM 和 YUV，到 iOS 10 之后，开发者可以拿到原始数据，但是只能录制 App 内的内容，如果切到后台，将停止录制，直到 iOS 11，苹果对屏幕共享进行了升级并开放了权限，既可以拿到原始数据，又可以录制整个系统，以下我们重点来说 iOS 11 之后的屏幕共享功能。

系统屏幕共享

- (void)initMode_1 {
    self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, ScreenWidth, 80)];
    self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.replaytest.Recoder";
    self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
    self.systemBroadcastPickerView.showsMicrophoneButton = NO;
    [self.view addSubview:self.systemBroadcastPickerView];
}

在 iOS 11 创建一个 Extension 之后，调用上面的代码就可以开启屏幕共享了，然后系统会为我们生成一个 SampleHandler 的类，在这个方法中，苹果会根据 RPSampleBufferType 上报不同类型的数据。

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType

那怎么通过融云的 RongRTCLib 将屏幕共享数据发送出去呢？

1. 基于 Socket 的逼格玩法

1.1. Replaykit 框架启动和创建 Socket

//
//  ViewController.m
//  Socket_Replykit
//
//  Created by Sun on 2020/5/19.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "ViewController.h"
#import <ReplayKit/ReplayKit.h>
#import "RongRTCServerSocket.h"

@interface ViewController ()<RongRTCServerSocketProtocol>

@property (nonatomic, strong) RPSystemBroadcastPickerView *systemBroadcastPickerView;
/**
 server socket
 */
@property(nonatomic , strong)RongRTCServerSocket *serverSocket;

@end

@implementation ViewController

- (void)viewDidLoad {
    [super viewDidLoad];
    self.view.backgroundColor = [UIColor whiteColor];
    // Do any additional setup after loading the view.
    [self.serverSocket createServerSocket];

    self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, [UIScreen mainScreen].bounds.size.width, 80)];
    self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.sealrtc.RongRTCRP";
    self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
    self.systemBroadcastPickerView.showsMicrophoneButton = NO;
    [self.view addSubview:self.systemBroadcastPickerView];
}

- (RongRTCServerSocket *)serverSocket {
    if (!_serverSocket) {
        RongRTCServerSocket *socket = [[RongRTCServerSocket alloc] init];
        socket.delegate = self;

        _serverSocket = socket;
    }
    return _serverSocket;
}

- (void)didProcessSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    // 这里拿到了最终的数据，比如最后可以使用融云的音视频SDK RTCLib 进行传输就可以了
}

@end

其中，包括了创建 Server Socket 的步骤，我们把主 App 当做 Server，然后屏幕共享 Extension 当做 Client ，通过 Socket 向我们的主 APP 发送数据。

在 Extension 里面，我们拿到 ReplayKit 框架上报的屏幕视频数据后：

//
//  SampleHandler.m
//  SocketReply
//
//  Created by Sun on 2020/5/19.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "SampleHandler.h"
#import "RongRTCClientSocket.h"
@interface SampleHandler()

/**
 Client Socket
 */
@property (nonatomic, strong) RongRTCClientSocket *clientSocket;

@end

@implementation SampleHandler

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
    self.clientSocket = [[RongRTCClientSocket alloc] init];
    [self.clientSocket createCliectSocket];
}

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            [self sendData:sampleBuffer];
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

- (void)sendData:(CMSampleBufferRef)sampleBuffer {
    [self.clientSocket encodeBuffer:sampleBuffer];
}

@end

可见 ，这里我们创建了一个 Client Socket，然后拿到屏幕共享的视频 sampleBuffer 之后，通过 Socket 发给我们的主 App，这就是屏幕共享的流程。

1.2 Local Socket 的使用

//
//  RongRTCSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "RongRTCSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import "RongRTCThread.h"

@interface RongRTCSocket()

/**
 receive thread
 */
@property (nonatomic, strong) RongRTCThread *receiveThread;

@end

@implementation RongRTCSocket
- (int)createSocket {
    int socket = socket(AF_INET, SOCK_STREAM, 0);
    self.socket = socket;
    if (self.socket == -1) {
        close(self.socket);
        NSLog(@"socket error : %d", self.socket);
    }

    self.receiveThread = [[RongRTCThread alloc] init];
    [self.receiveThread run];
    return socket;
}

- (void)setSendBuffer {
    int optVal = 1024 * 1024 * 2;
    int optLen = sizeof(int);
    int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDBUF, (char *)&optVal,optLen);
    NSLog(@"set send buffer:%d", res);
}

- (void)setReceiveBuffer {
    int optVal = 1024 * 1024 * 2;
    int optLen = sizeof(int);
    int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVBUF, (char*)&optVal,optLen );
    NSLog(@"set send buffer:%d",res);
}

- (void)setSendingTimeout {
    struct timeval timeout = {10,0};
    int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(int));
    NSLog(@"set send timeout:%d", res);
}

- (void)setReceiveTimeout {
    struct timeval timeout = {10, 0};
    int  res = setsockopt(self.socket, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(int));
    NSLog(@"set send timeout:%d", res);
}

- (BOOL)connect {
    NSString *serverHost = [self ip];
    struct hostent *server = gethostbyname([serverHost UTF8String]);
    if (server == NULL) {
        close(self.socket);
        NSLog(@"get host error");
        return NO;
    }

    struct in_addr *remoteAddr = (struct in_addr *)server->h_addr_list[0];
    struct sockaddr_in addr;
    addr.sin_family = AF_INET;
    addr.sin_addr = *remoteAddr;
    addr.sin_port = htons(CONNECTPORT);
    int res = connect(self.socket, (struct sockaddr *) &addr, sizeof(addr));
    if (res == -1) {
        close(self.socket);
        NSLog(@"connect error");
        return NO;
    }

    NSLog(@"socket connect to server success");
    return YES;
}

- (BOOL)bind {
    struct sockaddr_in client;
    client.sin_family = AF_INET;
    NSString *ipStr = [self ip];
    if (ipStr.length <= 0) {
        return NO;
    }

    const char *ip = [ipStr cStringUsingEncoding:NSASCIIStringEncoding];
    client.sin_addr.s_addr = inet_addr(ip);
    client.sin_port = htons(CONNECTPORT);
    int bd = bind(self.socket, (struct sockaddr *) &client, sizeof(client));
    if (bd == -1) {
        close(self.socket);
        NSLog(@"bind error: %d", bd);
        return NO;
    }
    return YES;
}

- (BOOL)listen {
    int ls = listen(self.socket, 128);
    if (ls == -1) {
        close(self.socket);
        NSLog(@"listen error: %d", ls);
        return NO;
    }
    return YES;
}

- (void)receive {
    dispatch_async(dispatch_get_global_queue(0, 0), ^{
        [self receiveData];
    });
}

- (NSString *)ip {
    NSString *ip = nil;
    struct ifaddrs *addrs = NULL;
    struct ifaddrs *tmpAddrs = NULL;
    BOOL res = getifaddrs(&addrs);
    if (res == 0) {
        tmpAddrs = addrs;
        while (tmpAddrs != NULL) {
            if (tmpAddrs->ifa_addr->sa_family == AF_INET) {
                // Check if interface is en0 which is the wifi connection on the iPhone
                NSLog(@"%@", [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]);
                if ([[NSString stringWithUTF8String:tmpAddrs->ifa_name] isEqualToString:@"en0"]) {
                    // Get NSString from C String
                    ip = [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)];
                }
            }
            tmpAddrs = tmpAddrs->ifa_next;
        }
    }

    // Free memory
    freeifaddrs(addrs);
    NSLog(@"%@",ip);
    return ip;
}

- (void)close {
    int res = close(self.socket);
    NSLog(@"shut down: %d", res);
}

- (void)receiveData {
}

- (void)dealloc {
    [self.receiveThread stop];
}
@end

首先创建了一个 Socket 的父类，然后用 Server Socket 和 Client Socket 分别继承类来实现链接、绑定等操作。可以看到有些数据可以设置，有些则不用，这里不是核心，核心是怎样收发数据。

1.3 发送屏幕共享数据

//
//  RongRTCClientSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "RongRTCClientSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoEncoder.h"

@interface RongRTCClientSocket() <RongRTCCodecProtocol> {
    pthread_mutex_t lock;
}

/**
 video encoder
 */
@property (nonatomic, strong) RongRTCVideoEncoder *encoder;

/**
 encode queue
 */
@property (nonatomic, strong) dispatch_queue_t encodeQueue;

@end

@implementation RongRTCClientSocket

- (BOOL)createClientSocket {
    if ([self createSocket] == -1) {
        return NO;
    }

    BOOL isC = [self connect];
    [self setSendBuffer];
    [self setSendingTimeout];

    if (isC) {
        _encodeQueue = dispatch_queue_create("cn.rongcloud.encodequeue", NULL);
        [self createVideoEncoder];
        return YES;
    } else {
        return NO;
    }
}

- (void)createVideoEncoder {
    self.encoder = [[RongRTCVideoEncoder alloc] init];
    self.encoder.delegate = self;

    RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
    settings.width = 720;
    settings.height = 1280;
    settings.startBitrate = 300;
    settings.maxFramerate = 30;
    settings.minBitrate = 1000;
    [self.encoder configWithSettings:settings onQueue:_encodeQueue];
}

- (void)clientSend:(NSData *)data {
    //data length
    NSUInteger dataLength = data.length;

    // data header struct
    DataHeader dataH;
    memset((void *)&dataH, 0, sizeof(dataH));

    // pre
    PreHeader preH;
    memset((void *)&preH, 0, sizeof(preH));
    preH.pre[0] = ‘&‘;
    preH.dataLength = dataLength;

    dataH.preH = preH;

    // buffer
    int headerlength = sizeof(dataH);
    int totalLength = dataLength + headerlength;

    // srcbuffer
    Byte *src = (Byte *)[data bytes];

    // send buffer
    char *buffer = (char *)malloc(totalLength * sizeof(char));
    memcpy(buffer, &dataH, headerlength);
    memcpy(buffer + headerlength, src, dataLength);

    // tosend
    [self sendBytes:buffer length:totalLength];
    free(buffer);
}

- (void)encodeBuffer:(CMSampleBufferRef)sampleBuffer {
    [self.encoder encode:sampleBuffer];
}

- (void)sendBytes:(char *)bytes length:(int)length {
    LOCK(self->lock);
    int hasSendLength = 0;

    while (hasSendLength < length) {
        // connect socket success
        if (self.socket > 0) {
            // send
            int sendRes = send(self.socket, bytes, length - hasSendLength, 0);
            if (sendRes == -1 || sendRes == 0) {
                UNLOCK(self->lock);
                NSLog(@"send buffer error");
                [self close];
                break;
            }

            hasSendLength += sendRes;
            bytes += sendRes;
        } else {
            NSLog(@"client socket connect error");
            UNLOCK(self->lock);
        }
    }
    UNLOCK(self->lock); 
}

- (void)spsData:(NSData *)sps ppsData:(NSData *)pps {
    [self clientSend:sps];
    [self clientSend:pps];
}

- (void)naluData:(NSData *)naluData {
    [self clientSend:naluData];
}

- (void)deallo c{
    NSLog(@"dealoc cliect socket");
}

@end

这里的核心思想是拿到屏幕共享的数据之后，先进行压缩，当压缩完成后会通过回调上报给当前类。既而通过 clientSend 方法，发给主 App。发给主 App 的数据中自定义了一个头部，头部添加了一个前缀和一个每次发送字节的长度，当接收端收到数据包后解析即可。

- (void)clientSend:(NSData *)data {
    //data length
    NSUInteger dataLength = data.length;

    // data header struct
    DataHeader dataH;
    memset((void *)&dataH, 0, sizeof(dataH));

    // pre
    PreHeader preH;
    memset((void *)&preH, 0, sizeof(preH));
    preH.pre[0] = ‘&‘;
    preH.dataLength = dataLength;

    dataH.preH = preH;

    // buffer
    int headerlength = sizeof(dataH);
    int totalLength = dataLength + headerlength;

    // srcbuffer
    Byte *src = (Byte *)[data bytes];

    // send buffer
    char *buffer = (char *)malloc(totalLength * sizeof(char));
    memcpy(buffer, &dataH, headerlength);
    memcpy(buffer + headerlength, src, dataLength);

    // to send
    [self sendBytes:buffer length:totalLength];
    free(buffer);
}

1.4 接收屏幕共享数据

//
//  RongRTCServerSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "RongRTCServerSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import <UIKit/UIKit.h>
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoDecoder.h"

@interface RongRTCServerSocket() <RongRTCCodecProtocol>
{
    pthread_mutex_t lock;
    int _frameTime;
    CMTime _lastPresentationTime;
    Float64 _currentMediaTime;
    Float64 _currentVideoTime;
    dispatch_queue_t _frameQueue;
}

@property (nonatomic, assign) int acceptSocket;

/**
 data length
 */
@property (nonatomic, assign) NSUInteger dataLength;

/**
 timeData
 */
@property (nonatomic, strong) NSData *timeData;

/**
 decoder queue
 */
@property (nonatomic, strong) dispatch_queue_t decoderQueue;

/**
 decoder
 */
@property (nonatomic, strong) RongRTCVideoDecoder *decoder;

@end

@implementation RongRTCServerSocket

- (BOOL)createServerSocket {
    if ([self createSocket] == -1) {
        return NO;
    }

    [self setReceiveBuffer];
    [self setReceiveTimeout];
    BOOL isB = [self bind];
    BOOL isL = [self listen];

    if (isB && isL) {
        _decoderQueue = dispatch_queue_create("cn.rongcloud.decoderQueue", NULL);
        _frameTime = 0;
        [self createDecoder];
        [self receive];
        return YES;
    } else {
        return NO;
    }
}

- (void)createDecoder {
    self.decoder = [[RongRTCVideoDecoder alloc] init];
    self.decoder.delegate = self;
    RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
    settings.width = 720;
    settings.height = 1280;
    settings.startBitrate = 300;
    settings.maxFramerate = 30;
    settings.minBitrate = 1000;
    [self.decoder configWithSettings:settings onQueue:_decoderQueue];
}

- (void)receiveData {
    struct sockaddr_in rest;
    socklen_t rest_size = sizeof(struct sockaddr_in);
    self.acceptSocket = accept(self.socket, (struct sockaddr *) &rest, &rest_size);
    while (self.acceptSocket != -1) {
        DataHeader dataH;
        memset(&dataH, 0, sizeof(dataH));

        if (![self receiveData:(char *)&dataH length:sizeof(dataH)]) {
            continue;
        }

        PreHeader preH = dataH.preH;
        char pre = preH.pre[0];
        if (pre == ‘&‘) {
            // rongcloud socket
            NSUInteger dataLenght = preH.dataLength;
            char *buff = (char *)malloc(sizeof(char) * dataLenght);
            if ([self receiveData:(char *)buff length:dataLenght]) {
                NSData *data = [NSData dataWithBytes:buff length:dataLenght];
                [self.decoder decode:data];
                free(buff);
            }
        } else {
            NSLog(@"pre is not &");
            return;
        }
    }
}

- (BOOL)receiveData:(char *)data length:(NSUInteger)length {
    LOCK(lock);
    int receiveLength = 0;
    while (receiveLength < length) {
        ssize_t res = recv(self.acceptSocket, data, length - receiveLength, 0);
        if (res == -1 || res == 0) {
            UNLOCK(lock);
            NSLog(@"receive data error");
            break;
        }

        receiveLength += res;
        data += res;
    }

    UNLOCK(lock);
    return YES;
}

- (void)didGetDecodeBuffer:(CVPixelBufferRef)pixelBuffer {
    _frameTime += 1000;
    CMTime pts = CMTimeMake(_frameTime, 1000);
    CMSampleBufferRef sampleBuffer = [RongRTCBufferUtil sampleBufferFromPixbuffer:pixelBuffer time:pts];
    // Check to see if there is a problem with the decoded data. If the image appears, you are right.
    UIImage *image = [RongRTCBufferUtil imageFromBuffer:sampleBuffer];
    [self.delegate didProcessSampleBuffer:sampleBuffer];
    CFRelease(sampleBuffer);
}

- (void)close {
    int res = close(self.acceptSocket);
    self.acceptSocket = -1;
    NSLog(@"shut down server: %d", res);
    [super close];
}

- (void)dealloc {
    NSLog(@"dealoc server socket");
}

@end

主 App 通过 Socket 会持续收到数据包，再将数据包进行解码，将解码后的数据通过代理 didGetDecodeBuffer 代理方法回调给 App 层。App 层就可以通过融云 RongRTCLib 的发送自定义流方法将视频数据发送到对端。

1.5 VideotoolBox 硬编码

//
//  RongRTCVideoEncoder.m
//  SealRTC
//
//  Created by Sun on 2020/5/13.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "RongRTCVideoEncoder.h"

#import "helpers.h"

@interface RongRTCVideoEncoder() {
    VTCompressionSessionRef _compressionSession;
    int _frameTime;
}

/**
 settings
 */
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;

/**
 callback queue
 */
@property (nonatomic , strong ) dispatch_queue_t callbackQueue;

- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer;
- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer;

@end

void compressionOutputCallback(void *encoder,
                               void *params,
                               OSStatus status,
                               VTEncodeInfoFlags infoFlags,
                               CMSampleBufferRef sampleBuffer) {
    RongRTCVideoEncoder *videoEncoder = (__bridge RongRTCVideoEncoder *)encoder;
    if (status != noErr) {
        return;
    }

    if (infoFlags & kVTEncodeInfo_FrameDropped) {
        return;
    }

    BOOL isKeyFrame = NO;
    CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);

    if (attachments != nullptr && CFArrayGetCount(attachments)) {
        CFDictionaryRef attachment = static_cast<CFDictionaryRef>(CFArrayGetValueAtIndex(attachments, 0)) ;
        isKeyFrame = !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync);
    }

    CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    CMBlockBufferRef contiguous_buffer = nullptr;

    if (!CMBlockBufferIsRangeContiguous(block_buffer, 0, 0)) {
        status = CMBlockBufferCreateContiguous(nullptr, block_buffer, nullptr, nullptr, 0, 0, 0, &contiguous_buffer);
        if (status != noErr) {
            return;
        }
    } else {
        contiguous_buffer = block_buffer;
        CFRetain(contiguous_buffer);
        block_buffer = nullptr;
    }

    size_t block_buffer_size = CMBlockBufferGetDataLength(contiguous_buffer);
    if (isKeyFrame) {
        [videoEncoder sendSpsAndPPSWithSampleBuffer:sampleBuffer];
    }

    if (contiguous_buffer) {
        CFRelease(contiguous_buffer);
    }

    [videoEncoder sendNaluData:sampleBuffer];
}

@implementation RongRTCVideoEncoder

@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;

- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(nonnull dispatch_queue_t)queue {
    self.settings = settings;
    if (queue) {
        _callbackQueue = queue;
    } else {
        _callbackQueue = dispatch_get_main_queue();
    }

    if ([self resetCompressionSession:settings]) {
        _frameTime = 0;
        return YES;
    } else {
        return NO;
    }
}

- (BOOL)resetCompressionSession:(RongRTCVideoEncoderSettings *)settings {
    [self destroyCompressionSession];
    OSStatus status = VTCompressionSessionCreate(nullptr, settings.width, settings.height, kCMVideoCodecType_H264, nullptr, nullptr, nullptr, compressionOutputCallback, (__bridge void * _Nullable)(self), &_compressionSession);
    if (status != noErr) {
        return NO;
    }

    [self configureCompressionSession:settings];
    return YES;
}

- (void)configureCompressionSession:(RongRTCVideoEncoderSettings *)settings {
    if (_compressionSession) {
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_RealTime, true);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AllowFrameReordering, false);

        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, 10);
        uint32_t targetBps = settings.startBitrate * 1000;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, targetBps);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ExpectedFrameRate, settings.maxFramerate);
        int bitRate = settings.width * settings.height * 3 * 4 * 4;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, bitRate);
        int bitRateLimit = settings.width * settings.height * 3 * 4;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimit);
    }
}

- (void)encode:(CMSampleBufferRef)sampleBuffer {
    CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
    CMTime pts = CMTimeMake(self->_frameTime++, 1000);
    VTEncodeInfoFlags flags;
    OSStatus res = VTCompressionSessionEncodeFrame(self->_compressionSession,
                                                   imageBuffer,
                                                   pts,
                                                   kCMTimeInvalid,
                                                   NULL, NULL, &flags);

    if (res != noErr) {
        NSLog(@"encode frame error:%d", (int)res);
        VTCompressionSessionInvalidate(self->_compressionSession);
        CFRelease(self->_compressionSession);
        self->_compressionSession = NULL;
        return;
    }
}

- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
    const uint8_t *sps ;
    const uint8_t *pps;
    size_t spsSize ,ppsSize , spsCount,ppsCount;
    OSStatus spsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sps, &spsSize, &spsCount, NULL);
    OSStatus ppsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pps, &ppsSize, &ppsCount, NULL);
    if (spsStatus == noErr && ppsStatus == noErr) {
        const char bytes[] = "x00x00x00x01";
        size_t length = (sizeof bytes) - 1;

        NSMutableData *spsData = [NSMutableData dataWithCapacity:4+ spsSize];
        NSMutableData *ppsData  = [NSMutableData dataWithCapacity:4 + ppsSize];
        [spsData appendBytes:bytes length:length];
        [spsData appendBytes:sps length:spsSize];

        [ppsData appendBytes:bytes length:length];
        [ppsData appendBytes:pps length:ppsSize];
        if (self && self.callbackQueue) {
            dispatch_async(self.callbackQueue, ^{
                if (self.delegate && [self.delegate respondsToSelector:@selector(spsData:ppsData:)]) {
                    [self.delegate spsData:spsData ppsData:ppsData];
                }
            });
        }
    } else {
        NSLog(@"sps status:%@, pps status:%@", @(spsStatus), @(ppsStatus));
    }
}

- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer {
    size_t totalLength = 0;
    size_t lengthAtOffset=0;
    char *dataPointer;
    CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    OSStatus status1 = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPointer);

    if (status1 != noErr) {
        NSLog(@"video encoder error, status = %d", (int)status1);
        return;
    }

    static const int h264HeaderLength = 4;
    size_t bufferOffset = 0;

    while (bufferOffset < totalLength - h264HeaderLength) {
        uint32_t naluLength = 0;
        memcpy(&naluLength, dataPointer + bufferOffset, h264HeaderLength);
        naluLength = CFSwapInt32BigToHost(naluLength);

        const char bytes[] = "x00x00x00x01";
        NSMutableData *naluData = [NSMutableData dataWithCapacity:4 + naluLength];
        [naluData appendBytes:bytes length:4];
        [naluData appendBytes:dataPointer + bufferOffset + h264HeaderLength length:naluLength];

        dispatch_async(self.callbackQueue, ^{
            if (self.delegate && [self.delegate respondsToSelector:@selector(naluData:)]) {
                [self.delegate naluData:naluData];
            }
        });

        bufferOffset += naluLength + h264HeaderLength;
    }
}

- (void)destroyCompressionSession {
    if (_compressionSession) {
        VTCompressionSessionInvalidate(_compressionSession);
        CFRelease(_compressionSession);
        _compressionSession = nullptr;
    }
}

- (void)dealloc {
    if (_compressionSession) {
        VTCompressionSessionCompleteFrames(_compressionSession, kCMTimeInvalid);
        VTCompressionSessionInvalidate(_compressionSession);
        CFRelease(_compressionSession);
        _compressionSession = NULL;
    }
}
@end

1.6 VideotoolBox 解码

//
//  RongRTCVideoDecoder.m
//  SealRTC
//
//  Created by Sun on 2020/5/14.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "RongRTCVideoDecoder.h"
#import <UIKit/UIKit.h>
#import "helpers.h"

@interface RongRTCVideoDecoder() {
    uint8_t *_sps;
    NSUInteger _spsSize;
    uint8_t *_pps;
    NSUInteger _ppsSize;
    CMVideoFormatDescriptionRef _videoFormatDescription;
    VTDecompressionSessionRef _decompressionSession;
}

/**
 settings
 */
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;

/**
 callback queue
 */
@property (nonatomic, strong) dispatch_queue_t callbackQueue;

@end

void DecoderOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,
                           void * CM_NULLABLE sourceFrameRefCon,
                           OSStatus status,
                           VTDecodeInfoFlags infoFlags,
                           CM_NULLABLE CVImageBufferRef imageBuffer,
                           CMTime presentationTimeStamp,
                           CMTime presentationDuration ) {
    if (status != noErr) {
        NSLog(@" decoder callback error :%@", @(status));
        return;
    }

    CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;
    *outputPixelBuffer = CVPixelBufferRetain(imageBuffer);
    RongRTCVideoDecoder *decoder = (__bridge RongRTCVideoDecoder *)(decompressionOutputRefCon);
    dispatch_async(decoder.callbackQueue, ^{
        [decoder.delegate didGetDecodeBuffer:imageBuffer];
        CVPixelBufferRelease(imageBuffer);
    });
}

@implementation RongRTCVideoDecoder

@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;

- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(dispatch_queue_t)queue {
    self.settings = settings;
    if (queue) {
        _callbackQueue = queue;
    } else {
        _callbackQueue = dispatch_get_main_queue();
    }
    return YES;
}

- (BOOL)createVT {
    if (_decompressionSession) {
        return YES;
    }

    const uint8_t * const parameterSetPointers[2] = {_sps, _pps};
    const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};
    int naluHeaderLen = 4;
    OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_videoFormatDescription );
    if (status != noErr) {
        NSLog(@"CMVideoFormatDescriptionCreateFromH264ParameterSets error:%@", @(status));
        return false;
    }

    NSDictionary *destinationImageBufferAttributes =
                                        @{
                                            (id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange],
                                            (id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:self.settings.width],
                                            (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:self.settings.height],
                                            (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]
                                        };

    VTDecompressionOutputCallbackRecord CallBack;
    CallBack.decompressionOutputCallback = DecoderOutputCallback;
    CallBack.decompressionOutputRefCon = (__bridge void * _Nullable)(self);
    status = VTDecompressionSessionCreate(kCFAllocatorDefault, _videoFormatDescription, NULL, (__bridge CFDictionaryRef _Nullable)(destinationImageBufferAttributes), &CallBack, &_decompressionSession);

    if (status != noErr) {
        NSLog(@"VTDecompressionSessionCreate error:%@", @(status));
        return false;
    }

    status = VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);
    return YES;
}

- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {
    CVPixelBufferRef outputPixelBuffer = NULL;
    CMBlockBufferRef blockBuffer = NULL;
    CMBlockBufferFlags flag0 = 0;

    OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);

    if (status != kCMBlockBufferNoErr) {
        NSLog(@"VCMBlockBufferCreateWithMemoryBlock code=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }

    CMSampleBufferRef sampleBuffer = NULL;
    const size_t sampleSizeArray[] = {frameSize};

    status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _videoFormatDescription, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);

    if (status != noErr || !sampleBuffer) {
        NSLog(@"CMSampleBufferCreateReady failed status=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }

    VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;
    VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous;

    status = VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);

    if (status == kVTInvalidSessionErr) {
        NSLog(@"decode frame error with session err status =%d", (int)status);
        [self resetVT];
    } else  {
        if (status != noErr) {
            NSLog(@"decode frame error with  status =%d", (int)status);
        }
    }

    CFRelease(sampleBuffer);
    CFRelease(blockBuffer);

    return outputPixelBuffer;
}

- (void)resetVT {
    [self destorySession];
    [self createVT];
}

- (void)decode:(NSData *)data {
    uint8_t *frame = (uint8_t*)[data bytes];
    uint32_t length = data.length;
    uint32_t nalSize = (uint32_t)(length - 4);
    uint32_t *pNalSize = (uint32_t *)frame;
    *pNalSize = CFSwapInt32HostToBig(nalSize);

    int type = (frame[4] & 0x1F);
    CVPixelBufferRef pixelBuffer = NULL;
    switch (type) {
        case 0x05:
            if ([self createVT]) {
                pixelBuffer= [self decode:frame withSize:length];
            }
            break;
        case 0x07:
            self->_spsSize = length - 4;
            self->_sps = (uint8_t *)malloc(self->_spsSize);
            memcpy(self->_sps, &frame[4], self->_spsSize);
            break;
        case 0x08:
            self->_ppsSize = length - 4;
            self->_pps = (uint8_t *)malloc(self->_ppsSize);
            memcpy(self->_pps, &frame[4], self->_ppsSize);
            break;
        default:
            if ([self createVT]) {
                pixelBuffer = [self decode:frame withSize:length];
            }
            break;
    }
}

- (void)dealloc {
    [self destorySession];
}

- (void)destorySession {
    if (_decompressionSession) {
        VTDecompressionSessionInvalidate(_decompressionSession);
        CFRelease(_decompressionSession);
        _decompressionSession = NULL;
    }
}

@end

1.7 工具类

//
//  RongRTCBufferUtil.m
//  SealRTC
//
//  Created by Sun on 2020/5/8.
//  Copyright ? 2020 RongCloud. All rights reserved.
//

#import "RongRTCBufferUtil.h"

// 下面的这些方法，一定要记得release，有的没有在方法里面release，但是在外面release了，要不然会内存泄漏

@implementation RongRTCBufferUtil

+ (UIImage *)imageFromBuffer:(CMSampleBufferRef)buffer {    
    CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(buffer);
    CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];

    CIContext *temporaryContext = [CIContext contextWithOptions:nil];
    CGImageRef videoImage = [temporaryContext createCGImage:ciImage fromRect:CGRectMake(0, 0, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer))];

    UIImage *image = [UIImage imageWithCGImage:videoImage];
    CGImageRelease(videoImage);

    return image;
}

+ (UIImage *)compressImage:(UIImage *)image newWidth:(CGFloat)newImageWidth {
    if (!image) return nil;

    float imageWidth = image.size.width;
    float imageHeight = image.size.height;
    float width = newImageWidth;
    float height = image.size.height/(image.size.width/width);
    float widthScale = imageWidth /width;
    float heightScale = imageHeight /height;
    UIGraphicsBeginImageContext(CGSizeMake(width, height));

    if (widthScale > heightScale) {
        [image drawInRect:CGRectMake(0, 0, imageWidth /heightScale , height)];
    }
    else {
        [image drawInRect:CGRectMake(0, 0, width , imageHeight /widthScale)];
    }

    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();
    return newImage;
}

+ (CVPixelBufferRef)CVPixelBufferRefFromUiImage:(UIImage *)img {
    CGSize size = img.size;
    CGImageRef image = [img CGImage];

    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil];
    CVPixelBufferRef pxbuffer = NULL;
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer);

    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);

    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(pxdata != NULL);

    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, 4*size.width, rgbColorSpace, kCGImageAlphaPremultipliedFirst);
    NSParameterAssert(context);

    CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image)), image);

    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);

    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);

    return pxbuffer;
}

+ (CMSampleBufferRef)sampleBufferFromPixbuffer:(CVPixelBufferRef)pixbuffer time:(CMTime)time {
    CMSampleBufferRef sampleBuffer = NULL;

    //获取视频信息
    CMVideoFormatDescriptionRef videoInfo = NULL;
    OSStatus result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixbuffer, &videoInfo);
    CMTime currentTime = time;

    //    CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
    CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
    result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,pixbuffer, true, NULL, NULL, videoInfo, &timing, &sampleBuffer);
    CFRelease(videoInfo);
    return sampleBuffer;
}

+ (size_t)getCMTimeSize {
    size_t size = sizeof(CMTime);
    return size;
}

@end

此工具类中实现是由 CPU 处理，当进行 CMSampleBufferRef 转 UIImage、UIImage 转 CVPixelBufferRef、 CVPixelBufferRef 转 CMSampleBufferRef 以及裁剪图片时，这里需要注意将使用后的对象及时释放，否则会出现内存大量泄漏。

2. 视频发送

2.1 准备阶段

使用融云的 RongRTCLib 的前提需要一个 AppKey，请在官网（https://www.rongcloud.cn/)获取，通过 AppKey 取得 token 之后进行 IM 连接，在连接成功后加入 RTC 房间，这是屏幕共享发送的准备阶段。

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.

    // 请填写您的 AppKey
    self.appKey = @"";
    // 请填写用户的 Token
    self.token = @"";
    // 请指定房间号
    self.roomId = @"123456";

    [[RCIMClient sharedRCIMClient] initWithAppKey:self.appKey];
    [[RCIMClient sharedRCIMClient] setLogLevel:RC_Log_Level_Verbose];

    // 连接 IM
    [[RCIMClient sharedRCIMClient] connectWithToken:self.token
                                           dbOpened:^(RCDBErrorCode code) {
        NSLog(@"dbOpened: %zd", code);
    } success:^(NSString *userId) {
        NSLog(@"connectWithToken success userId: %@", userId);
        // 加入房间
        [[RCRTCEngine sharedInstance] joinRoom:self.roomId
                                    completion:^(RCRTCRoom * _Nullable room, RCRTCCode code) {
            self.room = room;
            self.room.delegate = self;
            [self publishScreenStream];
        }];
    } error:^(RCConnectErrorCode errorCode) {
        NSLog(@"ERROR status: %zd", errorCode);
    }];
}

如上是连接 IM 和加入 RTC 房间的全过程，其中还包含调用发布自定义视频 [self publishScreenStream]; 此方法在加入房间成功后才可以进行。

- (void)publishScreenStream {
    RongRTCStreamParams *param = [[RongRTCStreamParams alloc] init];
    param.videoSizePreset = RongRTCVideoSizePreset1280x720;
    self.videoOutputStream = [[RongRTCAVOutputStream alloc] initWithParameters:param tag:@"RongRTCScreenVideo"];
    [self.room publishAVStream:self.videoOutputStream extra:@"" completion:^(BOOL isSuccess, RongRTCCode desc) {
        if (isSuccess) {
            NSLog(@"发布自定义流成功");
        }
    }];
}

自定义一个 RongRTCAVOutputStream 流即可，使用此流发送屏幕共享数据。

2.2 开始发送屏幕共享数据

上面我们已经连接了融云的 IM 和加入了 RTC 房间，并且自定义了一个发送屏幕共享的自定义流，接下来，如何将此流发布出去呢？

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            [self.videoOutputStream write:sampleBuffer error:nil];
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

但我们接收到了苹果上报的数据之后，调用 RongRTCAVOutputStream 中的 write:error: 方法，将 sampleBuffer 发送给远端，至此，屏幕共享数据就发送出去啦。

[self.videoOutputStream write:sampleBuffer error:nil];

融云的核心代码就是通过上面的连接 IM，加入房间，发布自定义流，然后通过自定义流的 write:error: 方法将 sampleBuffer 发送出去。

不管是通过 ReplayKit 取得屏幕视频，还是使用 Socket 在进程间传输，都是为最终的 write:error: 服务。

总结

1. Extension 内存是有限制的，最大 50M，所以在 Extension 里面处理数据需要格外注意内存释放；
2. 如果 VideotoolBox 在后台解码一直失败，只需把 VideotoolBox 重启一下即可，此步骤在上面的代码中有体现；
3. 如果不需要将 Extension 的数据传到主 App，只需在 Extension 里直接将流通过 RongRTCLib 发布出去即可，缺点是 Extension 中发布自定义流的用户与主 App 中的用户不是同一个，这也是上面通过 Socket 将数据传递给主 App 要解决的问题；
4. 如果主 App 需要拿到屏幕共享的数据处理，使用 Socket 将流先发给主 App，然后在主 App 里面通过 RongRTCLib 将流发出去。

最后附上 Demo