CHCSVWriter用于写入更大的CSV文件的内存使用情况

Posted 2021-03-27

我想在将数据写入CSV文件时最小化内存使用量。

对于更大的表，它使用更多的内存，即使它是临时的。

有人可以建议如何减少内存使用量？

也许我可以分离更大的表的动作，写更多的文件，然后合并它们但我还没有尝试，也许我错过了一些明显的东西。

这是当前使用的代码：

 @autoreleasepool {
    NSOutputStream *csvStream = [[NSOutputStream alloc] initToMemory];
    [csvStream open];

    CHCSVWriter *writer = [[CHCSVWriter alloc] initWithOutputStream:csvStream encoding:NSUTF8StringEncoding delimiter:';'];
    NSArray *keySortDescriptors = @[[NSSortDescriptor sortDescriptorWithKey:@"self" ascending:YES]];
    if (writeHeader==YES) {
        //> write header
        NSMutableDictionary *firstRow = [[self sharedUploadManager].modifiedRows firstObject];
        if (firstRow==nil) {
            result = NO;
            return result;
        }

        NSArray *orderedKeys = [[firstRow allKeys] sortedArrayUsingDescriptors:keySortDescriptors];
        for (NSString *columnName in  orderedKeys) {
            [writer writeField:columnName];
        }
    }
    [writer finishLine];

    @autoreleasepool {
        //> write the rows
        for (NSMutableDictionary *row in [self sharedUploadManager].modifiedRows) {

            NSArray *orderedKeys = [[row allKeys] sortedArrayUsingDescriptors:keySortDescriptors];

            for (NSString *key in orderedKeys ) {

                NSString *field = [row objectForKey:key];
                if ([field isKindOfClass:[NSNull class]]) {
                    [writer writeField:nil];
                } else {
                    [writer writeField:field];
                }
            }

            //> finish the line
            [writer finishLine];
        }
    }

    [writer closeStream];

    NSData *buffer = [csvStream propertyForKey:NSStreamDataWrittenToMemoryStreamKey];
    NSString *output = [[NSString alloc] initWithData:buffer encoding:NSUTF8StringEncoding];

    if (![[NSFileManager defaultManager] fileExistsAtPath:csvPath]) {
        [[NSFileManager defaultManager] createFileAtPath:csvPath contents:nil attributes:nil];
    }

    BOOL res = [[output dataUsingEncoding:NSUTF8StringEncoding] writeToFile:csvPath atomically:NO];

    if (!res) {
        NSLog(@"Error Creating CSV File path = %@", csvPath);
    } else{
        NSLog(@"Data saved! File path = %@", csvPath);

    }
}

我之前也试过这个逻辑 - 有点干净，但结果相同：

NSOutputStream *csvStream = [[NSOutputStream alloc] initToFileAtPath:csvPath append:YES];
[csvStream open];

CHCSVWriter *writer = [[CHCSVWriter alloc] initWithOutputStream:csvStream encoding:NSUTF8StringEncoding delimiter:';'];

if (writeHeader==YES) {
    //> write header
    NSMutableDictionary *firstRow = [rows firstObject];
    if (firstRow==nil) {
        result = NO;
        return result;
    }

    NSArray *orderedKeys = [[firstRow allKeys] sortedArrayUsingDescriptors:@[[NSSortDescriptor sortDescriptorWithKey:@"self" ascending:YES]]];

    for (NSString *columnName in  orderedKeys) {
        [writer writeField:columnName];
    }
    [writer finishLine];
}


//> write the rows
for (NSMutableDictionary *row in rows) {

    NSArray *orderedKeys = [[row allKeys] sortedArrayUsingDescriptors:@[[NSSortDescriptor sortDescriptorWithKey:@"self" ascending:YES]]];

    for (NSString *key in orderedKeys ) {

        NSString *field = [row objectForKey:key];
        if ([field isKindOfClass:[NSNull class]]) {
            [writer writeField:nil];
        }
        else {
            [writer writeField:field];
        }

    }
    //> finish the line
    [writer finishLine];
}
[writer closeStream];

答案

如果在创建大型CSV文件时不想使用大量内存，则不要创建基于内存的输出流。创建实际文件的输出流。然后CSV数据将写入文件而不是内存。然后该文件可以是千兆字节并且使用非常少的内存。

这样做的另一个好处是不需要访问缓冲区数据，从中创建一个字符串（现在是内存使用量的两倍），然后将字符串写入文件。

NSOutputStream *csvStream = [NSOutputStream outputStreamToFileAtPath:csvPath append:NO];
[csvStream open];
CHCSVWriter *writer = [[CHCSVWriter alloc] initWithOutputStream:csvStream encoding:NSUTF8StringEncoding delimiter:';'];

// write your CSV entries

[writer closeStream];

而已。无需其他代码即可创建该文件。

除了这些更改之外，您还需要更改使用自动释放池的位置。它应该在外部for循环内。

//> write the rows
for (NSMutableDictionary *row in [self sharedUploadManager].modifiedRows) {
    @autoreleasepool {
        NSArray *orderedKeys = [[row allKeys] sortedArrayUsingDescriptors:keySortDescriptors];

        for (NSString *key in orderedKeys ) {

            NSString *field = [row objectForKey:key];
            if ([field isKindOfClass:[NSNull class]]) {
                [writer writeField:nil];
            } else {
                [writer writeField:field];
            }
        }

        //> finish the line
        [writer finishLine];
    }
}

这将确保在每行之后清除自动释放对象的内存。