使用 EPPlus 库从 SQL Server 导出到 Excel 时出现“内存不足”异常

Posted 2023-05-07

【中文标题】使用 EPPlus 库从 SQL Server 导出到 Excel 时出现“内存不足”异常

【英文标题】："Out Of Memory" exception while exporting from SQL Server to Excel, using EPPlus library

【发布时间】：2012-12-16 13:53:17

【问题描述】：

我正在使用 EPPlus .NET 库将数据从 SQL Server 导出到 Excel 文件。

我正在使用SqlConnection 类来读取数据。对于SqlDataReader游标的每一行，我都会遍历对应行的所有excel单元格，并输入来自阅读器的数据。

问题是当我将这个函数用于大型表时出现“内存不足”错误。我需要一种在Read CURSOR 中创建某种缓冲区的方法。

简洁的代码示例：

Dim sqlConnection As SqlConnection = New SqlConnection()
sqlConnection.ConnectionString = sqlConnectionString.ConnectionString 'connectionstring built before

Dim query As SqlCommand = New SqlCommand(query...)

Dim newFileStream As New FileStream("c:\junk\test.xlsx", System.IO.FileMode.Create,System.IO.FileAccess.ReadWrite)

Using excelApp As New ExcelPackage(newFileStream)
    sqlConnection.Open()
    Dim sqlReader As SqlDataReader = query.ExecuteReader()

    Dim numOfColumns As Byte = sqlReader.FieldCount()
    Dim rowNumber As Integer = 1

    While sqlReader.Read()
        Dim currentColumn As Byte

        For currentColumn = 1 To numOfColumns
            ws.Cells(rowNumber,currentColumn).Value = sqlReader.Item(currentColumn - 1)
        Next
     rowNumber += 1             
    End While

    excelApp.Save()
End Using

newFileStream.Close()

【问题讨论】：

这是相当不完整的。 rowNumber 在哪里定义和增加？ query 中有多少列？

那些东西与问题无关，所以我把它们排除在外。无论如何，我编辑了代码并添加了它。

我正在尝试复制问题，我们在这里谈论的表大小大约是多少？你知道 excel = 2007 有 1M 行限制：office.microsoft.com/en-001/excel-help/…, office.microsoft.com/en-001/excel-help/…

我知道这些规格。谢谢你。数据库中的表大小约为 470 MB。

【参考方案1】：

因为当您达到 Excel 的限制时，无论如何您都必须拆分文件，所以这里有一些代码可以从数据库中分块读取到多个 Excel 文件中：

static class Program

    private static string _dataSource;
    private static string _database;
    private static string _table;
    private static string _outputPath;
    private static int _batchSize;

    public static void Main()
    
        try
        
           _dataSource = ConfigurationManager.AppSettings["DataSource"];
           _database = ConfigurationManager.AppSettings["Database"];
           _table = ConfigurationManager.AppSettings["Table"];
           _outputPath = ConfigurationManager.AppSettings["OutputPath"];
           _batchSize = int.Parse(ConfigurationManager.AppSettings["BatchSize"]);

            CreateExcel(_dataSource, _database, _table, _outputPath, "SELECT * FROM " + _table);
        
        catch (Exception e)
        
            Console.WriteLine(e);
        

        Console.WriteLine("All done!");
    

    public static void CreateExcel(string dataSource, string databaseName, string tableName, string outputFilePath, string queryNoParameters)
    
        var sqlConnectionString = new SqlConnectionStringBuilder
            
                DataSource = dataSource,
                InitialCatalog = databaseName,
                IntegratedSecurity = true
            ;

        using (var connection = new SqlConnection(sqlConnectionString.ConnectionString))
        
            connection.Open();
            using (var command = new SqlCommand  Connection = connection, CommandType = CommandType.Text, CommandText = queryNoParameters )
            using (var sqlReader = command.ExecuteReader())
            
                int i = 0;
                while (WriteExcelFile(tableName, GetFileInfo(databaseName, tableName, outputFilePath, i++),
                                      sqlReader, sqlReader.FieldCount, _batchSize))
                
                    Console.WriteLine("Reading next batch...");
                
                            
        
    

    private static bool WriteExcelFile(string tableName, FileInfo fileInfo, IDataReader sqlReader, int numOfColumns, int count)
    
        using (var excelPackage = new ExcelPackage(fileInfo))
        
            ExcelWorksheet worksheet = excelPackage.Workbook.Worksheets.Add(tableName);

            Console.WriteLine("Populating header row...");
            for (var currentColumn = 1; currentColumn <= numOfColumns; currentColumn++)
            
                worksheet.Cells[1, currentColumn].Value = sqlReader.GetName(currentColumn - 1);
                worksheet.Column(currentColumn).Style.Numberformat.Format =
                    TranslateSystemtypeToExceltype(sqlReader.GetFieldType(currentColumn - 1));
            

            Console.WriteLine("Reading data rows...");
            int rowNumber = 2;
            while (rowNumber <= count + 1 && sqlReader.Read())
            
                for (var currentColumn = 1; currentColumn <= numOfColumns; currentColumn++)
                    worksheet.Cells[rowNumber, currentColumn].Value = sqlReader[currentColumn - 1];
                rowNumber++;
            

            if (rowNumber == 2) //nothing read
            
                Console.WriteLine("Nothing to read, reached end of table!");
                return false;
            

            Console.WriteLine("Saving Excel file...");
            excelPackage.Save();
            return rowNumber == count + 2; //in which case we want to read more
        
    

    private static FileInfo GetFileInfo(string databaseName, string tableName, string outputFilePath, int i)
    
        return new FileInfo(Path.Combine(outputFilePath,
                                      Path.ChangeExtension(
                                          string.Format("0_1_2", databaseName, tableName.Replace('.', '-'), i), "xlsx")));
    

    public static string TranslateSystemtypeToExceltype(Type sysType)
    
        if (sysType == typeof(string))
            return "@";
        if (sysType == typeof(DateTime))
                return "dd/MM/YYYY";
        if (sysType == typeof(Decimal))
                return "0.000";
        if (sysType == typeof(bool))
                return "@";
        if (sysType == typeof(int))
                return "0";
        if (sysType == typeof(short))
                return "0";
        if (sysType == typeof(double))
                return "0.000";
        return "General";
    

【讨论】：

我最终成功地使用了这个方法。谢谢！