如何将大文件上传到 Azure Blob 存储 (.NET Core)

Posted 2023-03-28

【中文标题】如何将大文件上传到 Azure Blob 存储 (.NET Core)

【英文标题】：How to upload big files to Azure Blob Storage (.NET Core)

【发布时间】：2021-12-18 13:56:00

【问题描述】：

我尝试将 2GB 文件上传到 Azure 存储，但我的代码失败并且出现异常。对于较小的文件，它工作正常。如何更改超时和文件大小限制？我在 .NET Core 上使用 Azure.Storage.Blobs 版本 12.10.0。

我的代码：

    public FileRepository(AppSettings appSettings)
    
        var blobServiceClient = new BlobServiceClient(appSettings.StorageConnectionString);

        var containers = blobServiceClient.GetBlobContainers();
        if (!containers.Any(x => x.Name == containerName))
        
            blobServiceClient.CreateBlobContainer(containerName);
        
        _blobContainerClient = blobServiceClient.GetBlobContainerClient(containerName);
    

    public async Task<string> Upload(string fileName, Stream content)
    
        var blobName = $"Guid.NewGuid()/fileName";
        await _blobContainerClient.UploadBlobAsync(blobName, content);
        return blobName;
    

我有例外：

System.AggregateException
  HResult=0x80131500
  Message=Retry failed after 6 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.)
  Source=Azure.Core
  StackTrace:
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.BlobRestClient.BlockBlob.<UploadAsync>d__0.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<UploadInternal>d__26.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<>c__DisplayClass48_0.<<GetPartitionedUploaderBehaviors>b__0>d.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.PartitionedUploader`2.<UploadInternal>d__19.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.BlobClient.<StagedUploadInternal>d__29.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.BlobContainerClient.<UploadBlobAsync>d__83.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at C:\...\FileRepository.<Upload>d__3.MoveNext() in C:\...\FileRepository.cs:line 38

Inner Exception 1:
TaskCanceledException: The operation was canceled.

Inner Exception 2:
HttpRequestException: Error while copying content to a stream.

Inner Exception 3:
IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..

Inner Exception 4:
SocketException: The I/O operation has been aborted because of either a thread exit or an application request.

【参考方案1】：

我认为使用BlobContainerClient上传时无法控制超时选项和文件大小限制。

您需要在BlobClient 类中使用UploadAsync 方法并在BlobUploadOptions.TransferOptions 中指定适当的值。

如果您想对上传过程进行更精细的控制，则需要在特定的 blob 客户端中使用上传方法。例如，如果您要上传块 blob，则可以使用 BlockBlobClient.StageBlockAsync 和 BlockBlobClient.CommitBlockListAsync。

【讨论】：

非常感谢，就是这样。 PS 我用过： var clientOptions = new BlobClientOptions(); clientOptions.Retry.MaxRetries = 1; clientOptions.Retry.NetworkTimeout = TimeSpan.FromMinutes(30);

【参考方案2】：

您可以在客户端将文件拆分为chunks/blocks。然后发送大量 100 MB 的小块。

将文件分割成更小的块，如果您使用 Stream 之类的代码，则代码只会读取一定数量的字节并处理这些字节直到完成。

请参考Implement File Chunking,Upload a file in blocks 和Put Blob 了解更多详情。

【讨论】：

问题是关于 Azure.Storage.Blobs v12，而您将链接发送到 [1] v11、[2] 甚至更旧的 skd [3] rest api，它们的工作方式都不同。我的问题是关于 v12