无法通过 asp.net 核心服务器端将 ~200mb 文件上传到 azure blob 存储

Posted 2023-03-28

【中文标题】无法通过 asp.net 核心服务器端将 ~200mb 文件上传到 azure blob 存储

【英文标题】：Cant upload file ~200mb to azure blob storage by asp.net core server side

【发布时间】：2018-10-07 06:35:27

【问题描述】：

我在 Asp.net 核心上有服务器部分，它接收 Content-Type: multipart/form-data 格式标头的文件并将其发送到流中的 azure blob 存储。但是当我发送大约 200 MB 或更多的文件时出现错误

“请求正文过大，超出最大允许限制”

在我搜索时，它可能发生在旧版本的 WindowsAzure.Storage 中，但我使用的是 9.1.1 版本。当我更深入地查看方法 UploadFromStreamAsync chank blob 时，默认为 4 MB。所以我不知道该怎么做才请求你的帮助。 我的控制器：

public async Task<IActionResult> Post(string folder)
    
        string azureBlobConnectionString = _configuration.GetConnectionString("BlobConnection");
        // Retrieve storage account from connection string.
        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(azureBlobConnectionString);

        HttpResponseUploadClass responseUploadClass = await Request.StreamFile(folder, storageAccount);
        FormValueProvider formModel = responseUploadClass.FormValueProvider;

        var viewModel = new MyViewModel();

        var bindingSuccessful = await TryUpdateModelAsync(viewModel, prefix: "",
            valueProvider: formModel);

        if (!bindingSuccessful)
        
            if (!ModelState.IsValid)
            
                return BadRequest(ModelState);
            
        

        return Ok(responseUploadClass.Url);
    

以及我将流文件流发送到 azure blob 的类

 public static async Task<HttpResponseUploadClass> StreamFile(this HttpRequest request, string folder, CloudStorageAccount blobAccount)
    
        CloudBlobClient blobClient = blobAccount.CreateCloudBlobClient();
        CloudBlobContainer container = blobClient.GetContainerReference(folder);
        CloudBlockBlob blockBlob = null;

        if (!MultipartRequestHelper.IsMultipartContentType(request.ContentType))
        
            throw new Exception($"Expected a multipart request, but got request.ContentType");
        
        var formAccumulator = new KeyValueAccumulator();

        var boundary = MultipartRequestHelper.GetBoundary(
            MediaTypeHeaderValue.Parse(request.ContentType),
            DefaultFormOptions.MultipartBoundaryLengthLimit);
        var reader = new MultipartReader(boundary, request.Body);

        var section = await reader.ReadNextSectionAsync();
        while (section != null)
        
            ContentDispositionHeaderValue contentDisposition;
            var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out contentDisposition);
            var disposition = ContentDispositionHeaderValue.Parse(section.ContentDisposition);
            if (hasContentDispositionHeader)
            
                if (MultipartRequestHelper.HasFileContentDisposition(contentDisposition))
                
                    try
                    
                        string fileName = HttpUtility.UrlEncode(disposition.FileName.Value.Replace("\"", ""), Encoding.UTF8);
                        blockBlob = container.GetBlockBlobReference(Guid.NewGuid().ToString());
                        blockBlob.Properties.ContentType = GetMimeTypeByWindowsRegistry(fileName);
                        blockBlob.Properties.ContentDisposition = "attachment; filename*=UTF-8''" + fileName;
                        await blockBlob.UploadFromStreamAsync(section.Body);
                    
                    catch (Exception e)
                    
                        Console.WriteLine(e);
                        throw;
                    
                
                else if (MultipartRequestHelper.HasFormDataContentDisposition(contentDisposition))
                
                    var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name);
                    var encoding = GetEncoding(section);
                    using (var streamReader = new StreamReader(
                        section.Body,
                        encoding,
                        detectEncodingFromByteOrderMarks: true,
                        bufferSize: 1024,
                        leaveOpen: true))
                    
                        var value = await streamReader.ReadToEndAsync();
                        if (String.Equals(value, "undefined", StringComparison.OrdinalIgnoreCase))
                        
                            value = String.Empty;
                        
                        formAccumulator.Append(key.Value, value);

                        if (formAccumulator.ValueCount > DefaultFormOptions.ValueCountLimit)
                        
                            throw new InvalidDataException($"Form key count limit DefaultFormOptions.ValueCountLimit exceeded.");
                        
                    
                
            
            section = await reader.ReadNextSectionAsync();
        
        var formValueProvider = new FormValueProvider(
            BindingSource.Form,
            new FormCollection(formAccumulator.GetResults()),
            CultureInfo.CurrentCulture);

        return new HttpResponseUploadClassFormValueProvider = formValueProvider, Url = blockBlob?.Uri.ToString();
    

【参考方案1】：

正如您所说，每个 block blob 的大小可以不同，最大为 100 MB（使用 2016 年 5 月 31 日之前的 REST 版本的请求为 4 MB），一个块 blob 最多可以包含 50,000 个块。

如果您编写的块 Blob 大小不超过 256 MB（对于使用 2016-05-31 之前的 REST 版本的请求，则为 64 MB），您可以将其上传到 一次写入操作即可完成全部操作，请参阅Put Blob。

存储客户端默认为 32 MB 最大单块上传，可使用 SingleBlobUploadThresholdInBytes 属性设置。 当块 blob 上传大于此属性中的值时，存储客户端会将文件分成块。 您可以使用ParallelOperationThreadCount 属性设置用于并行上传块的线程数。

BlobRequestOptions requestoptions = new BlobRequestOptions()

    SingleBlobUploadThresholdInBytes = 1024 * 1024 * 50, //50MB
    ParallelOperationThreadCount = 12,
;

CloudStorageAccount account = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobclient = account.CreateCloudBlobClient();
blobclient.DefaultRequestOptions = requestoptions;
CloudBlobContainer blobcontainer = blobclient.GetContainerReference("uploadfiles");
blobcontainer.CreateIfNotExists();
CloudBlockBlob blockblob = blobcontainer.GetBlockBlobReference("bigfiles");

更多详情可以参考这个thread。