如何跟踪异步文件上传到 Azure 存储的进度

Posted 2023-03-28

【中文标题】如何跟踪异步文件上传到 Azure 存储的进度

【英文标题】：How to track progress of async file upload to azure storage

【发布时间】：2014-01-16 23:48:27

【问题描述】：

有没有办法跟踪文件上传到 Azure 存储容器的进度？我正在尝试制作一个控制台应用程序，用于使用 C# 将数据上传到 Azure。

我当前的代码如下所示：

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
using System.Configuration;
using System.IO;
using System.Threading;

namespace AdoAzure

    class Program
    
        static void Main(string[] args)
        
            CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
            ConfigurationManager.ConnectionStrings["StorageConnectionString"].ConnectionString);
            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer container = blobClient.GetContainerReference("adokontajnerneki");
            container.CreateIfNotExists();
            CloudBlobClient myBlobClient = storageAccount.CreateCloudBlobClient();
            CloudBlockBlob myBlob = container.GetBlockBlobReference("racuni.adt");
            CancellationToken ca = new CancellationToken();
            var ado = myBlob.UploadFromFileAsync(@"c:\bo\racuni.adt", FileMode.Open, ca);
            Console.WriteLine(ado.Status); //Does Not Help Much
            ado.ContinueWith(t =>
            
                Console.WriteLine("It is over"); //this is working OK
            );
            Console.WriteLine(ado.Status); //Does Not Help Much
            Console.WriteLine("theEnd");
            Console.ReadKey();
        
    

这段代码运行良好，但我希望有某种进度条，以便用户可以看到正在运行的任务。 WindowsAzure.Storage.Blob 命名空间中是否有内置的东西可供我使用，例如戴帽子的兔子？

【参考方案1】：

我不认为这是可能的，因为上传文件是一个单一的任务，即使在内部文件被分成多个块并且这些块被上传，代码实际上等待整个任务完成。

一种可能性是手动将文件拆分成块并使用PutBlockAsync 方法异步上传这些块。上传所有块后，您可以调用 PutBlockListAsync 方法来提交 blob。请参阅下面的代码：

using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApplication1

    class Program
    
        static CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials("accountname", "accountkey"), true);
        static void Main(string[] args)
        
            CloudBlobClient myBlobClient = storageAccount.CreateCloudBlobClient();
            myBlobClient.SingleBlobUploadThresholdInBytes = 1024 * 1024;
            CloudBlobContainer container = myBlobClient.GetContainerReference("adokontajnerneki");
            //container.CreateIfNotExists();
            CloudBlockBlob myBlob = container.GetBlockBlobReference("cfx.zip");
            var blockSize = 256 * 1024;
            myBlob.StreamWriteSizeInBytes = blockSize;
            var fileName = @"D:\cfx.zip";
            long bytesToUpload = (new FileInfo(fileName)).Length;
            long fileSize = bytesToUpload;

            if (bytesToUpload < blockSize)
            
                CancellationToken ca = new CancellationToken();
                var ado = myBlob.UploadFromFileAsync(fileName, FileMode.Open, ca);
                Console.WriteLine(ado.Status); //Does Not Help Much
                ado.ContinueWith(t =>
                
                    Console.WriteLine("Status = " + t.Status);
                    Console.WriteLine("It is over"); //this is working OK
                );
            
            else
            
                List<string> blockIds = new List<string>();
                int index = 1;
                long startPosition = 0;
                long bytesUploaded = 0;
                do
                
                    var bytesToRead = Math.Min(blockSize, bytesToUpload);
                    var blobContents = new byte[bytesToRead];
                    using (FileStream fs = new FileStream(fileName, FileMode.Open))
                    
                        fs.Position = startPosition;
                        fs.Read(blobContents, 0, (int)bytesToRead);
                    
                    ManualResetEvent mre = new ManualResetEvent(false);
                    var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(index.ToString("d6")));
                    Console.WriteLine("Now uploading block # " + index.ToString("d6"));
                    blockIds.Add(blockId);
                    var ado = myBlob.PutBlockAsync(blockId, new MemoryStream(blobContents), null);
                    ado.ContinueWith(t =>
                    
                        bytesUploaded += bytesToRead;
                        bytesToUpload -= bytesToRead;
                        startPosition += bytesToRead;
                        index++;
                        double percentComplete = (double)bytesUploaded / (double)fileSize;
                        Console.WriteLine("Percent complete = " + percentComplete.ToString("P"));
                        mre.Set();
                    );
                    mre.WaitOne();
                
                while (bytesToUpload > 0);
                Console.WriteLine("Now committing block list");
                var pbl = myBlob.PutBlockListAsync(blockIds);
                pbl.ContinueWith(t =>
                
                    Console.WriteLine("Blob uploaded completely.");
                );
            
            Console.ReadKey();
        
    

【讨论】：

向导！！！太感谢了。经过数周的研究，这极大地帮助解决了我的问题。

请注意，您不应该在循环内多次打开文件流 - 文件流应该在循环外打开一次。

【参考方案2】：

Gaurav 的解决方案效果很好，与http://blogs.msdn.com/b/kwill/archive/2011/05/30/asynchronous-parallel-block-blob-transfers-with-progress-change-notification.aspx 非常相似。这段代码面临的挑战是您正在做很多复杂的工作，而错误处理很少。我并不是说 Gaurav 的代码有什么问题——它看起来很可靠——但尤其是与网络相关的通信代码，有很多变量和很多问题需要你考虑。

出于这个原因，我修改了我的原始博客以使用存储客户端库中的上传代码（假设来自 Azure 存储团队的代码比我可以编写的任何代码都更强大）并使用 ProgressStream 类跟踪进度.您可以在http://blogs.msdn.com/b/kwill/archive/2013/03/06/asynchronous-parallel-block-blob-transfers-with-progress-change-notification-2-0.aspx 看到更新后的代码。

【讨论】：

+1 @kwill。我同意我们必须处理上传过程失败的情况，我在上面的代码中没有考虑到这种情况。此外，代码一次上传一个块，但在生产代码中，可能需要并行上传多个块。

您应该在此处放置该页面的示例，而不仅仅是链接。 @GauravMantri 的回答简短而好。那个链接太可怕了。

请注意，CloudBlobClient.SingleBlobUploadThresholdInBytes 属性现在已过时。使用 DefaultRequestOptions.SingleBlobUploadThresholdInBytes。

【参考方案3】：

这个怎么样。

public class ObservableFileStream : FileStream

    private Action<long> _callback;

    public ObservableFileStream(String fileName, FileMode mode, Action<long> callback) : base(fileName, mode)
    
        _callback = callback;
    

    public override void Write(byte[] array, int offset, int count)
    
        _callback?.Invoke(Length);
        base.Write(array, offset, count);
    

    public override int Read(byte[] array, int offset, int count)
    
        _callback?.Invoke(Position);
        return base.Read(array, offset, count);
    

public class Test

    private async void Upload(String filePath, CloudBlockBlob blob)
    
        ObservableFileStream fs = null;

        using (fs = new ObservableFileStream(filePath, FileMode.Open, (current) =>
        
            Console.WriteLine("Uploading " + ((double)current / (double)fs.Length) * 100d);
        ))
        
            await blob.UploadFromStreamAsync(fs);
        
    

【参考方案4】：

以下代码使用 Azure Blob Storage SDK v12。 参考链接：https://www.craftedforeveryone.com/upload-or-download-file-from-azure-blob-storage-with-progress-percentage-csharp/

public void UploadBlob(string fileToUploadPath)

    var file = new FileInfo(fileToUploadPath);
    uploadFileSize = file.Length; //Get the file size. This is need to calculate the file upload progress

    //Initialize a progress handler. When the file is being uploaded, the current uploaded bytes will be published back to us using this progress handler by the Blob Storage Service
    var progressHandler = new Progress();
    progressHandler.ProgressChanged += UploadProgressChanged;

    var blob = new BlobClient(connectionString, containerName, file.Name); //Initialize the blob client
    blob.Upload(fileToUploadPath, progressHandler: progressHandler); //Make sure to pass the progress handler here



private void UploadProgressChanged(object sender, long bytesUploaded)

    //Calculate the progress and update the progress bar.
    //Note: the bytes uploaded published back to us is in long. In order to calculate the percentage, the value has to be converted to double. 
    //Auto type casting from long to double happens here as part of function call
    Console.WriteLine(GetProgressPercentage(uploadFileSize, bytesUploaded));


private double GetProgressPercentage(double totalSize,double currentSize)

    return (currentSize / totalSize) * 100;

【参考方案5】：

使用BlockBlobClient 代替BlobClient (using Azure.Storage.Blobs.Specialized;)

private const string StorageConnectionString = "<Your storage connection string>";
private const string StorageContainerName = "<Your storage container name>";

[HttpPost]
public async Task<IActionResult> UploadFiles(IList<IFormFile> files)

    foreach (var file in files)
    
        var blockBlobClient = new BlockBlobClient(StorageConnectionString, StorageContainerName, file.FileName); // Add Guid to FileName to ensure uniqueness

        using var output = blockBlobClient.OpenWrite(overwrite: true);
        using var input = file.OpenReadStream();

        var buffer = new byte[64 * 1024];
        var totalReadBytes = 0L;
        var readBytes = 0;

        while ((readBytes = input.Read(buffer, 0, buffer.Length)) > 0)
        
            await output.WriteAsync(buffer, 0, readBytes);

            totalReadBytes += readBytes;

            //progress = Math.Round((totalReadBytes * 1d / sessionUploadFileInfo.FileLength * 100d), 2, MidpointRounding.AwayFromZero);
            //do what you do with the progress, save it in session or however you display it to the user...
            //With WebAPI, I use a static dictionary and a client Id to store file upload progress
            //A separate REST request with client Id gets the file progress' from the dictionary
        
    
    return Content("success");