使用 HttpClient 进行异步文件下载时的错误检查

Posted

技术标签:

【中文标题】使用 HttpClient 进行异步文件下载时的错误检查【英文标题】:Error checking when using HttpClient for asynchronous file downloads 【发布时间】:2015-01-21 19:25:55 【问题描述】:

这个问题是Threading issues when using HttpClient for asynchronous file downloads的后续问题。

要使用 HttpClient 异步完成文件传输,您需要将 HttpCompletionOption.ResponseHeadersRead 添加到 SendAsync 请求中。因此,当该调用完成时,您将能够通过添加对 EnsureSuccessStatusCode 的调用来确定请求和响应标头一切正常。然而,此时数据可能仍在传输中。

如何检测在返回标头之后但在数据传输完成之前发生的错误?上述错误会如何表现出来?

下面是一些示例代码,问题的要点标记在第 109 行)并带有注释:“// ***** WANT TO DO MORE ERROR CHECKING HERE**”

using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;

namespace TestHttpClient2

  class Program
  
    /* Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */

    static string baseUrl = "http://real-chart.finance.yahoo.com/";
    static string requestUrlFormat = "/table.csv?s=0&d=0&e=1&f=2016&g=d&a=0&b=1&c=1901&ignore=.csv";

    static void Main(string[] args)
    
      var activeTaskList = new List<Task>();

      string outputDirectory = "StockQuotes";
      if (!Directory.Exists(outputDirectory))
      
        Directory.CreateDirectory(outputDirectory);
      

      while (true)
      
        Console.WriteLine("Enter symbol or [ENTER] to exit:");
        string symbol = Console.ReadLine();
        if (string.IsNullOrEmpty(symbol))
        
          break;
        

        Task downloadTask = DownloadDataForStockAsync(outputDirectory, symbol);
        if (TaskIsActive(downloadTask))
        
          // This is an asynchronous world - lock the list before updating it!
          lock (activeTaskList)
          
            activeTaskList.Add(downloadTask);
          

        
        else
        
          Console.WriteLine("task completed already?!??!?");
        
        CleanupTasks(activeTaskList);
      

      Console.WriteLine("Cleaning up");
      while (CleanupTasks(activeTaskList))
      
        Task.Delay(1).Wait();
      
    

    private static bool CleanupTasks(List<Task> activeTaskList)
    
      // reverse loop to allow list item deletions
      // This is an asynchronous world - lock the list before updating it!
      lock (activeTaskList)
      
        for (int i = activeTaskList.Count - 1; i >= 0; i--)
        
          if (!TaskIsActive(activeTaskList[i]))
          
            activeTaskList.RemoveAt(i);
          
        
        return activeTaskList.Count > 0;
      
    

    private static bool TaskIsActive(Task task)
    
      return task != null
          && task.Status != TaskStatus.Canceled
          && task.Status != TaskStatus.Faulted
          && task.Status != TaskStatus.RanToCompletion;
    

    static async Task DownloadDataForStockAsync(string outputDirectory, string symbol)
    
      try
      
        using (var client = new HttpClient())
        
          client.BaseAddress = new Uri(baseUrl);
          client.Timeout = TimeSpan.FromMinutes(5);
          string requestUrl = string.Format(requestUrlFormat, symbol);

          var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
          var response = await client.SendAsync(request, 
            HttpCompletionOption.ResponseHeadersRead);
          response.EnsureSuccessStatusCode();

          using (var httpStream = await response.Content.ReadAsStreamAsync())
          
            var timestampedName = FormatTimestampedString(symbol, true);
            var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
            using (var fileStream = File.Create(filePath))
            
              await httpStream.CopyToAsync(fileStream);
            
          
          // *****WANT TO DO MORE ERROR CHECKING HERE*****
        
      
      catch (HttpRequestException ex)
      
        Console.WriteLine("Exception on thread: 0: 1\r\n",
          System.Threading.Thread.CurrentThread.ManagedThreadId,
          ex.Message,
          ex.StackTrace);
      
      catch (Exception ex)
      
        Console.WriteLine("Exception on thread: 0: 1\r\n",
          System.Threading.Thread.CurrentThread.ManagedThreadId,
          ex.Message,
          ex.StackTrace);
      
    

    static volatile string lastTimestampedString = string.Empty;
    static volatile string dummy = string.Empty;

    static string FormatTimestampedString(string message, bool uniquify = false)
    
      // This is an asynchronous world - lock the shared resource before using it!
      lock (dummy)
      //lock (lastTimestampedString)
      
        Console.WriteLine("IN  - Thread: 0:D2 lastTimestampedString: 1",
            System.Threading.Thread.CurrentThread.ManagedThreadId,
            lastTimestampedString);

        string newTimestampedString;

        while (true)
        
          DateTime lastDateTime = DateTime.Now;

          newTimestampedString = string.Format(
              "1:D4_2:D2_3:D2_4:D2_5:D2_6:D2_7:D3_0",
                message,
                lastDateTime.Year, lastDateTime.Month, lastDateTime.Day,
                lastDateTime.Hour, lastDateTime.Minute, lastDateTime.Second,
                lastDateTime.Millisecond
                );
          if (!uniquify)
          
            break;
          
          if (newTimestampedString != lastTimestampedString)
          
            break;
          

          //Task.Delay(1).Wait();
        ;

        lastTimestampedString = newTimestampedString;
        Console.WriteLine("OUT - Thread: 0:D2 lastTimestampedString: 1",
            System.Threading.Thread.CurrentThread.ManagedThreadId,
            lastTimestampedString);

        return lastTimestampedString;
      
    
  

【问题讨论】:

with the point of the question marked at line 109): 如果您没有显示/显示行号,您是否希望个人手动计算行数,则无济于事.. 哈哈 有没有办法在不搞乱代码的情况下显示行号?如果是这样,我将进行编辑。实际上,我的屏幕上有大约 30 行,如果我在滚动条上单击 3 次,会弹出:// *****想在此处检查更多错误 *****跨度> 【参考方案1】:

我已经复制并稍微清理了相关代码。

var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
    HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())

    var timestampedName = FormatTimestampedString(symbol, true);
    var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
    using (var fileStream = File.Create(filePath))
    
        await httpStream.CopyToAsync(fileStream);
    

问题是,如果在读取流并将其复制到文件中时出现问题怎么办?

所有逻辑错误已作为 HTTP 请求和响应周期的一部分得到解决:服务器已收到您的请求,它已确定它是有效的,它已成功响应(标题部分响应),它现在正在向您发送结果(响应的正文部分)。

现在唯一可能发生的错误是服务器崩溃、连接丢失等。我的理解是这些将表现为HttpRequestException,这意味着您可以编写如下代码:

try

    using (var httpStream = await response.Content.ReadAsStreamAsync())
    
        var timestampedName = FormatTimestampedString(symbol, true);
        var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
        using (var fileStream = File.Create(filePath))
        
            await httpStream.CopyToAsync(fileStream);
        
    

catch (HttpRequestException e)

    ...

The documenation doesn't say much, unfortunately. The reference source doesn't either. 所以最好的办法是从这里开始,并可能记录所有不是HttpRequestException 的异常,以防在下载响应正文期间可能抛出另一种异常类型。

【讨论】:

【参考方案2】:

如果你想把它缩小到标题读取和内容读取之间的部分,你实际上把异步缓冲区读取留给自己:

var httpStream = await response.Content.ReadAsStreamAsync();

如果你查看方法内部发生了什么,你会看到:

public Task<Stream> ReadAsStreamAsync()

    this.CheckDisposed();
    TaskCompletionSource<Stream> tcs = new TaskCompletionSource<Stream>();
    if (this.contentReadStream == null && this.IsBuffered)
    
        this.contentReadStream = new MemoryStream(this.bufferedContent.GetBuffer(),
                                                  0, (int)this.bufferedContent.Length,
                                                  false, false);
    
    if (this.contentReadStream != null)
    
        tcs.TrySetResult(this.contentReadStream);
        return tcs.Task;
    
    this.CreateContentReadStreamAsync().ContinueWithStandard(delegate(Task<Stream> task)
    
        if (!HttpUtilities.HandleFaultsAndCancelation<Stream>(task, tcs))
        
            this.contentReadStream = task.Result;
            tcs.TrySetResult(this.contentReadStream);
        
    );
    return tcs.Task;

CreateContentReadStreamAsync 是负责所有阅读的人,在内部,它将调用LoadIntoBufferAsync,您可以使用find here。

基本上,您可以看到它封装了IOExceptionObjectDisposedException,或者ArgumentOutOfRangeException 是大于2GB 的缓冲区(尽管我认为这非常罕见)。

【讨论】:

很明显,我在分发好东西的那天生病了 ;-) 你如何看待这个来源? @anyoneis 哈哈。你可以看看here,或者启动ILSpy 好吧,我可以看到一些使用此信息的来源:referencesource.microsoft.com/setup.html

以上是关于使用 HttpClient 进行异步文件下载时的错误检查的主要内容,如果未能解决你的问题,请参考以下文章

使用 HttpClient 进行异步文件下载

在 PCL 中使用 HttpClient 进行异步调用

异步httpclient(httpasyncclient)的使用与总结

C#:HttpClient,将多个文件作为MultipartFormDataContent上传时的文件上传进度

在 HttpClient 中使用 await 的异步调用永远不会返回

使用异步httpclient框架做get,post提交数据