使用 HttpClient 进行异步文件下载时的错误检查
Posted
技术标签:
【中文标题】使用 HttpClient 进行异步文件下载时的错误检查【英文标题】:Error checking when using HttpClient for asynchronous file downloads 【发布时间】:2015-01-21 19:25:55 【问题描述】:这个问题是Threading issues when using HttpClient for asynchronous file downloads的后续问题。
要使用 HttpClient 异步完成文件传输,您需要将 HttpCompletionOption.ResponseHeadersRead 添加到 SendAsync 请求中。因此,当该调用完成时,您将能够通过添加对 EnsureSuccessStatusCode 的调用来确定请求和响应标头一切正常。然而,此时数据可能仍在传输中。
如何检测在返回标头之后但在数据传输完成之前发生的错误?上述错误会如何表现出来?
下面是一些示例代码,问题的要点标记在第 109 行)并带有注释:“// ***** WANT TO DO MORE ERROR CHECKING HERE**”
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace TestHttpClient2
class Program
/* Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */
static string baseUrl = "http://real-chart.finance.yahoo.com/";
static string requestUrlFormat = "/table.csv?s=0&d=0&e=1&f=2016&g=d&a=0&b=1&c=1901&ignore=.csv";
static void Main(string[] args)
var activeTaskList = new List<Task>();
string outputDirectory = "StockQuotes";
if (!Directory.Exists(outputDirectory))
Directory.CreateDirectory(outputDirectory);
while (true)
Console.WriteLine("Enter symbol or [ENTER] to exit:");
string symbol = Console.ReadLine();
if (string.IsNullOrEmpty(symbol))
break;
Task downloadTask = DownloadDataForStockAsync(outputDirectory, symbol);
if (TaskIsActive(downloadTask))
// This is an asynchronous world - lock the list before updating it!
lock (activeTaskList)
activeTaskList.Add(downloadTask);
else
Console.WriteLine("task completed already?!??!?");
CleanupTasks(activeTaskList);
Console.WriteLine("Cleaning up");
while (CleanupTasks(activeTaskList))
Task.Delay(1).Wait();
private static bool CleanupTasks(List<Task> activeTaskList)
// reverse loop to allow list item deletions
// This is an asynchronous world - lock the list before updating it!
lock (activeTaskList)
for (int i = activeTaskList.Count - 1; i >= 0; i--)
if (!TaskIsActive(activeTaskList[i]))
activeTaskList.RemoveAt(i);
return activeTaskList.Count > 0;
private static bool TaskIsActive(Task task)
return task != null
&& task.Status != TaskStatus.Canceled
&& task.Status != TaskStatus.Faulted
&& task.Status != TaskStatus.RanToCompletion;
static async Task DownloadDataForStockAsync(string outputDirectory, string symbol)
try
using (var client = new HttpClient())
client.BaseAddress = new Uri(baseUrl);
client.Timeout = TimeSpan.FromMinutes(5);
string requestUrl = string.Format(requestUrlFormat, symbol);
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
await httpStream.CopyToAsync(fileStream);
// *****WANT TO DO MORE ERROR CHECKING HERE*****
catch (HttpRequestException ex)
Console.WriteLine("Exception on thread: 0: 1\r\n",
System.Threading.Thread.CurrentThread.ManagedThreadId,
ex.Message,
ex.StackTrace);
catch (Exception ex)
Console.WriteLine("Exception on thread: 0: 1\r\n",
System.Threading.Thread.CurrentThread.ManagedThreadId,
ex.Message,
ex.StackTrace);
static volatile string lastTimestampedString = string.Empty;
static volatile string dummy = string.Empty;
static string FormatTimestampedString(string message, bool uniquify = false)
// This is an asynchronous world - lock the shared resource before using it!
lock (dummy)
//lock (lastTimestampedString)
Console.WriteLine("IN - Thread: 0:D2 lastTimestampedString: 1",
System.Threading.Thread.CurrentThread.ManagedThreadId,
lastTimestampedString);
string newTimestampedString;
while (true)
DateTime lastDateTime = DateTime.Now;
newTimestampedString = string.Format(
"1:D4_2:D2_3:D2_4:D2_5:D2_6:D2_7:D3_0",
message,
lastDateTime.Year, lastDateTime.Month, lastDateTime.Day,
lastDateTime.Hour, lastDateTime.Minute, lastDateTime.Second,
lastDateTime.Millisecond
);
if (!uniquify)
break;
if (newTimestampedString != lastTimestampedString)
break;
//Task.Delay(1).Wait();
;
lastTimestampedString = newTimestampedString;
Console.WriteLine("OUT - Thread: 0:D2 lastTimestampedString: 1",
System.Threading.Thread.CurrentThread.ManagedThreadId,
lastTimestampedString);
return lastTimestampedString;
【问题讨论】:
with the point of the question marked at line 109):
如果您没有显示/显示行号,您是否希望个人手动计算行数,则无济于事.. 哈哈
有没有办法在不搞乱代码的情况下显示行号?如果是这样,我将进行编辑。实际上,我的屏幕上有大约 30 行,如果我在滚动条上单击 3 次,会弹出:// *****想在此处检查更多错误 *****跨度>
【参考方案1】:
我已经复制并稍微清理了相关代码。
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var response = await client.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
using (var httpStream = await response.Content.ReadAsStreamAsync())
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
await httpStream.CopyToAsync(fileStream);
问题是,如果在读取流并将其复制到文件中时出现问题怎么办?
所有逻辑错误已作为 HTTP 请求和响应周期的一部分得到解决:服务器已收到您的请求,它已确定它是有效的,它已成功响应(标题部分响应),它现在正在向您发送结果(响应的正文部分)。
现在唯一可能发生的错误是服务器崩溃、连接丢失等。我的理解是这些将表现为HttpRequestException
,这意味着您可以编写如下代码:
try
using (var httpStream = await response.Content.ReadAsStreamAsync())
var timestampedName = FormatTimestampedString(symbol, true);
var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
using (var fileStream = File.Create(filePath))
await httpStream.CopyToAsync(fileStream);
catch (HttpRequestException e)
...
The documenation doesn't say much, unfortunately. The reference source doesn't either. 所以最好的办法是从这里开始,并可能记录所有不是HttpRequestException
的异常,以防在下载响应正文期间可能抛出另一种异常类型。
【讨论】:
【参考方案2】:如果你想把它缩小到标题读取和内容读取之间的部分,你实际上把异步缓冲区读取留给自己:
var httpStream = await response.Content.ReadAsStreamAsync();
如果你查看方法内部发生了什么,你会看到:
public Task<Stream> ReadAsStreamAsync()
this.CheckDisposed();
TaskCompletionSource<Stream> tcs = new TaskCompletionSource<Stream>();
if (this.contentReadStream == null && this.IsBuffered)
this.contentReadStream = new MemoryStream(this.bufferedContent.GetBuffer(),
0, (int)this.bufferedContent.Length,
false, false);
if (this.contentReadStream != null)
tcs.TrySetResult(this.contentReadStream);
return tcs.Task;
this.CreateContentReadStreamAsync().ContinueWithStandard(delegate(Task<Stream> task)
if (!HttpUtilities.HandleFaultsAndCancelation<Stream>(task, tcs))
this.contentReadStream = task.Result;
tcs.TrySetResult(this.contentReadStream);
);
return tcs.Task;
CreateContentReadStreamAsync
是负责所有阅读的人,在内部,它将调用LoadIntoBufferAsync
,您可以使用find here。
基本上,您可以看到它封装了IOException
和ObjectDisposedException
,或者ArgumentOutOfRangeException
是大于2GB 的缓冲区(尽管我认为这非常罕见)。
【讨论】:
很明显,我在分发好东西的那天生病了 ;-) 你如何看待这个来源? @anyoneis 哈哈。你可以看看here,或者启动ILSpy 好吧,我可以看到一些使用此信息的来源:referencesource.microsoft.com/setup.html以上是关于使用 HttpClient 进行异步文件下载时的错误检查的主要内容,如果未能解决你的问题,请参考以下文章
异步httpclient(httpasyncclient)的使用与总结
C#:HttpClient,将多个文件作为MultipartFormDataContent上传时的文件上传进度