使用 HttpClient 进行异步文件下载时的线程问题

Posted 2023-03-30

【中文标题】使用 HttpClient 进行异步文件下载时的线程问题

【英文标题】：Threading issues when using HttpClient for asynchronous file downloads

【发布时间】：2015-01-13 01:40:19

【问题描述】：

这个问题是Using HttpClient for Asynchronous File downloads 的后续问题。

2015/01/15 已编辑添加适应多线程 - 仍然有一个谜，

using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;

namespace TestHttpClient2

  class Program
  
    /* Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */

    static string baseUrl = "http://real-chart.finance.yahoo.com/";
    static string requestUrlFormat = "/table.csv?s=0&d=0&e=1&f=2016&g=d&a=0&b=1&c=1901&ignore=.csv";

    static void Main(string[] args)
    
      var activeTaskList = new List<Task>();

      string outputDirectory = "StockQuotes";
      if (!Directory.Exists(outputDirectory))
      
        Directory.CreateDirectory(outputDirectory);
      

      while (true)
      
        Console.WriteLine("Enter symbol or [ENTER] to exit:");
        string symbol = Console.ReadLine();
        if (string.IsNullOrEmpty(symbol))
        
          break;
        

        Task downloadTask = DownloadDataForStockAsync(outputDirectory, symbol);
        if (TaskIsActive(downloadTask))
        
          // This is an asynchronous world - lock the list before updating it!
          lock (activeTaskList)
          
            activeTaskList.Add(downloadTask);
          

        
        else
        
          Console.WriteLine("task completed already?!??!?");
        
        CleanupTasks(activeTaskList);
      

      Console.WriteLine("Cleaning up");
      while (CleanupTasks(activeTaskList))
      
        Task.Delay(1).Wait();
      
    

    private static bool CleanupTasks(List<Task> activeTaskList)
    
      // reverse loop to allow list item deletions
      // This is an asynchronous world - lock the list before updating it!
      lock (activeTaskList)
      
        for (int i = activeTaskList.Count - 1; i >= 0; i--)
        
          if (!TaskIsActive(activeTaskList[i]))
          
            activeTaskList.RemoveAt(i);
          
        
        return activeTaskList.Count > 0;
      
    

    private static bool TaskIsActive(Task task)
    
      return task != null
          && task.Status != TaskStatus.Canceled
          && task.Status != TaskStatus.Faulted
          && task.Status != TaskStatus.RanToCompletion;
    

    static async Task DownloadDataForStockAsync(string outputDirectory, string symbol)
    
      try
      
        using (var client = new HttpClient())
        
          client.BaseAddress = new Uri(baseUrl);
          client.Timeout = TimeSpan.FromMinutes(5);
          string requestUrl = string.Format(requestUrlFormat, symbol);

          var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
          var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
          var response = await sendTask;
          response.EnsureSuccessStatusCode();
          var httpStream = await response.Content.ReadAsStreamAsync();

          string timestampedName = FormatTimestampedString(symbol, true);
          var filePath = Path.Combine(outputDirectory, timestampedName + ".csv");
          using (var fileStream = File.Create(filePath))
          using (var reader = new StreamReader(httpStream))
          
            await httpStream.CopyToAsync(fileStream);
            fileStream.Flush();
          
        
      
      catch (Exception ex)
      
        Console.WriteLine("Exception on thread: 0: 1\r\n",
          System.Threading.Thread.CurrentThread.ManagedThreadId,
          ex.Message,
          ex.StackTrace);
      
    

    static volatile string lastTimestampedString = string.Empty;
    static volatile string dummy = string.Empty;
    static HashSet<string> oldStrings = new HashSet<string>();

    static string FormatTimestampedString(string message, bool uniquify = false)
    
      // This is an asynchronous world - lock the shared resource before using it!
      //lock (dummy)
      lock (lastTimestampedString)
      
        Console.WriteLine("IN  - Thread: 0:D2 lastTimestampedString: 1", 
            System.Threading.Thread.CurrentThread.ManagedThreadId,
            lastTimestampedString);

        string newTimestampedString;

        while (true)
        
          DateTime lastDateTime = DateTime.Now;

          newTimestampedString = string.Format(
              "1:D4_2:D2_3:D2_4:D2_5:D2_6:D2_7:D3_0",
                message,
                lastDateTime.Year, lastDateTime.Month, lastDateTime.Day,
                lastDateTime.Hour, lastDateTime.Minute, lastDateTime.Second,
                lastDateTime.Millisecond
                );
          if (!uniquify)
          
            break;
          
          if (newTimestampedString != lastTimestampedString)
          
            break;
          

          //Task.Delay(1).Wait();
        ;

        lastTimestampedString = newTimestampedString;
        Console.WriteLine("OUT - Thread: 0:D2 lastTimestampedString: 1",
            System.Threading.Thread.CurrentThread.ManagedThreadId,
            lastTimestampedString);

        if (uniquify)
        
          oldStrings.Add(lastTimestampedString);
        
        return lastTimestampedString;
      
    
  

Q）为什么我会收到这个间歇性错误（在此输出的末尾。）（我正在将一长串“NES”重复行复制到剪贴板并粘贴到控制台中以复制问题） ：

Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
NES
Enter symbol or [ENTER] to exit:
IN  - Thread: 18 lastTimestampedString:
OUT - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_472_NES
IN  - Thread: 17 lastTimestampedString: 2015_01_15_11_19_44_472_NES
OUT - Thread: 17 lastTimestampedString: 2015_01_15_11_19_44_473_NES
IN  - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_473_NES
OUT - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_493_NES
IN  - Thread: 16 lastTimestampedString: 2015_01_15_11_19_44_493_NES
OUT - Thread: 16 lastTimestampedString: 2015_01_15_11_19_44_494_NES
IN  - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_494_NES
OUT - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_495_NES
IN  - Thread: 17 lastTimestampedString: 2015_01_15_11_19_44_495_NES
IN  - Thread: 16 lastTimestampedString: 2015_01_15_11_19_44_495_NES
OUT - Thread: 17 lastTimestampedString: 2015_01_15_11_19_44_496_NES
IN  - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_495_NES
OUT - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_496_NES
IN  - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_496_NES
OUT - Thread: 16 lastTimestampedString: 2015_01_15_11_19_44_495_NES
OUT - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_497_NES
IN  - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_497_NES
OUT - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_523_NES
IN  - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_523_NES
OUT - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_532_NES
IN  - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_532_NES
OUT - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_533_NES
IN  - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_533_NES
Exception on thread: 17: The process cannot access the file 'C:\Users\drogers\_code\Tests\TestHttpClient\TestHttpClient2\bin\Debug\StockQuot
es\2015_01_15_11_19_44_495_NES.csv' because it is being used by another process.

Exception on thread: 16: The process cannot access the file 'C:\Users\drogers\_code\Tests\TestHttpClient\TestHttpClient2\bin\Debug\StockQuot
es\2015_01_15_11_19_44_496_NES.csv' because it is being used by another process.

OUT - Thread: 18 lastTimestampedString: 2015_01_15_11_19_44_540_NES
IN  - Thread: 17 lastTimestampedString: 2015_01_15_11_19_44_540_NES
IN  - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_540_NES
OUT - Thread: 17 lastTimestampedString: 2015_01_15_11_19_44_557_NES
OUT - Thread: 19 lastTimestampedString: 2015_01_15_11_19_44_560_NES
Exception on thread: 19: The process cannot access the file 'C:\Users\drogers\_code\Tests\TestHttpClient\TestHttpClient2\bin\Debug\StockQuot
es\2015_01_15_11_19_44_560_NES.csv' because it is being used by another process.

我可以避免取消注释第 126 行和注释第 127 行的问题，如：

  // This is an asynchronous world - lock the shared resource before using it!
  lock (dummy)
  //lock (lastTimestampedString)

看il，为FormatTimestampedString生成的代码唯一不同的是

ldsfld     string modreq([mscorlib]System.Runtime.CompilerServices.IsVolatile)
            TestHttpClient2.Program::**lastTimestampedString**

对

ldsfld     string modreq([mscorlib]System.Runtime.CompilerServices.IsVolatile) 
            TestHttpClient2.Program::**dummy**

【参考方案1】：

您有 2 个任务试图将股票数据写入同一个文件。

更改FormatTimestampedString 以检查生成文件名是否存在，如果存在则生成一个新的。

【讨论】：

甚至，只要发生异常就不要写入文件。用相同的确切时间戳写入数据似乎不太可能真正有用，精确到毫秒，不止一次。如果已经存在同名文件，您可能可以跳过再次写入文件。警告：您不能致电File.Exists() 进行测试；您必须实际尝试创建文件，并在发生异常时捕获异常。考虑直接使用FileStream 和FileMode.CreateNew，以便更容易区分可报告的异常和已经存在的文件。

重点是避免根据所选的命名算法创建重复的资源名称。

【参考方案2】：

字符串是不可变的。因此，当我在 lastTimestampedString 引用上设置我的锁，然后更改它时，我不再拥有我认为拥有的锁。锁在旧绳子上。其他任何人都将测试新字符串上的锁定，因此将被允许进入。

过失。