HttpClient 异步请求未完成循环发送的大批量请求

我想我已经成功地进行了一次测试，至少在我的系统上重复地显示了这个问题。这个问题与 HttpClient 用于不良端点（端点不存在，目标已关闭）有关。

问题是已完成的任务数量低于总数，通常是几个左右。我不介意请求不起作用，但这只会导致应用程序在等待结果时挂在那里。

我从下面的测试代码中得到以下结果：

已用时间：237.2009884 秒。 批处理数组中的任务：8000 已完成的任务：7993

如果我将批量大小设置为 8 而不是 8000，它就会完成。对于 8000，它会卡在 WhenAll 上。

我想知道其他人是否得到相同的结果，我是否做错了什么，以及这是否是一个错误。

using System;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

namespace CustomArrayTesting
{

    /// <summary>
    /// Problem: a large batch of async http requests is done in a loop using HttpClient, and a few of them never complete
    /// </summary>
    class ProgramTestHttpClient
    {
        static readonly int batchSize = 8000; //large batch size brings about the problem

        static readonly Uri Target = new Uri("http://localhost:8080/BadAddress");

        static TimeSpan httpClientTimeout = TimeSpan.FromSeconds(3);  // short Timeout seems to bring about the problem.

        /// <summary>
        /// Sends off a bunch of async httpRequests using a loop, and then waits for the batch of requests to finish.
        /// I installed asp.net web api client libraries Nuget package.
        /// </summary>
        static void Main(String[] args)
        {
            httpClient.Timeout = httpClientTimeout; 

            stopWatch = new Stopwatch();
            stopWatch.Start();


            // this timer updates the screen with the number of completed tasks in the batch (See timerAction method bellow Main)
            TimerCallback _timerAction = timerAction;
            TimerCallback _resetTimer = ResetTimer;
            TimerCallback _timerCallback = _timerAction + _resetTimer;

            timer = new Timer(_timerCallback, null, TimeSpan.FromSeconds(1), Timeout.InfiniteTimeSpan);
            //

            for (int i = 0; i < batchSize; i++)
            {
                Task<HttpResponseMessage> _response = httpClient.PostAsJsonAsync<Object>(Target, new Object());//WatchRequestBody()

                Batch[i] = _response;
            }

            try
            {
                Task.WhenAll(Batch).Wait();
            }
            catch (Exception ex)
            {

            }

            timer.Dispose();
            timerAction(null);
            stopWatch.Stop();


            Console.WriteLine("Done");
            Console.ReadLine();
        }

        static readonly TimeSpan timerRepeat = TimeSpan.FromSeconds(1);

        static readonly HttpClient httpClient = new HttpClient();

        static Stopwatch stopWatch;

        static System.Threading.Timer timer;

        static readonly Task[] Batch = new Task[batchSize];

        static void timerAction(Object state)
        {
            Console.Clear();
            Console.WriteLine("Elapsed: {0} seconds.", stopWatch.Elapsed.TotalSeconds);
            var _tasks = from _task in Batch where _task != null select _task;
            int _tasksCount = _tasks.Count();

            var _completedTasks = from __task in _tasks where __task.IsCompleted select __task;
            int _completedTasksCount = _completedTasks.Count();

            Console.WriteLine("Tasks in batch array: {0}       Completed Tasks : {1} ", _tasksCount, _completedTasksCount);

        }

        static void ResetTimer(Object state)
        {
            timer.Change(timerRepeat, Timeout.InfiniteTimeSpan);
        }
    }
}

有时它只是在完成之前崩溃并出现访问冲突未处理的异常。调用堆栈只是说：

>   mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode = 1225, uint numBytes = 0, System.Threading.NativeOverlapped* pOVERLAP = 0x08b38b98) 
    [Native to Managed Transition]  
    kernel32.dll!@BaseThreadInitThunk@12()  
    ntdll.dll!___RtlUserThreadStart@8()     
    ntdll.dll!__RtlUserThreadStart@8()  

大多数时候它不会崩溃，但只是永远不会完成对whenall的等待。在任何情况下，每个请求都会抛出以下第一次机会异常：

A first chance exception of type 'System.Net.Sockets.SocketException' occurred in System.dll
A first chance exception of type 'System.Net.WebException' occurred in System.dll
A first chance exception of type 'System.AggregateException' occurred in mscorlib.dll
A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll

我让调试器在对象已处置异常上停止，并得到以下调用堆栈：

>   System.dll!System.Net.Sockets.NetworkStream.UnsafeBeginWrite(byte[] buffer, int offset, int size, System.AsyncCallback callback, object state) + 0x136 bytes    
    System.dll!System.Net.PooledStream.UnsafeBeginWrite(byte[] buffer, int offset, int size, System.AsyncCallback callback, object state) + 0x19 bytes  
    System.dll!System.Net.ConnectStream.WriteHeaders(bool async = true) + 0x105 bytes   
    System.dll!System.Net.HttpWebRequest.EndSubmitRequest() + 0x8a bytes    
    System.dll!System.Net.HttpWebRequest.SetRequestSubmitDone(System.Net.ConnectStream submitStream) + 0x11d bytes  
    System.dll!System.Net.Connection.CompleteConnection(bool async, System.Net.HttpWebRequest request = {System.Net.HttpWebRequest}) + 0x16c bytes  
    System.dll!System.Net.Connection.CompleteConnectionWrapper(object request, object state) + 0x4e bytes   
    System.dll!System.Net.PooledStream.ConnectionCallback(object owningObject, System.Exception e, System.Net.Sockets.Socket socket, System.Net.IPAddress address) + 0xf0 bytes 
    System.dll!System.Net.ServicePoint.ConnectSocketCallback(System.IAsyncResult asyncResult) + 0xe6 bytes  
    System.dll!System.Net.LazyAsyncResult.Complete(System.IntPtr userToken) + 0x65 bytes    
    System.dll!System.Net.ContextAwareResult.Complete(System.IntPtr userToken) + 0x92 bytes 
    System.dll!System.Net.LazyAsyncResult.ProtectedInvokeCallback(object result, System.IntPtr userToken) + 0xa6 bytes  
    System.dll!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped) + 0x98 bytes 
    mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP) + 0x6e bytes    
    [Native to Managed Transition]

异常消息是：

{"Cannot access a disposed object.\r\nObject name: 'System.Net.Sockets.NetworkStream'."}    System.Exception {System.ObjectDisposedException}

请注意与我很少看到的未处理的访问冲突异常的关系。

因此，当目标关闭时，HttpClient 似乎并不健壮。顺便说一句，我正在 Windows 7 32 上执行此操作。

我使用反射器查看了 HttpClient 的源代码。据我所知，对于操作的同步执行部分（当它启动时），返回的任务似乎没有超时。有一些超时实现在 HttpWebRequest 对象上调用 Abort()，但它们似乎再次错过了异步函数这一侧返回任务的任何超时取消或故障。回调方面可能有一些东西，但有时回调可能“丢失”，导致返回的任务永远不会完成。

我发布了一个问题，询问如何为任何任务添加超时，回答者给出了这个非常好的解决方案（此处作为扩展方法）：

public static Task<T> WithTimeout<T>(this Task<T> task, TimeSpan timeout)
{
    var delay = task.ContinueWith(t => t.Result
        , new CancellationTokenSource(timeout).Token);
    return Task.WhenAny(task, delay).Unwrap();
}

因此，像这样调用 HttpClient 应该可以防止任何“任务变坏”永远不会结束：

Task<HttpResponseMessage> _response = httpClient.PostAsJsonAsync<Object>(Target, new Object()).WithTimeout<HttpResponseMessage>(httpClient.Timeout);

我认为还有一些事情可以减少请求丢失的可能性： 1. 将超时从 3 秒增加到 30 秒，使我发布的带有此问题的程序中的所有任务都完成。 2. 增加允许的并发连接数，例如 System.Net.ServicePointManager.DefaultConnectionLimit = 100；