Well, my recent post on .NET 3.5 Brings BREAKING Changes to ThreadPool sparked quite a bit of interest in the .NET community.

But this is also something difficult to convince people of because it depends so heavily on configuration. The source code doesn't change, the environment does.

So I've put together a screencast demonstrating the problem and elaborating further. If you doubt the validity of the previous post, or can't reproduce the problem, please watch the video:




                     Download the video (approx 18 MB)

Kick it:

In the video I work with a modified version of the program. Here's that for your enjoyment:

The application: NetThreading.exe (From Video).zip (2.59 KB)

The source code:

using System;
using System.Threading;

namespace NewThreadPoolBehavior
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine( "Running on " + Environment.Version );
            int w, c;
            ThreadPool.GetMaxThreads( out w, out c );
            Console.WriteLine( "Max Currently: " + w + ", " + c );
            ThreadPool.GetMinThreads( out w, out c );
            Console.WriteLine( "Min Currently: " + w + ", " + c );

            Console.WriteLine( "Set min thread count 20? (y/n) " );
            string txt = Console.ReadLine();
            if ( txt == "y" )
            {
                Console.WriteLine( "Setting min to 20" );
                ThreadPool.SetMinThreads( 20, 100 );
                ThreadPool.GetMinThreads( out w, out c );
                Console.WriteLine( "Min Currently: " + w + ", " + c );
            }
            UseThreadPool( 200 );

            Console.ReadLine();
        }

        private static DateTime startTime;

        private static void UseThreadPool(int count)
        {
            startTime = DateTime.Now;
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

        private static int concurrent = 0;

        private static void SlowMethod()
        {
            TimeSpan dt = DateTime.Now - startTime;
            concurrent++;
            Console.WriteLine( "Starting ops (" + concurrent + " concurrent, elapsed=" +
               dt.TotalSeconds.ToString( "N3" ) + " sec.) " );
            Thread.Sleep( 20000 );
            Console.WriteLine( "Finished ops (" + concurrent + " concurrent)" );
            concurrent--;
        }
    }
}

Holy smokes! I thought we had figured out something significant when I posted .NET 3.5 Brings Major (Undocumented) Changes to ThreadPool where we discovered that the .NET 3.5 ThreadPool changed the allocation algorithm for adding threads from linear to logarithmic.

This is bigger. [Recently updated see note below]

Here's the scenario. I have a server - say in the financial sector - that must process many requests and it must get up to speed immediately. I can't pay the 500 ms warm up time for the ThreadPool (.NET 2.0) or the even slower model in .NET 2.0 SP1. What do I do? I call ThreadPool.SetMinThreads(x, x) where x < (the current max), but much higher than 2 (the default). So I might call ThreadPool.SetMinThreads(100, 100) or something like that.

In .NET 1.0 - .NET 2.0 (without SP1) you would go from the warm up time model:


                 warm up time model (.NET 2.0)

To this immediate "ready" threading model.

                 ready model (.NET 2.0)

Notice that we immediately have 100 threads available in this case. BUT, and here's the but: .NET 3.5 (read .NET 2.0 SP1) *ignores* SetMinThreads. Oh, it claims to respect SetMinThreads:

   Running on 2.0.50727.1433
   Max Currently: 500, 1000
   Min Currently: 2, 2
   Set min thread count 100 (y/n)? y
   Setting min to 100
   Min Currently: 100, 100

But, look at the thread graph that results!


           ready model (.NET 2.0 SP1  -- i.e. .NET 3.5)

I don't know about you, but that doesn't look any different AT ALL than if we do not call SetMinThreads:


         warm up time model (.NET 2.0 SP1  -- i.e. .NET 3.5)

What do we conclude from this? .NET 2.0 SP1 does not respect ThreadPool.SetMinThreads. To me, that's a big breaking change. My financial app just stopped working. Yikes! So what happened to my thread pool and SetMinThreads in .NET 2.0 SP1? Anyone?

Kick it:

Don't take my word for it. Run this program on both .NET 2.0 (SP0) and .NET 2.0 SP1 and you'll see for yourself.

Here's the EXE:

   NetThreading.zip (2.3 KB)

And the source code is below:

using System;
using System.Threading;

namespace NewThreadPoolBehavior
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine( "Running on " + Environment.Version );
            int w, c;
            ThreadPool.GetMaxThreads( out w, out c );
            Console.WriteLine( "Max Currently: " + w + ", " + c );
            ThreadPool.GetMinThreads( out w, out c );
            Console.WriteLine( "Min Currently: " + w + ", " + c );

            Console.WriteLine( "Set min thread count 100? (y/n) " );
            string txt = Console.ReadLine();
            if ( txt == "y" )
            {
                Console.WriteLine( "Setting min to 100" );
                ThreadPool.SetMinThreads( 100, 100 );
                ThreadPool.GetMinThreads( out w, out c );
                Console.WriteLine( "Min Currently: " + w + ", " + c );
            }
            UseThreadPool( 200 );

            Console.ReadLine();
        }

        private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

        private static int concurrent = 0;

        private static void SlowMethod()
        {
            concurrent++;
            Console.WriteLine( "Starting ops (" + concurrent + " concurrent)" );
            Thread.Sleep( 20000 );
            Console.WriteLine( "Finished ops (" + concurrent + " concurrent)" );
            concurrent--;
        }
    }
}

[Update: 2/25/2008 - You may be thinking so what's the big deal? Most applications don't directly program against the ThreadPool so this is just some edge case. To make this more real for everyone, this applies to you if you use any of the following: ASP.NET, WCF, .NET Remoting, Delegate.BeginInvoke, SqlCommand.BeginExecute*, Windows Workflow, and more. Sounds serious now right?]

[Update: 5/12/2008 - Please see my lastest followup on this topic: More on the ThreadPool Bug in .NET 2.0 SP1.]

It was all going so smoothly. Jason Whittington, Mark Smith and I were teaching the big DevelopMentor event here in Los Angeles (Guerrilla.NET) when my presentation on the ThreadPool took a nose dive. It started with a great joke involving Wilson (the volleyball from Cast Away).

Wilson and I built an application to compute a multiplication table where each computation was (artificially) slow. To speed it up we threw it at the thread pool using delegate.BeginInvoke. We figured that the ThreadPool would allocate 25 or so threads and the table would display quickly. Here's the expected output - pretty much the same thing we've seen since about .NET 1.0:

    

Each color represents the thread that did that computation.

For the last 7 years, the behavior has been that as the ThreadPool was overloaded, it would steadily start up new threads at the rate of one every 500 milliseconds until it hits its upper limit (typically). Using Performance Monitor (perfmon) we can watch the thread pool adding threads. It usually looks something like this:

    

Much to our surprise we saw completely different behavior. The thread pool added the first 15 or so threads quickly (as expected) but then stalled. New threads were not created every 500ms, instead they were added at increasingly long intervals. My demo took almost twice as long to run as it had the last time I did this demo a few months ago.

Jason, Mark, and I took this code, ported it back to .NET 1.1 and ran it side-by-side with .NET 3.5 and here's what we saw (blue = 3.5, red = 1.1):

    

As of .NET 3.5 the upper limit of the ThreadPool was increased: Knew that.

But, it appears that v3.5 of the CLR changes the policy for adding threads to the thread pool. Rather than adding threads regularly when under load the thread pool uses a logarithmic backoff. My colleague Jason Whittington remarked that this behavior looked similar to the behavior of the thread pool in "Rotor" (the shared-source version of the CLR). We speculated that this backoff algorithm makes sense given the new 250-thread per CPU maximum - it would take a long time to reach that if the runtime waits longer and longer to start a new thread. The 250-thread limits makes it less likely that your application will deadlock the thread pool, and the exponential backoff algorithm keeps the thread pool from creating too many threads too quickly.

Why should you care? Usually you won't, but it could have dramatic impact if you count on that behavior. For example, in our contrived case, .NET 1.1 ran about twice as fast as .NET 3.5 ( ! ):

    

Here's the program and source code (trimmed down to run in both .NET 1.1 and 3.5).

    Math.zip (6.38 KB)

Needless to say, Wilson and I felt kinda stood up.

     

Please see the follow up post on Breaking changes and now there is screencast, movie version as well.

Thanks to everyone who attended Guerrilla .NET last week. I had a great time meeting all of you. Here are the slides and demos from all our presentations.

     Demos.rar (22 MB)

    Challenge.rar (8 MB)

- Michael