I'm glad to confirm that the ThreadPool bug introduced in .NET 3.5 has been fixed in SP1 for .NET 3.5.

For more background information, look at these posts.

1. .NET 3.5 Brings Breaking Changes to ThreadPool
2. Breaking Changes in the ThreadPool: The Movie
3. Follow up on .NET 2.0 SP1 ThreadPool Bug
4. More on the ThreadPool Bug in .NET 2.0 SP1

Now go forth, and create some threads!

When I talk about LINQ people often ask whether it’s possible to have dynamic queries with LINQ. An example of this is presenting the user with a UI that allows them to optionally filter by some criteria or other and sort by some criteria. This was straightforward if you built-up a SQL string in code.

The compiled, static nature of LINQ makes dynamic queries appear difficult at first. They are not and this video and sample application shows you how to accomplish just this.

                        Screen shot of sample application

You can download the source code here:

           Kennedy-Dynamic-Linq.zip (88.34 KB)

And you can watch the screencast video here:

          Kennedy-Dynamic-Linq.wmv (35.7 MB)

Of course, if you like this post, please kick it:

Do you watch a lot of video on your computer? I sure do. I download conference talks and interviews. I especially watch a lot of TV recorded by my Windows Media Center PC by copying the video files to my laptop. Media Center is SO much better than TiVo because you can copy / transfer / save video to other places (like your laptop).

That’s all well and good, except I really miss the ability to skip commercials and pause video via my media center remote control.


That’s been a pain to say the least inside Windows Media Player. Well goodbye days of frustrating commercials and TV via the mouse!

Introducing “Minimalist TV Player” (by me!):


It looks like Windows Media Player and has the same basic features for playing video.

But – you can control it like a Media Center TV to skip commercials and replay an action scene (like the crash shown above at the last Indy 500, just jump back 15 seconds to see the replay)!

You can use the keys PageUp to jump back 15 seconds and PageDown to skip commercials (forward 30 seconds). That’s a good start. But, if you have one of these presentation remotes:


Then you can use it to control the TV Player like a Media Center Remote! Here’s the control mapping:



Now you have the same controls as your Media Center PC on any computer that has Windows Media Player installed! How cool is that?

Download the beta 1 version here:  MinimalistTVPlayer (128 KB)

This beta version is fully functional but will expire in 2009. The supported file types depends somewhat on your system. The program will play any video files that your Windows Media Player can play. This usually includes WMV, MPG, AVI, and possibly DVR-MS files from Windows Media Center and DivX if you have the decoder installed.

One final tip: You can associate the application with your video file types and choose “Open With” to directly launch the program from a file:



Just be careful to not set the TV Player as the default program to load your video with unless that’s what you want.

If you this program, be sure to help promote it by kicking it!

[Update June 5, 2008: I'm working on a much slicker UI which I'll try to get out soon.]

There appears to be some confusion about this .NET 2.0 SP1 ThreadPool bug I found. Can you believe someone might be wrong on the internet?

If you’ve been following the posts (1, 2, 3), then you’ll know I was in touch with the key guys at Microsoft and the root of the problem is that if you create asynchronous work faster than the ThreadPool can create threads, then that work "misses" the SetMinThreads command and are created more slowly.

That is why the Thread.Sleep(1) probably solves the issue for you. If it doesn’t, sleep slightly longer. It’s a timing issue.

With this insight, you’ll see that this does not really affect true servers such as ASP.NET and WCF but does affect things like Delegate.BeginInvoke and friends called in quick succession.

In short, if your server handles more than 1,000 requests per second, during the first second of startup, then your server may run into this problem (ok, ok, more precisely, if your server handles more than 1 request / millisecond).

Let me address a few common concerns out there:

• Is the sky falling? No.
• Is it great situation? No.
• Will be fixed with .NET 2.0 SP2? Yes
• When? Maybe this summer, maybe later - they aren’t sure.
• Does this affect WCF and ASP.NET? Probably not. The bug, no. The new ThreadPool behavior, yes.
• Is there a work around? Yes, create threads more slowly. This is automatic on the server.

You may have heard that the sessions for Mix 08 were posted online and you can view or download them until your heart is content.

Mix 08 is becoming an increasingly important event in the .NET space. I spent a fair amount of time watching the sessions in the evenings. I was teaching a class and couldn't attend myself.

So who wouldn't want to curl up with a cold beer, a warm laptop, and a bunch of .NET presentations? That's what I want to know. But, maybe you don't have time to go through all 89 sessions? Don't despair, I'll distill them down for you.

BTW, I'd love to link directly to them, but SilverLight and Flash are broken web metaphors (you can't link their content) so you'll have to find them at http://sessions.visitmix.com/.

Without further ado, I present to you my top 5 most important sessions from Mix 08:

1. T22 - Developing ASP.NET Applications Using the Model View Controller Pattern
   Scott Hanselman
   
   
2. T01 - Creating a RESTful API with Windows Communication Foundation
   Aaron Sloman and Haider Sabri
   
   
3. UX03 - The Back of the Napkin: Solving Design Problems (and Selling Your Solutions) with Pictures
   Dan Roam
   
   
4. T11 - What's New in Windows Presentation Foundation 3.5
   Rob Relyea
   
   
5. T26 - Building Applicaitons and Services with.NET Framework 3.5
   Justin Smith

Of course the keynotes were great too - but they don't really belong in this category.

So get out there and learn something! :) And kick it if you like it:

I've been looking around CodePlex lately and there's some really cool stuff there. For example, the source code to ASP.NET MVC.

That got me thinking, what else is out there? Here are my (current) top 5 favorite CodePlex projects.

#1. ASP.NET MVC
http://www.codeplex.com/aspnet
This project gives you access to the source code for upcoming releases that the Microsoft ASP.NET team is working on, starting with the ASP.NET MVC Framework. The project gives you a look at the design and lets you have a voice in it. You can send us feedback for the ASP.NET MVC framework through the ASP.NET MVC forums on the ASP.NET site.

#2 Script#
http://www.codeplex.com/scriptsharp
Script# enables more productive Ajax application development by allowing you to compile your C# source code into JavaScript. It allows you to use standard .NET tools like msbuild projects, Visual Studio and IDE intellisense, reflector, refactoring tools, amongst various others.

#3 AJAX Control Toolkit
http://www.codeplex.com/AtlasControlToolkit
The AJAX Control Toolkit is a joint project between the community and Microsoft. Built upon the ASP.NET 2.0 AJAX Extensions, the Toolkit aims to be the biggest and best collection of web-client components available.

#4 .NET Mass Downloader
http://www.codeplex.com/NetMassDownloader
Mass Downloader For .Net Framework which allows you do download .Net Framework source code in batch.
The tool which enables offline debugging of .Net Framework in VS2008(including Express Editions) , VS2005 (including Express Editions), and Codegear Rad Studio.

#5 MVC Contrib - MvcContrib.org
http://www.codeplex.com/MVCContrib
This project will be a series of assemblies that add functionality to Microsoft's ASP.NET MVC Framework and make the framework easier to use. *Download the MVC Framework CTP here: http://asp.net/downloads/3.5-extensions/. Download here: http://go.microsoft.com/fwlink/?LinkID=105029. Some of the items to be included are ...

#5+1 ASP.NET Upload Utility: Big Mailer
http://www.codeplex.com/bigmailer
Email something huge: The big mailer project allows you to upload large files to your website and then retrieve a convenient url for email and other uses.

#5+2 Visual Studio Most Recent Files Utility
http://www.codeplex.com/vsrecentfiles
Manage your recent projects and solutions list for Visual Studio. Supports VS 2003, 2005, and 2008.

[OK, those last two are projects are mine, but they are pretty cool.]

Did I miss something spectacular? Add it to the comments below!

- Michael Kennedy

If you'v been using my Visual Studio Recent Files Utility, then you may be interested to find out that I just released it as open source on CodePlex here.

    Visual Studio Recent Files Utility on CodePlex

Enjoy!

Hi All,

I recently added my Big Mailer project to CodePlex.

     Big Mailer on CodePlex

You can download the source code among other things.

Enjoy!

I often have to send large files around by email. For example, I had to email a colleague a 10 MB file. It seems a little rude to hit him out of the blue with a 10 MB email. I wanted something cleaner and less intrusive.

So I created a simple utility I called "Big Mailer". I figured I'd blog about it and share it with the world. Feel free to use it as you see fit. This program consists of both a client and server piece that simplifies uploading content to your web site. Then the program gives you a regular web link you can send via email.

Here's a screen shot:

You can download and install it via ClickOnce here:

Install Big Mailer (700 KB)

You must have the following to use this program:

1. .NET 3.5 Framework installed on the client
2. .NET 3.5 Framework on the server
3. An ASP.NET web site to host the WCF service

After you install the client, click "Host Service" and you'll get the server side code to drop onto your ASP.NET web site. There is a test web site with instructions in that code.

If you don't have an ASP.NET web site to host the service at you can always use public services like Drop Boks (a great service!) or other file sharing sites. The benefit of this program / service is that you retain control of the files and you can conceivably send more sensitive content.

You can also just use FTP if you have that for your web site, but I hate FTP personally. I don't like fighting the firewall issues and I don't want it running on my servers.

So this program allows you to upload content of unlimited size, without FTP, without sending your files to a third party. Also, it sends everything in 16KB blocks, rather than one giant http message. So you get the benefit of both being able to send huge files (say 1 GB) and you get progress / cancel support.

One feature that's notably lacking is authentication. I'll probably release an update with security built-in. For now, take that into consideration before using it.

Enjoy!

PS - This project is now Open Source and is host on CodePlex.

Here's a followup post on the .NET ThreadPool bug that I described here:

Breaking Changes in the ThreadPool: The Movie

I have been in touch with the guys who are in charge of the ThreadPool and they have both confirmed that this is a bug and that they are planning on fixing it in .NET 2.0 SP2 - but they are not sure of the timeline for its release.

In the meantime, Vance Morrison, a .NET Runtime Performance Architect at Microsoft, has given me this work-around.

Take this "broken" code:

	private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

And add a strategic Thread.Sleep and it's fixed:

	private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
		Thread.Sleep(1);
            }
        }

Well, my recent post on .NET 3.5 Brings BREAKING Changes to ThreadPool sparked quite a bit of interest in the .NET community.

But this is also something difficult to convince people of because it depends so heavily on configuration. The source code doesn't change, the environment does.

So I've put together a screencast demonstrating the problem and elaborating further. If you doubt the validity of the previous post, or can't reproduce the problem, please watch the video:




                     Download the video (approx 18 MB)

Kick it:

In the video I work with a modified version of the program. Here's that for your enjoyment:

The application: NetThreading.exe (From Video).zip (2.59 KB)

The source code:

using System;
using System.Threading;

namespace NewThreadPoolBehavior
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine( "Running on " + Environment.Version );
            int w, c;
            ThreadPool.GetMaxThreads( out w, out c );
            Console.WriteLine( "Max Currently: " + w + ", " + c );
            ThreadPool.GetMinThreads( out w, out c );
            Console.WriteLine( "Min Currently: " + w + ", " + c );

            Console.WriteLine( "Set min thread count 20? (y/n) " );
            string txt = Console.ReadLine();
            if ( txt == "y" )
            {
                Console.WriteLine( "Setting min to 20" );
                ThreadPool.SetMinThreads( 20, 100 );
                ThreadPool.GetMinThreads( out w, out c );
                Console.WriteLine( "Min Currently: " + w + ", " + c );
            }
            UseThreadPool( 200 );

            Console.ReadLine();
        }

        private static DateTime startTime;

        private static void UseThreadPool(int count)
        {
            startTime = DateTime.Now;
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

        private static int concurrent = 0;

        private static void SlowMethod()
        {
            TimeSpan dt = DateTime.Now - startTime;
            concurrent++;
            Console.WriteLine( "Starting ops (" + concurrent + " concurrent, elapsed=" +
               dt.TotalSeconds.ToString( "N3" ) + " sec.) " );
            Thread.Sleep( 20000 );
            Console.WriteLine( "Finished ops (" + concurrent + " concurrent)" );
            concurrent--;
        }
    }
}

Holy smokes! I thought we had figured out something significant when I posted .NET 3.5 Brings Major (Undocumented) Changes to ThreadPool where we discovered that the .NET 3.5 ThreadPool changed the allocation algorithm for adding threads from linear to logarithmic.

This is bigger. [Recently updated see note below]

Here's the scenario. I have a server - say in the financial sector - that must process many requests and it must get up to speed immediately. I can't pay the 500 ms warm up time for the ThreadPool (.NET 2.0) or the even slower model in .NET 2.0 SP1. What do I do? I call ThreadPool.SetMinThreads(x, x) where x < (the current max), but much higher than 2 (the default). So I might call ThreadPool.SetMinThreads(100, 100) or something like that.

In .NET 1.0 - .NET 2.0 (without SP1) you would go from the warm up time model:


                 warm up time model (.NET 2.0)

To this immediate "ready" threading model.

                 ready model (.NET 2.0)

Notice that we immediately have 100 threads available in this case. BUT, and here's the but: .NET 3.5 (read .NET 2.0 SP1) *ignores* SetMinThreads. Oh, it claims to respect SetMinThreads:

   Running on 2.0.50727.1433
   Max Currently: 500, 1000
   Min Currently: 2, 2
   Set min thread count 100 (y/n)? y
   Setting min to 100
   Min Currently: 100, 100

But, look at the thread graph that results!


           ready model (.NET 2.0 SP1  -- i.e. .NET 3.5)

I don't know about you, but that doesn't look any different AT ALL than if we do not call SetMinThreads:


         warm up time model (.NET 2.0 SP1  -- i.e. .NET 3.5)

What do we conclude from this? .NET 2.0 SP1 does not respect ThreadPool.SetMinThreads. To me, that's a big breaking change. My financial app just stopped working. Yikes! So what happened to my thread pool and SetMinThreads in .NET 2.0 SP1? Anyone?

Kick it:

Don't take my word for it. Run this program on both .NET 2.0 (SP0) and .NET 2.0 SP1 and you'll see for yourself.

Here's the EXE:

   NetThreading.zip (2.3 KB)

And the source code is below:

using System;
using System.Threading;

namespace NewThreadPoolBehavior
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine( "Running on " + Environment.Version );
            int w, c;
            ThreadPool.GetMaxThreads( out w, out c );
            Console.WriteLine( "Max Currently: " + w + ", " + c );
            ThreadPool.GetMinThreads( out w, out c );
            Console.WriteLine( "Min Currently: " + w + ", " + c );

            Console.WriteLine( "Set min thread count 100? (y/n) " );
            string txt = Console.ReadLine();
            if ( txt == "y" )
            {
                Console.WriteLine( "Setting min to 100" );
                ThreadPool.SetMinThreads( 100, 100 );
                ThreadPool.GetMinThreads( out w, out c );
                Console.WriteLine( "Min Currently: " + w + ", " + c );
            }
            UseThreadPool( 200 );

            Console.ReadLine();
        }

        private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

        private static int concurrent = 0;

        private static void SlowMethod()
        {
            concurrent++;
            Console.WriteLine( "Starting ops (" + concurrent + " concurrent)" );
            Thread.Sleep( 20000 );
            Console.WriteLine( "Finished ops (" + concurrent + " concurrent)" );
            concurrent--;
        }
    }
}

[Update: 2/25/2008 - You may be thinking so what's the big deal? Most applications don't directly program against the ThreadPool so this is just some edge case. To make this more real for everyone, this applies to you if you use any of the following: ASP.NET, WCF, .NET Remoting, Delegate.BeginInvoke, SqlCommand.BeginExecute*, Windows Workflow, and more. Sounds serious now right?]

[Update: 5/12/2008 - Please see my lastest followup on this topic: More on the ThreadPool Bug in .NET 2.0 SP1.]

It was all going so smoothly. Jason Whittington, Mark Smith and I were teaching the big DevelopMentor event here in Los Angeles (Guerrilla.NET) when my presentation on the ThreadPool took a nose dive. It started with a great joke involving Wilson (the volleyball from Cast Away).

Wilson and I built an application to compute a multiplication table where each computation was (artificially) slow. To speed it up we threw it at the thread pool using delegate.BeginInvoke. We figured that the ThreadPool would allocate 25 or so threads and the table would display quickly. Here's the expected output - pretty much the same thing we've seen since about .NET 1.0:

    

Each color represents the thread that did that computation.

For the last 7 years, the behavior has been that as the ThreadPool was overloaded, it would steadily start up new threads at the rate of one every 500 milliseconds until it hits its upper limit (typically). Using Performance Monitor (perfmon) we can watch the thread pool adding threads. It usually looks something like this:

    

Much to our surprise we saw completely different behavior. The thread pool added the first 15 or so threads quickly (as expected) but then stalled. New threads were not created every 500ms, instead they were added at increasingly long intervals. My demo took almost twice as long to run as it had the last time I did this demo a few months ago.

Jason, Mark, and I took this code, ported it back to .NET 1.1 and ran it side-by-side with .NET 3.5 and here's what we saw (blue = 3.5, red = 1.1):

    

As of .NET 3.5 the upper limit of the ThreadPool was increased: Knew that.

But, it appears that v3.5 of the CLR changes the policy for adding threads to the thread pool. Rather than adding threads regularly when under load the thread pool uses a logarithmic backoff. My colleague Jason Whittington remarked that this behavior looked similar to the behavior of the thread pool in "Rotor" (the shared-source version of the CLR). We speculated that this backoff algorithm makes sense given the new 250-thread per CPU maximum - it would take a long time to reach that if the runtime waits longer and longer to start a new thread. The 250-thread limits makes it less likely that your application will deadlock the thread pool, and the exponential backoff algorithm keeps the thread pool from creating too many threads too quickly.

Why should you care? Usually you won't, but it could have dramatic impact if you count on that behavior. For example, in our contrived case, .NET 1.1 ran about twice as fast as .NET 3.5 ( ! ):

    

Here's the program and source code (trimmed down to run in both .NET 1.1 and 3.5).

    Math.zip (6.38 KB)

Needless to say, Wilson and I felt kinda stood up.

     

Please see the follow up post on Breaking changes and now there is screencast, movie version as well.

Thanks to everyone who attended Guerrilla .NET last week. I had a great time meeting all of you. Here are the slides and demos from all our presentations.

     Demos.rar (22 MB)

    Challenge.rar (8 MB)

- Michael

I recently wrote up an overview of the new ASP.NET MVC Framework for the Developments newsletter.

I encourage you to read it on the DevelopMentor website. It's an interesting programming model.

I've been playing with my fresh copy of Vista Ultimate - which I am surprised to find that I absolutely love.

Being a big fan of System.Transactions, I naturally wanted to use it with Vista's TxF (Transactional NTFS) file system. But unlike the data libraries, the file APIs don't auto-enlist in the transaction. In fact, there are only COM / PInvoke APIs currently.

There is a nice article about how to work with these APIs in the MSDN article: "NTFS: Enhance Your Apps With File System Transactions". But I was unimpressed with the managed wrapper they created there. In particular, I don't like that the lifetime of the file stream is not forced to be part of a client initiated transaction scope. So I built my own transactional file stream in C#. With this TxFileStream class, you can write succinct code like this:

[Test]
public void ContentAddedDoesNotPersistsAfterRollbackTest()
{
    string fileName = "file3.txt";
    string originalContents = "First contents";
   
    using (TransactionScope scope = new TransactionScope())
    {
        string newContents = "Hello transacted NTFS.";
        using (StreamWriter sw = TxFileStream.CreateWriter(fileName))
        {
            sw.Write(newContents);
        }

        using (StreamReader sr = TxFileStream.CreateReader(fileName))
        {
            string text = sr.ReadToEnd();
            Assert.That(text, Is.EqualTo(newContents));
        }
        // no call to scope.Complete() forces a rollback.
    }

    Assert.That(File.Exists(fileName));
    Assert.That(File.ReadAllText(fileName),
        Is.EqualTo(originalContents));
}

Feel free to download the code and give it a spin!

    Kennedy.TxFiles.zip (36 KB)

Of course, my library comes with comprehensive unit tests. Look here first to figure out how out use the library.

Note: I fixed a bug in creating transactional StreamWriter's in append mode. Previously they partially overwrote the existing content.