[note: This screencast has been adapted from my earlier blog post.]

      

In Part 1 of my Building a Cloud OS for .NET Developers series, I talked about setting up a pure cloud OS focused on developers. But the one crucial aspect I left out was the developer tools.

In this second installment, we’ll be covering exactly that. How do we manage having Visual Studio and associated tools and servers universally accessible in the cloud, even on mobile devices such as iPads?

We’ll start with the king of the cloud:

    Amazon Elastic Compute Cloud (Amazon EC2)
    http://aws.amazon.com/ec2/

At EC2 we can create a variety of Windows instances, get full admin access via remote desktop, and install anything we want (Visual Studio, SQL Server, MongoDB, etc). We can also run these for just 30 minutes here, 2 hours there and we’ll only be charge for the time we use it.

We are going to setup a Windows Server in EC2 which we can access from our Cloud OS, or any other machine or mobile device, and deck it out with all the developer tools we want. You have to choose your server type based on what you’re planning on doing as well as how much you want to pay. Here are the available options and pricing. You can roughly expect to pay $0.50/hr for a good workstation, or $0.12/hr for an acceptable but small one.

Amazon EC2 Instance Types: http://aws.amazon.com/ec2/instance-types/
Amazon EC2 Pricing: http://aws.amazon.com/ec2/pricing/

I’m currently using a Micro Instance for an SVN server which is always running as well as High-Memory Extra Large Instance for my development machine that I start and stop on-demand when I need to get some programming done. Here are the specs for the developer workstation:

High-Memory Extra Large Instance

17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform
I/O Performance: Moderate
API name: m2.xlarge

You have to admit, that’s a pretty killer workstation. It’s a overkill on the memory, but has better processors than the cheaper ones which also have decent memory.

The price of the dev machine is $0.69 / hour. I’ve also built some tools which help shut down the machine in case I get distracted. So far I’ve been using it pretty heavily this past few weeks and have spent probably $13 on it. That’s not bad for the possibility of being anywhere in the world (with an Internet connection) and getting my workstation running in an instant. OK, maybe it takes longer than an instant. To be fair, it usually it’s 10-15 seconds :).

Take the Free Lunch

One thing you should do straight away if you’re going to try this is sign up for the EC2 Free Tier:

AWS Free Usage Tier
http://aws.amazon.com/free/

This won’t cover the EC2 CPU time (that’s Linux only), but it will cover storage, bandwidth, S3 snapshots, etc. You also get a free Linux machine to play with if you like.

Setting up Windows in the Cloud

OK, so if you’re chosen an instance type and are ready to start, you should do the EC2 walk-through I put together here to get your machine up and running and login for the first time.

Building Windows Machines in Amazon EC2
http://www.michaelckennedy.net/blog/2010/01/31/BuildingWindowsMachinesInAmazonEC2.aspx

I’d guess that takes about 15 minutes. When you’re done, come back here...

Installing Software in EC2 and Windows

So you’ve logged in and you want to install Visual Studio, SQL Server, and other MSDN goodies. The easiest way to get them is to use the ISO DVD images from the MSDN download site. In order to mount them on your EC2 machine (which has no DVD drive), you should install

Virtual CloneDrive
http://www.slysoft.com/en/virtual-clonedrive.html

which lets you mount ISO images as if they were proper DVD drives. Then you can download and install all the MSDN files you need. A good place to start is with Visual Studio:

Microsoft Visual Studio 2010 Premium Trial - ISO
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=f81412a2-d48e-4040-9b32-27eaf771c5db&displaylang=en

One word of warning here: I’ve noticed that Chrome can freak out and fail to download very large files inside your EC2 instance. Don’t know why, but it does. So be sure to use IE or FireFox to do the downloads for the ISOs.

Syncing your Files Between your Local Machine and Dev EC2 Machine

There are two techniques we can use to sync files, depending on what category of files we’re sharing and how. For the source files, you can always setup an SVN Server (that’s what my micro instance is about). I’m a fan of Visual SVN. It’s free and easy to setup and admin.

VISUALSVN SERVER
http://www.visualsvn.com/server/

Then on your EC2 dev machine (and your others you may have) setup:

TortoiseSVN
http://tortoisesvn.net/

and

AnkhSVN for Visual Studio Integration
http://ankhsvn.open.collab.net/

Then you have super easy, reliable, and secure sharing of your source files.

    

For everything else, you can just setup a Dropbox share between your Cloud OS and you EC2 Dev Machine:


      


      

If you don’t have Dropbox and want to try it, use this link to create an account

http://db.tt/Qo7m9PA

and you’ll get me some extra free space which I always appreciate. :) Then if you have a file on your local computer you want accessible on your cloud dev machine, just drop it in your Dropbox folder on the originating computer and it’ll be there as soon as it uploads.

Waste Not, Want Not

Now here is a problem you will encounter:

You’re working on your dev machine in EC2, something comes up and you get distracted and forget about it for the rest of the day. When you come back the next day you see your EC2 system is still running and you’ve wasted $12 by leaving it running idle overnight. That sucks. If it happens on Friday, it might run the whole weekend which is even worse.

To avoid this, I wrote a little tray utility that watches for extended idle times and will shutdown your EC2 instance if it’s idle for too long.
     

Below is how you configure it. Simple, right?

    
Add it to your startup menu to make sure it starts when you log in.

     

You can download it here. I may blog about this app in greater detail later, for now feel free to use it.

Shutdown on Idle
http://www.michaelckennedy.com/Downloads/Tools/ShutdownAfterIdle.zip

Reliable Access to EC2

One annoyance in EC2 is that every time you turn off your EC2 computer, it will take on a new random public DNS name. If you really want to access to your EC2 machine from anywhere, you probably want a reliable DNS name / IP address. That’s where Elastic IPs come into play in EC2:

Feature Guide: Amazon EC2 Elastic IP Addresses
http://aws.amazon.com/articles/1346

If you create an elastic IP, then you can map it to a sub-domain in your domain and always use that. For example I could register it for cloud-developer.michaelckennedy.net. Then as long as I have a way to start my machine and re-associate the IP, all my config on my various machines (such as remembered RDP settings and passwords and such) will “stick”.

Be aware that while your EC2 machine is off, you’ll be charged a small fee (few dollars / month) for the address. You can decide whether it’s worth the trouble to save the few dollars.

Mobile / iPad Access

I’m sure when you think of accessing your EC2 dev machines, you’re thinking of doing that form Windows or maybe a Mac (using  http://www.microsoft.com/mac/remote-desktop-client). But the iPad (and even the iPhone) has apps to allow you to use Remote Desktop to EC2. For example, here’s editing the code for http://chatpast.com in Visual Studio via my iPad:


      
                                    (click for full size image)

And you can even bring up the keyboard or attach a Bluetooth keyboard.

      
                                    (click for full size image)

The app I was using is called “Jump”, and you can find it here:

Jump for iPad
http://jumpdesktop.com/

Summary

I hope that gives you some inspiration to try a new way to work in the cloud. You saw that we can create fairly serious EC2 Windows computers which can serve as developer workstations. We can share files via source control and Dropbox and we can access the computers from Windows, from Macs, and even mobile devices like an iPad.

Good luck!
@mkennedy

[Update: Read Part 2...]

Let’s talk about cloud operating systems. This article explores the options and potential of moving entirely “To The Cloud” for developers who normally demand significant offline power from their applications (IDEs, compilers, debuggers, etc). We’ll focus on .NET / Visual Studio developers, but I’m sure you can adapt this to your technology of choice.

I’ve been fascinated with what Google is doing with Chrome OS. I think finding a way to fully “live in the cloud” has a lot of promise. However, in practice Chrome OS is entirely unappealing to me. It’s just a single browser window, maximized. Yuck. I don’t mind doing most things in the browser, but I’d like multiple non-maximized browser windows and a desktop to organize things like shortcuts. That’s OK though because Chromebooks aren’t the only option. We have decent operating systems right now that can function largely in the same way if we set them up with discipline.

What we’re going to do is take a pristine Windows 7 instance and deck it out for the cloud. Even with all the choices these days, Windows 7 is still my favorite OS. I do have a Mac, several iOS devices, and I could always install Linux. Nevertheless, it’s Windows 7 for me so that’s where we’ll start.

Creating a Fresh Windows 7 Instance - Gently

I want to start from the very beginning. I don’t want old apps I’ve already installed encouraging me to “cheat” and work outside of the cloud more than necessary. At the same time I need to be able to fire up Visual Studio and do some work with an unreliable or nonexistent Internet connection So I’m keeping my current “full” Windows system intact. We’ll install a secondary instance of Windows using the awesome Boot to VHD feature in Windows 7 Ultimate. If you haven’t heard of it, I did a video walk-through here:

   Boot to VHD Screencast:
   http://www.michaelckennedy.net/blog/2009/10/20/BootToVHDScreencast.aspx

Scott Hanselman’s write up is very helpful too.

   Less Virtual, More Machine - Windows 7 and the magic of Boot to VHD:
   http://www.hanselman.com/blog/LessVirtualMoreMachineWindows7AndTheMagicOfBootToVHD.aspx

This process takes about 15 minutes. Once you’ve got everything setup, I recommend you activate and then snapshot that VHD file (from your main OS) so you can always get back to “clean”. If you’re really living the cloud lifestyle, than paving your system is much easier than before. I always keep a secondary partition around for data files anyway so that can be shared across OSes.

Now that you’ve got a truly fresh Windows 7 copy, it’s time to install just a few apps needed to set you free (and a few you’d rather not live without).

Installing “The Cloud”

While your fresh copy of Windows 7 does have IE 8 installed, that browser is entirely not up to the task of being where you spend 90% of your time. I strongly encourage you to give the latest Chrome beta a shot as your main browser. The “application shortcuts” feature of Chrome makes it much more immersive than anything IE 9 or FireFox 4 are doing (yes, I know they have pinned sites, that’s not even close).

   Chrome Beta
   http://www.google.com/intl/en/landing/chrome/beta/

Got Chrome installed and set as your default browser? Good. Don’t forget to install all your Windows updates and Windows 7 SP1 while you’re at it.

What about Visual Studio?

You may be thinking, “I *NEED* Visual Studio 2010”. Yes, we do need it for sure, but stick with me here. We are NOT installing that beast on our fresh Windows 7 system. Remember, this is your Cloud OS. I even named mine Cloud OS in my boot loader to keep that focused in my mind.

      

We’ll talk about Visual Studio soon.

Installing Your Web Applications

You may think that you do not need to install your web apps cloud apps. There are a few installers we’ll be running such as DropBox and ChatPast, but even the plain vanilla websites such as GMail are better if you install them. That’s why Chrome is way better than the other browsers.

Let’s suppose you’ll want to use Google Docs to work with documents, presentations, Excel worksheets and so on (you will want this!). Visit https://docs.google.com, then choose “Create application shortcut” in Chrome (see image):

   

This not only gives you start menu items and desktop shortcuts, but it makes your taskbar truly useful and your web apps behave as regular Windows apps.

  

In addition, you get the true feel of your websites being applications without all the browser toolbars and other junk around it. For example, here’s how this article (barely written) looks in Google Docs right now. Notice, how it looks like an app rather than a tab buried in a busy browser. It remembers its window size and position. In short, it’s more application-like.



What Cloud Apps Should I Install?

OK, so I hope I convinced you that treating your web apps as true applications is worthwhile. Now which ones do I install? Here are a few I recommend. You may have your own favorite sites. We’ll also need a few apps that do cloudy stuff but run MSI installers on our system.

- GMail: https://mail.google.com/ (obviously :) )
- Google Docs: https://docs.google.com (good Microsoft Office replacement)
- Google Calendar: https://www.google.com/calendar/
- Google Contacts: http://mail.google.com/mail/contacts/u/0/ui/ContactManager
- 37 Signals LaunchPad: https://launchpad.37signals.com (Great web apps for projects)
- Amazon Web Service: https://console.aws.amazon.com/ec2/ (You’ll see why soon)
- Twitter: http://twitter.com/
- Office Online: http://office.live.com (for when Google docs fail you)
- Pandora: http://www.pandora.com/

Here are a few proper apps that are cloud-based in some way or another that require installers.

- Skype (obviously :) http://www.skype.com
- DropBox: https://www.dropbox.com
- ChatPast Sync Client: https://chatpast.com/

By using GMail, Google Contacts, and Google Calendar, you obviously get portability from the web. But if you have a modern phone or tablet, chances are you can keep them 100% in sync using Google’s Exchange support. I do that for both my iPhone and iPad.

There are also a couple of light-weight apps that I just didn’t want to do without and still have a solid use in a Cloud OS. I needed a good image editing app (for writing blog posts like this) and one for screen captures. So we have

- Paint.NET: http://download.cnet.com/Paint-NET/3000-2192_4-10338146.html
- Window Clippings: http://www.windowclippings.com/

Finally, there are a couple that I use to just keep an eye on my system such as

- Process Explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653

Here is the full list of *everything* I have installed on my system. Just 11 applications installed. Note that I’ve hidden some that Windows installs without asking. For example, using my Microsoft Mouse forces an install of Microsoft’s Intellipoint software when I plug it in. Similarly for the touch-pad on my laptop. To me, those don’t count so I edited them out.

   

Escape from Outlook

You may be thinking that because you still use your company’s email that you’ll need to install Microsoft Office and the 800-lbs gorilla that is Outlook. Chances are you will not need Outlook. I have several “regular” email accounts I have to check that don’t have reasonable web options.

You can setup your GMail account to automatically pull from your other email accounts as if GMail itself was a POP3 client like Outlook. Just go to Gmail > Settings >  Accounts and Import > and configure “Check mail using POP3” as well as “Send mail as”. Here you see my other accounts have recently been checked and received mail.


  

Sometimes having a “Send and Receive” button for these accounts is handy. Visit the Labs section of GMail and install the “Refresh POP accounts” add-on to make the refresh button propagate a refresh off to your other accounts as well.

  
While you are in there, you should turn on the keyboard shortcuts and take 5 minutes to learn the important ones. For example, / will take you to search, i to the inbox, e archives, <shift>-c composes an email in a separate window, etc.

Where Are the Developer Tools?

You probably didn’t see Visual Studio in that list, because it wasn’t there! In the next blog post, I’ll take you through setting up an Visual Studio 2010 instance in the cloud along with all the other developer tools and servers you might need such as SQL Server, and MongoDB, and LINQPad, and so on. Stay tuned.

When I have that post online, I’ll be sure to add a link here. If you just can’t wait, here’s a hint of where we're going next.

[Update: Read Part 2 Now]

Good luck!
@mkennedy

So maybe I'm late to the party, but I recently started playing with NuGet. It's a killer new way to find, install, maintain, and manage references to open source libraries in Visual Studio 2010. Plenty of people have written about it (Phil Haack and Scott Hanselman for example). Let's just say you should learn about NuGet if you don't know it already.

What I want to talk about is all the cool open source projects I found just by flipping through the pages of the NuGet directory in the Visual Studio "Add Library Package Reference" dialog.

1. RazorEngine at http://razorengine.codeplex.com/

RazorEngine is templating engine built upon Microsoft's Razor parsing technology. The Razor Templating Engine allows you to use Razor syntax to build robust templates. No need to learn a custom clunky API for generating things like HTML and emails and so on. Just use the hot new @Razor syntax from ASP.NET MVC 3.

2. YUI Compressor for .Net at http://yuicompressor.codeplex.com/

YUI Compressor for .Net is is a .NET port of the Yahoo! UI Library's YUI Compressor Java project. Do you have a bunch of CSS and JavaScript files and you want your page to load faster. This is a great way to do it from ASP.NET.

3. 51degrees.mobi at http://51degrees.codeplex.com/

Want to build an ASP.NET MVC website that has both a desktop and mobile version from the same project? 51degrees.mobi Foundation is an ASP.NET open source module which detects mobile devices and browsers, enhancing the information available to ASP.NET. Mobile handsets can optionally be redirected to a home page designed for mobile phones. Smart phone and feature phones are all supported.

4. Lucene.Net at http://lucene.apache.org/lucene.net/

Lucene.Net is a source code, class-per-class, API-per-API and algorithmatic port of the Java Lucene search engine to the C# and .NET platform utilizing Microsoft .NET Framework. Want indexed full-text search from .NET? Here you go.

5. MvcMailer at http://www.codeproject.com/KB/aspnet/MvcMailerNuGet.aspx

Send a professional looking HTML email from your ASP.NET MVC simply by pointing at a particular view.

6. DotNetZip at http://dotnetzip.codeplex.com/

DotNetZip is an easy-to-use, FAST, FREE class library and toolset for manipulating zip files or folders. Zip and Unzip is easy: with DotNetZip, .NET applications written in VB, C# - any .NET language - can easily create, read, extract, or update zip files. For Mono or MS .NET.

7. FileDB at http://filedb.codeplex.com/

Ever wanted to have a single, simple file that you can access as a basic database without the fuss? FileDB is a free, fast, lightweight C# (v3.5) DLL project to store, retrieve and delete files using a single archive file as a container on disk. It's ideal for storing files (all kind, all sizes) without databases and keeping them organized on a single disk file.

8. Chargify.NET at http://chargify.codeplex.com/

If you need to do recurring billing, like I need to over at http://chatpast.com, then Chargify.NET is super handy. It's comprehensive C# API wrapper library for accessing http://chargify.com, using XML or JSON to read/write.

9. Facebook C# SDK at http://facebooksdk.codeplex.com/

Facebook is all the rage these days, and python, well... Here's a .NET SDK for Facebook. The Facebook C# SDK helps .Net developers build web, desktop, Silverlight, and Windows Phone 7 applications that integrate with Facebook.

10. OpenSearch Toolkit at http://opensearchtoolkit.codeplex.com/

OpenSearch is a great way to "light-up" the search features of your site. For example, if you visit http://chatpast.com with Firefox and pull down the search engine selector, you'll see "Add ChatPast Search". Do the same in Chrome and then type chatpast.com in the omnibox and hit tab then you're searching directly into your chat history (even when you're not on chatpast.com). So obviously OpenSearch is awesome. This toolkit makes that easier. The OpenSearch Toolkit provides ASP.NET developers with drop-in support for OpenSearch. This lets you quickly and easily publish valid search suggestions to all the major browsers.

11. NoRM at http://normproject.org/

LINQ, NoSQL and MongoDB, what more could you want? NoRM is how you make it happen. NoRM is a .Net library for connecting to the document-oriented database, MongoDB.

11.5. Simple MVVM for WPF, SL, and WP7 at http://simplemvvmtoolkit.codeplex.com/ (added after original post, hence the .5)

Simple MVVM Toolkit makes it easier to develop Silverlight, WPF and WP7 applications using the Model-View-ViewModel design pattern. The purpose of the toolkit is to provide a simple framework and set of tools for getting up to speed quickly with Silverlight applications based on the MVVM design pattern. The emphasis is on simplicity, but it should contain everything you need to implement MVVM for real-world line of business applications.


Keep in mind, that to get the latest version of any of these projects, you can just use the Visual Studio "Add Library Package Reference" dialog because of all the NuGet goodness. No need to download and manage these project references at a low level.

Enjoy!
@mkennedy

I'm working on a fantastic website that I hope will have significant impact when it's ready. I'm planning on launching in roughly one month.

I came across what I think is an awesome technique for seeing how your web page will look as you edit it. This is WAY beyond WYSIWIG:

1. Load the page you're working on in ALL the browser you care about. I'm using Chrome 4, FireFox 3.6, and IE 8.
2. If you have the monitor space, cascade these browsers side-by-side.
3. Add a meta-refresh tag to the header of that HTML file you're working on (or which consumes the CSS you're building)
   
      <meta http-equiv="refresh" content="5" />
   
   
4. Now here's the sweet part:
   
   Edit the page in Visual Studio, notepad, whatever. When you press save all browsers reload their view in a few seconds!
   
   
5. Now you get real WYSIWIG on real browsers.

That's it. The technique is totally low tech and would have worked for years. But I found it really helpful. Hope you do too.

Be sure to keep watching here. I promise a cool site will be announced soon!

Cheers!
Michael
@mkennedy

I recently wrote an article for DevelopMentor’s Developments entitled

   “Six Things That’ll Surprise You About .NET 4.0”

You can read the entire article (republished just below this introduction) or if you’d rather see it as a quick set of 6 sides, you can see those here:

     

---

Six Things That’ll Surprise You About .NET 4.0

At DevelopMentor we have been running a bunch of free webcasts. Last month it was TDD and Agile. This month we are running 4 webcasts celebrating the announcements around .NET 4.0, Visual Studio 2010, and PDC 2009.

Join me Monday, November 23rd and register here:

     http://bit.ly/aspwebforms

We’ll talk about integrating ASP.NET’s routing infrastructure into existing an ASP.NET WebForms application. This allows you to build SEO websites with URLs like

      http://dotnet.ubbuzz.com/tag/.NET+4.0

while still taking advantage of all the productivity features of WebForms such as post-backs, controls, UpdatePanel, and so on.

We have room for a couple hundred more attendees so please register and be part of the fun. I promise lots of demos and somedisdainful comments about PowerPoint!

Share it with your friends (social, virtual, real, and other types) using the widgets below!

ASP.NET MVC: What’s that, you’d rather hear about ASP.NET MVC, not this creaky old WebForms stuff? That’s Brock Allen’s talk: http://bit.ly/intromvc

WF 4: Is WF 4 and visual programming your thing? Check out Maurice de Beijer’s WF 4 talk: http://bit.ly/meetwf4

New Parallel Extensions your thing: Check out Andy Clymer’s PFX talk. (link to follow soon).

Cheers,
Michael

I'm pleased to announce that MSDN Magazine just published my Windows Workflow article entitled

    "ASP.NET WORKFLOW: Web Apps That Support Long-Running Operations"

I hope you find it useful and interesting.

I just launched the beta version of an entirely differnet kind of .NET community website:

.NET Dev Buzz
Bringing you absolutely fresh conversations about .NET and the Microsoft developer community.

http://dotnet.ubbuzz.com/

I'll follow this up with some highlights of a bunch of interesting .NET concepts being used here.

For starters, consider this. This is a website built in ASP.NET web forms, but there is no view state, no form posts, and no (visible) aspx files. It's fully RESTful.



If you like it, then

[Update: See the follow up post "Significant Advances in Unit Testing Windows Workflow"]

If you've been working with Windows Workflow, you'll find it has some cool features for orchestration, long running operations, state machines, etc.

However you won't find very much support for Test Driven Development (TDD) or unit testing in general. In fact the architecture that makes Windows Workflow powerful (strict separation of workflow, activities, and the host for example) really gets in the way of unit tests.

There has been some work done on unit testing Windows Workflows. Here's some links:

• Unit Testing Activities with Windows Workflow Foundation by Ron Jacobs
  
  
• TDD and Windows Workflow Foundation by Ron Jacobs
  
  
• WF - Unit Testing: Simple approach to Activity testing and improving the XOML loader by Andreas Erben
  
  
• Unit Testing Workflow Activities by K. Scott Allen
  
  

These are all very creative solutions. But, personally I find all of them more complex than they need to be. So in the near future I'll be putting together some libraries and samples on unit testing Windows Workflow. I think you'll find them far more powerful and at the same time simpler than anything out there.

So until I get that finished, if you have any feedback or considerations on unit testing Windows Workflow I'd love to hear it. If there are other articles I'm missing, please post them in the comments.

I think you're going to like this…

ClickOnce is a great deployment model for many Windows applications built with the .NET Framework. Too bad it isn't supported for C++, VB 6, or other technologies. Or is it...

Surprisingly, you can deploy your unmanaged apps with ClickOnce. You just need a tiny .NET app to get it started.

Here's how it works:

1. Take your existing C++ project.
   
   
2. Add to the solution a .NET console application.
   
   
3. Change the project settings on the console application to "Windows Application".
   
   
4. Write the following code for your Main method of your .NET launcher application:
   

   static void Main(string[] args)
   {
       try
       {
           Process.Start( "TheRealApp.exe" );
       }
       catch ( Exception x )
       {
           string msg = "Error launch application:\n\n" + x;
           string cap = "Error Launching Application";
           MessageBox.Show( msg, cap, MessageBoxButtons.OK, MessageBoxIcon.Error );
       }
   }
   

5. Set the build path of your C++ app to be in the main folder for your .NET application.
   
   
6. Add the C++ app and its libraries (if any) as existing items in the .NET app.
   
   
7. Change the build action to "Always Copy" as shown here:
   
   
8. Then you publish your .NET application and when it runs, TADA!, the C++ app is deployed, versioned, and kept up to date as well.
   

If you want to try it yourself, you can run this sample application here:

   Run Michael's Useless C++ App via ClickOnce...

You can also download the source.

I'm glad to confirm that the ThreadPool bug introduced in .NET 3.5 has been fixed in SP1 for .NET 3.5.

For more background information, look at these posts.

1. .NET 3.5 Brings Breaking Changes to ThreadPool
2. Breaking Changes in the ThreadPool: The Movie
3. Follow up on .NET 2.0 SP1 ThreadPool Bug
4. More on the ThreadPool Bug in .NET 2.0 SP1

Now go forth, and create some threads!

When I talk about LINQ people often ask whether it’s possible to have dynamic queries with LINQ. An example of this is presenting the user with a UI that allows them to optionally filter by some criteria or other and sort by some criteria. This was straightforward if you built-up a SQL string in code.

The compiled, static nature of LINQ makes dynamic queries appear difficult at first. They are not and this video and sample application shows you how to accomplish just this.

                        Screen shot of sample application

You can download the source code here:

           Kennedy-Dynamic-Linq.zip (88.34 KB)

And you can watch the screencast video here:

          Kennedy-Dynamic-Linq.wmv (35.7 MB)

Of course, if you like this post, please kick it:

If you'v been using my Visual Studio Recent Files Utility, then you may be interested to find out that I just released it as open source on CodePlex here.

    Visual Studio Recent Files Utility on CodePlex

Enjoy!

Here's a followup post on the .NET ThreadPool bug that I described here:

Breaking Changes in the ThreadPool: The Movie

I have been in touch with the guys who are in charge of the ThreadPool and they have both confirmed that this is a bug and that they are planning on fixing it in .NET 2.0 SP2 - but they are not sure of the timeline for its release.

In the meantime, Vance Morrison, a .NET Runtime Performance Architect at Microsoft, has given me this work-around.

Take this "broken" code:

	private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

And add a strategic Thread.Sleep and it's fixed:

	private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
		Thread.Sleep(1);
            }
        }

Well, my recent post on .NET 3.5 Brings BREAKING Changes to ThreadPool sparked quite a bit of interest in the .NET community.

But this is also something difficult to convince people of because it depends so heavily on configuration. The source code doesn't change, the environment does.

So I've put together a screencast demonstrating the problem and elaborating further. If you doubt the validity of the previous post, or can't reproduce the problem, please watch the video:




                     Download the video (approx 18 MB)

Kick it:

In the video I work with a modified version of the program. Here's that for your enjoyment:

The application: NetThreading.exe (From Video).zip (2.59 KB)

The source code:

using System;
using System.Threading;

namespace NewThreadPoolBehavior
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine( "Running on " + Environment.Version );
            int w, c;
            ThreadPool.GetMaxThreads( out w, out c );
            Console.WriteLine( "Max Currently: " + w + ", " + c );
            ThreadPool.GetMinThreads( out w, out c );
            Console.WriteLine( "Min Currently: " + w + ", " + c );

            Console.WriteLine( "Set min thread count 20? (y/n) " );
            string txt = Console.ReadLine();
            if ( txt == "y" )
            {
                Console.WriteLine( "Setting min to 20" );
                ThreadPool.SetMinThreads( 20, 100 );
                ThreadPool.GetMinThreads( out w, out c );
                Console.WriteLine( "Min Currently: " + w + ", " + c );
            }
            UseThreadPool( 200 );

            Console.ReadLine();
        }

        private static DateTime startTime;

        private static void UseThreadPool(int count)
        {
            startTime = DateTime.Now;
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

        private static int concurrent = 0;

        private static void SlowMethod()
        {
            TimeSpan dt = DateTime.Now - startTime;
            concurrent++;
            Console.WriteLine( "Starting ops (" + concurrent + " concurrent, elapsed=" +
               dt.TotalSeconds.ToString( "N3" ) + " sec.) " );
            Thread.Sleep( 20000 );
            Console.WriteLine( "Finished ops (" + concurrent + " concurrent)" );
            concurrent--;
        }
    }
}

[Note: This has been fixed in .NET 3.5 SP1, read more on this post.]

Holy smokes! I thought we had figured out something significant when I posted .NET 3.5 Brings Major (Undocumented) Changes to ThreadPool where we discovered that the .NET 3.5 ThreadPool changed the allocation algorithm for adding threads from linear to logarithmic.

This is bigger. [Recently updated see note below]

Here's the scenario. I have a server - say in the financial sector - that must process many requests and it must get up to speed immediately. I can't pay the 500 ms warm up time for the ThreadPool (.NET 2.0) or the even slower model in .NET 2.0 SP1. What do I do? I call ThreadPool.SetMinThreads(x, x) where x < (the current max), but much higher than 2 (the default). So I might call ThreadPool.SetMinThreads(100, 100) or something like that.

In .NET 1.0 - .NET 2.0 (without SP1) you would go from the warm up time model:


                 warm up time model (.NET 2.0)

To this immediate "ready" threading model.

                 ready model (.NET 2.0)

Notice that we immediately have 100 threads available in this case. BUT, and here's the but: .NET 3.5 (read .NET 2.0 SP1) *ignores* SetMinThreads. Oh, it claims to respect SetMinThreads:

   Running on 2.0.50727.1433
   Max Currently: 500, 1000
   Min Currently: 2, 2
   Set min thread count 100 (y/n)? y
   Setting min to 100
   Min Currently: 100, 100

But, look at the thread graph that results!


           ready model (.NET 2.0 SP1  -- i.e. .NET 3.5)

I don't know about you, but that doesn't look any different AT ALL than if we do not call SetMinThreads:


         warm up time model (.NET 2.0 SP1  -- i.e. .NET 3.5)

What do we conclude from this? .NET 2.0 SP1 does not respect ThreadPool.SetMinThreads. To me, that's a big breaking change. My financial app just stopped working. Yikes! So what happened to my thread pool and SetMinThreads in .NET 2.0 SP1? Anyone?

Kick it:

Don't take my word for it. Run this program on both .NET 2.0 (SP0) and .NET 2.0 SP1 and you'll see for yourself.

Here's the EXE:

   NetThreading.zip (2.3 KB)

And the source code is below:

using System;
using System.Threading;

namespace NewThreadPoolBehavior
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine( "Running on " + Environment.Version );
            int w, c;
            ThreadPool.GetMaxThreads( out w, out c );
            Console.WriteLine( "Max Currently: " + w + ", " + c );
            ThreadPool.GetMinThreads( out w, out c );
            Console.WriteLine( "Min Currently: " + w + ", " + c );

            Console.WriteLine( "Set min thread count 100? (y/n) " );
            string txt = Console.ReadLine();
            if ( txt == "y" )
            {
                Console.WriteLine( "Setting min to 100" );
                ThreadPool.SetMinThreads( 100, 100 );
                ThreadPool.GetMinThreads( out w, out c );
                Console.WriteLine( "Min Currently: " + w + ", " + c );
            }
            UseThreadPool( 200 );

            Console.ReadLine();
        }

        private static void UseThreadPool(int count)
        {
            for ( int i = 0; i < count; i++ )
            {
                ThreadPool.QueueUserWorkItem(
                    delegate { SlowMethod(); } );
            }
        }

        private static int concurrent = 0;

        private static void SlowMethod()
        {
            concurrent++;
            Console.WriteLine( "Starting ops (" + concurrent + " concurrent)" );
            Thread.Sleep( 20000 );
            Console.WriteLine( "Finished ops (" + concurrent + " concurrent)" );
            concurrent--;
        }
    }
}

[Update: 2/25/2008 - You may be thinking so what's the big deal? Most applications don't directly program against the ThreadPool so this is just some edge case. To make this more real for everyone, this applies to you if you use any of the following: ASP.NET, WCF, .NET Remoting, Delegate.BeginInvoke, SqlCommand.BeginExecute*, Windows Workflow, and more. Sounds serious now right?]

[Update: 5/12/2008 - Please see my lastest followup on this topic: More on the ThreadPool Bug in .NET 2.0 SP1.]

It was all going so smoothly. Jason Whittington, Mark Smith and I were teaching the big DevelopMentor event here in Los Angeles (Guerrilla.NET) when my presentation on the ThreadPool took a nose dive. It started with a great joke involving Wilson (the volleyball from Cast Away).

Wilson and I built an application to compute a multiplication table where each computation was (artificially) slow. To speed it up we threw it at the thread pool using delegate.BeginInvoke. We figured that the ThreadPool would allocate 25 or so threads and the table would display quickly. Here's the expected output - pretty much the same thing we've seen since about .NET 1.0:

    

Each color represents the thread that did that computation.

For the last 7 years, the behavior has been that as the ThreadPool was overloaded, it would steadily start up new threads at the rate of one every 500 milliseconds until it hits its upper limit (typically). Using Performance Monitor (perfmon) we can watch the thread pool adding threads. It usually looks something like this:

    

Much to our surprise we saw completely different behavior. The thread pool added the first 15 or so threads quickly (as expected) but then stalled. New threads were not created every 500ms, instead they were added at increasingly long intervals. My demo took almost twice as long to run as it had the last time I did this demo a few months ago.

Jason, Mark, and I took this code, ported it back to .NET 1.1 and ran it side-by-side with .NET 3.5 and here's what we saw (blue = 3.5, red = 1.1):

    

As of .NET 3.5 the upper limit of the ThreadPool was increased: Knew that.

But, it appears that v3.5 of the CLR changes the policy for adding threads to the thread pool. Rather than adding threads regularly when under load the thread pool uses a logarithmic backoff. My colleague Jason Whittington remarked that this behavior looked similar to the behavior of the thread pool in "Rotor" (the shared-source version of the CLR). We speculated that this backoff algorithm makes sense given the new 250-thread per CPU maximum - it would take a long time to reach that if the runtime waits longer and longer to start a new thread. The 250-thread limits makes it less likely that your application will deadlock the thread pool, and the exponential backoff algorithm keeps the thread pool from creating too many threads too quickly.

Why should you care? Usually you won't, but it could have dramatic impact if you count on that behavior. For example, in our contrived case, .NET 1.1 ran about twice as fast as .NET 3.5 ( ! ):

    

Here's the program and source code (trimmed down to run in both .NET 1.1 and 3.5).

    Math.zip (6.38 KB)

Needless to say, Wilson and I felt kinda stood up.

     

Please see the follow up post on Breaking changes and now there is screencast, movie version as well.

I recently wrote up an overview of the new ASP.NET MVC Framework for the Developments newsletter.

I encourage you to read it on the DevelopMentor website. It's an interesting programming model.

I've been playing with my fresh copy of Vista Ultimate - which I am surprised to find that I absolutely love.

Being a big fan of System.Transactions, I naturally wanted to use it with Vista's TxF (Transactional NTFS) file system. But unlike the data libraries, the file APIs don't auto-enlist in the transaction. In fact, there are only COM / PInvoke APIs currently.

There is a nice article about how to work with these APIs in the MSDN article: "NTFS: Enhance Your Apps With File System Transactions". But I was unimpressed with the managed wrapper they created there. In particular, I don't like that the lifetime of the file stream is not forced to be part of a client initiated transaction scope. So I built my own transactional file stream in C#. With this TxFileStream class, you can write succinct code like this:

[Test]
public void ContentAddedDoesNotPersistsAfterRollbackTest()
{
    string fileName = "file3.txt";
    string originalContents = "First contents";
   
    using (TransactionScope scope = new TransactionScope())
    {
        string newContents = "Hello transacted NTFS.";
        using (StreamWriter sw = TxFileStream.CreateWriter(fileName))
        {
            sw.Write(newContents);
        }

        using (StreamReader sr = TxFileStream.CreateReader(fileName))
        {
            string text = sr.ReadToEnd();
            Assert.That(text, Is.EqualTo(newContents));
        }
        // no call to scope.Complete() forces a rollback.
    }

    Assert.That(File.Exists(fileName));
    Assert.That(File.ReadAllText(fileName),
        Is.EqualTo(originalContents));
}

Feel free to download the code and give it a spin!

    Kennedy.TxFiles.zip (36 KB)

Of course, my library comes with comprehensive unit tests. Look here first to figure out how out use the library.

Note: I fixed a bug in creating transactional StreamWriter's in append mode. Previously they partially overwrote the existing content.

Welcome to my third Visual Studio tricks post. This time it's more of a utility, than a tip. I want to talk about managing the recently projects list. If you're like me, then you work with many different projects (especially after I teach a class) and your recent project list becomes polluted with projects you don't care about.

     


In this post, "Recent Projects in Visual Studio 2005", .net DEvHammer discusses how to access the registry to alter that list.

Well, I didn't feel like going to the registry everytime I wanted to clean that list. So I whipped up a simple UI to manage that list (basically manage that registry list).




You're welcome to download this program if it looks useful to you. I decided to publish it via ClickOnce so it will always be up-to-date. If you're using FireFox, you'll need to FFClickOnce add-on to make this work.

    Install Visual Studio Recent Files Utility (approx 200 KB)



I hope you enjoy it!

This project is now hosted on CodePlex and is Open Source.

Welcome to the next installment of my Visual Studio Tricks series. Continuing on from last time when I discussed how to quickly switch between startup projects using hot-keys, we'll cover another startup project trick.

Here's the scenario. Suppose you're working on a brand new WCF application that has both a client and server piece.

        

You need to start both the client and server to accomplish anything interesting. Most people fumble around starting the server, then the client.

Did you know that you can tell VS to run both at once. You just have to know where to look. The place to look is the properties of the solution. i.e. Right-click on the solution and choose properties and you'll get this dialog:

                                   Click for full image

By default this is set to "Single startup project". But you just have to select "Multiple startup projects" and you're on your way to smooth sailing with your client / server application. You even get debugging of both the client and server when you press F5.

Enjoy!

Welcome to my Visual Studio Tricks series. Here I’ll give you some quick tips for saving you lots of time when working with Visual Studio.

In this first installment, I'll show you how to switch between projects more quickly. Any time you work on a large project, you’ll have several projects that you might want to launch from within a solution. This includes at least an EXE and a unit test project right? Hint, hint, nudge, nudge.

Typically you do this by right-clicking on the project and say “Set as Startup Project”. But did you know you can set a hot key for this? Just go to the keyboard options in VS 2005 and type “SetAsStar” and you’ll see this screen:

If you select the startup project option, you can now add a keyboard shortcut. Sounds simple but it saves lots of time and fumbling with the mouse. There are no hotkeys assigned by default. Note that I chose <ctrl>-<shift>+P. That seems like a good choice for me.