In Part 1 of my Building a Cloud OS for .NET Developers series, I talked about setting up a pure cloud OS focused on developers. But the one crucial aspect I left out was the developer tools.

In this second installment, we’ll be covering exactly that. How do we manage having Visual Studio and associated tools and servers universally accessible in the cloud, even on mobile devices such as iPads?

We’ll start with the king of the cloud:

    Amazon Elastic Compute Cloud (Amazon EC2)
    http://aws.amazon.com/ec2/

At EC2 we can create a variety of Windows instances, get full admin access via remote desktop, and install anything we want (Visual Studio, SQL Server, MongoDB, etc). We can also run these for just 30 minutes here, 2 hours there and we’ll only be charge for the time we use it.

We are going to setup a Windows Server in EC2 which we can access from our Cloud OS, or any other machine or mobile device, and deck it out with all the developer tools we want. You have to choose your server type based on what you’re planning on doing as well as how much you want to pay. Here are the available options and pricing. You can roughly expect to pay $0.50/hr for a good workstation, or $0.12/hr for an acceptable but small one.

Amazon EC2 Instance Types: http://aws.amazon.com/ec2/instance-types/
Amazon EC2 Pricing: http://aws.amazon.com/ec2/pricing/

I’m currently using a Micro Instance for an SVN server which is always running as well as High-Memory Extra Large Instance for my development machine that I start and stop on-demand when I need to get some programming done. Here are the specs for the developer workstation:

High-Memory Extra Large Instance

17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform
I/O Performance: Moderate
API name: m2.xlarge

You have to admit, that’s a pretty killer workstation. It’s a overkill on the memory, but has better processors than the cheaper ones which also have decent memory.

The price of the dev machine is $0.69 / hour. I’ve also built some tools which help shut down the machine in case I get distracted. So far I’ve been using it pretty heavily this past few weeks and have spent probably $13 on it. That’s not bad for the possibility of being anywhere in the world (with an Internet connection) and getting my workstation running in an instant. OK, maybe it takes longer than an instant. To be fair, it usually it’s 10-15 seconds :).

Take the Free Lunch

One thing you should do straight away if you’re going to try this is sign up for the EC2 Free Tier:

AWS Free Usage Tier
http://aws.amazon.com/free/

This won’t cover the EC2 CPU time (that’s Linux only), but it will cover storage, bandwidth, S3 snapshots, etc. You also get a free Linux machine to play with if you like.

Setting up Windows in the Cloud

OK, so if you’re chosen an instance type and are ready to start, you should do the EC2 walk-through I put together here to get your machine up and running and login for the first time.

Building Windows Machines in Amazon EC2
http://www.michaelckennedy.net/blog/2010/01/31/BuildingWindowsMachinesInAmazonEC2.aspx

I’d guess that takes about 15 minutes. When you’re done, come back here...

Installing Software in EC2 and Windows

So you’ve logged in and you want to install Visual Studio, SQL Server, and other MSDN goodies. The easiest way to get them is to use the ISO DVD images from the MSDN download site. In order to mount them on your EC2 machine (which has no DVD drive), you should install

Virtual CloneDrive
http://www.slysoft.com/en/virtual-clonedrive.html

which lets you mount ISO images as if they were proper DVD drives. Then you can download and install all the MSDN files you need. A good place to start is with Visual Studio:

Microsoft Visual Studio 2010 Premium Trial - ISO
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=f81412a2-d48e-4040-9b32-27eaf771c5db&displaylang=en

One word of warning here: I’ve noticed that Chrome can freak out and fail to download very large files inside your EC2 instance. Don’t know why, but it does. So be sure to use IE or FireFox to do the downloads for the ISOs.

Syncing your Files Between your Local Machine and Dev EC2 Machine

There are two techniques we can use to sync files, depending on what category of files we’re sharing and how. For the source files, you can always setup an SVN Server (that’s what my micro instance is about). I’m a fan of Visual SVN. It’s free and easy to setup and admin.

VISUALSVN SERVER
http://www.visualsvn.com/server/

Then on your EC2 dev machine (and your others you may have) setup:

TortoiseSVN
http://tortoisesvn.net/

and

AnkhSVN for Visual Studio Integration
http://ankhsvn.open.collab.net/

Then you have super easy, reliable, and secure sharing of your source files.

    

For everything else, you can just setup a Dropbox share between your Cloud OS and you EC2 Dev Machine:


      


      

If you don’t have Dropbox and want to try it, use this link to create an account

http://db.tt/Qo7m9PA

and you’ll get me some extra free space which I always appreciate. :) Then if you have a file on your local computer you want accessible on your cloud dev machine, just drop it in your Dropbox folder on the originating computer and it’ll be there as soon as it uploads.

Waste Not, Want Not

Now here is a problem you will encounter:

You’re working on your dev machine in EC2, something comes up and you get distracted and forget about it for the rest of the day. When you come back the next day you see your EC2 system is still running and you’ve wasted $12 by leaving it running idle overnight. That sucks. If it happens on Friday, it might run the whole weekend which is even worse.

To avoid this, I wrote a little tray utility that watches for extended idle times and will shutdown your EC2 instance if it’s idle for too long.
     

Below is how you configure it. Simple, right?

    
Add it to your startup menu to make sure it starts when you log in.

     

You can download it here. I may blog about this app in greater detail later, for now feel free to use it.

Shutdown on Idle
http://www.michaelckennedy.com/Downloads/Tools/ShutdownAfterIdle.zip

Reliable Access to EC2

One annoyance in EC2 is that every time you turn off your EC2 computer, it will take on a new random public DNS name. If you really want to access to your EC2 machine from anywhere, you probably want a reliable DNS name / IP address. That’s where Elastic IPs come into play in EC2:

Feature Guide: Amazon EC2 Elastic IP Addresses
http://aws.amazon.com/articles/1346

If you create an elastic IP, then you can map it to a sub-domain in your domain and always use that. For example I could register it for cloud-developer.michaelckennedy.net. Then as long as I have a way to start my machine and re-associate the IP, all my config on my various machines (such as remembered RDP settings and passwords and such) will “stick”.

Be aware that while your EC2 machine is off, you’ll be charged a small fee (few dollars / month) for the address. You can decide whether it’s worth the trouble to save the few dollars.

Mobile / iPad Access

I’m sure when you think of accessing your EC2 dev machines, you’re thinking of doing that form Windows or maybe a Mac (using  http://www.microsoft.com/mac/remote-desktop-client). But the iPad (and even the iPhone) has apps to allow you to use Remote Desktop to EC2. For example, here’s editing the code for http://chatpast.com in Visual Studio via my iPad:


      
                                    (click for full size image)

And you can even bring up the keyboard or attach a Bluetooth keyboard.

      
                                    (click for full size image)

The app I was using is called “Jump”, and you can find it here:

Jump for iPad
http://jumpdesktop.com/

Summary

I hope that gives you some inspiration to try a new way to work in the cloud. You saw that we can create fairly serious EC2 Windows computers which can serve as developer workstations. We can share files via source control and Dropbox and we can access the computers from Windows, from Macs, and even mobile devices like an iPad.

Good luck!
@mkennedy

[Update: Read Part 2...]

Let’s talk about cloud operating systems. This article explores the options and potential of moving entirely “To The Cloud” for developers who normally demand significant offline power from their applications (IDEs, compilers, debuggers, etc). We’ll focus on .NET / Visual Studio developers, but I’m sure you can adapt this to your technology of choice.

I’ve been fascinated with what Google is doing with Chrome OS. I think finding a way to fully “live in the cloud” has a lot of promise. However, in practice Chrome OS is entirely unappealing to me. It’s just a single browser window, maximized. Yuck. I don’t mind doing most things in the browser, but I’d like multiple non-maximized browser windows and a desktop to organize things like shortcuts. That’s OK though because Chromebooks aren’t the only option. We have decent operating systems right now that can function largely in the same way if we set them up with discipline.

What we’re going to do is take a pristine Windows 7 instance and deck it out for the cloud. Even with all the choices these days, Windows 7 is still my favorite OS. I do have a Mac, several iOS devices, and I could always install Linux. Nevertheless, it’s Windows 7 for me so that’s where we’ll start.

Creating a Fresh Windows 7 Instance - Gently

I want to start from the very beginning. I don’t want old apps I’ve already installed encouraging me to “cheat” and work outside of the cloud more than necessary. At the same time I need to be able to fire up Visual Studio and do some work with an unreliable or nonexistent Internet connection So I’m keeping my current “full” Windows system intact. We’ll install a secondary instance of Windows using the awesome Boot to VHD feature in Windows 7 Ultimate. If you haven’t heard of it, I did a video walk-through here:

   Boot to VHD Screencast:
   http://www.michaelckennedy.net/blog/2009/10/20/BootToVHDScreencast.aspx

Scott Hanselman’s write up is very helpful too.

   Less Virtual, More Machine - Windows 7 and the magic of Boot to VHD:
   http://www.hanselman.com/blog/LessVirtualMoreMachineWindows7AndTheMagicOfBootToVHD.aspx

This process takes about 15 minutes. Once you’ve got everything setup, I recommend you activate and then snapshot that VHD file (from your main OS) so you can always get back to “clean”. If you’re really living the cloud lifestyle, than paving your system is much easier than before. I always keep a secondary partition around for data files anyway so that can be shared across OSes.

Now that you’ve got a truly fresh Windows 7 copy, it’s time to install just a few apps needed to set you free (and a few you’d rather not live without).

Installing “The Cloud”

While your fresh copy of Windows 7 does have IE 8 installed, that browser is entirely not up to the task of being where you spend 90% of your time. I strongly encourage you to give the latest Chrome beta a shot as your main browser. The “application shortcuts” feature of Chrome makes it much more immersive than anything IE 9 or FireFox 4 are doing (yes, I know they have pinned sites, that’s not even close).

   Chrome Beta
   http://www.google.com/intl/en/landing/chrome/beta/

Got Chrome installed and set as your default browser? Good. Don’t forget to install all your Windows updates and Windows 7 SP1 while you’re at it.

What about Visual Studio?

You may be thinking, “I *NEED* Visual Studio 2010”. Yes, we do need it for sure, but stick with me here. We are NOT installing that beast on our fresh Windows 7 system. Remember, this is your Cloud OS. I even named mine Cloud OS in my boot loader to keep that focused in my mind.

      

We’ll talk about Visual Studio soon.

Installing Your Web Applications

You may think that you do not need to install your web apps cloud apps. There are a few installers we’ll be running such as DropBox and ChatPast, but even the plain vanilla websites such as GMail are better if you install them. That’s why Chrome is way better than the other browsers.

Let’s suppose you’ll want to use Google Docs to work with documents, presentations, Excel worksheets and so on (you will want this!). Visit https://docs.google.com, then choose “Create application shortcut” in Chrome (see image):

   

This not only gives you start menu items and desktop shortcuts, but it makes your taskbar truly useful and your web apps behave as regular Windows apps.

  

In addition, you get the true feel of your websites being applications without all the browser toolbars and other junk around it. For example, here’s how this article (barely written) looks in Google Docs right now. Notice, how it looks like an app rather than a tab buried in a busy browser. It remembers its window size and position. In short, it’s more application-like.



What Cloud Apps Should I Install?

OK, so I hope I convinced you that treating your web apps as true applications is worthwhile. Now which ones do I install? Here are a few I recommend. You may have your own favorite sites. We’ll also need a few apps that do cloudy stuff but run MSI installers on our system.

- GMail: https://mail.google.com/ (obviously :) )
- Google Docs: https://docs.google.com (good Microsoft Office replacement)
- Google Calendar: https://www.google.com/calendar/
- Google Contacts: http://mail.google.com/mail/contacts/u/0/ui/ContactManager
- 37 Signals LaunchPad: https://launchpad.37signals.com (Great web apps for projects)
- Amazon Web Service: https://console.aws.amazon.com/ec2/ (You’ll see why soon)
- Twitter: http://twitter.com/
- Office Online: http://office.live.com (for when Google docs fail you)
- Pandora: http://www.pandora.com/

Here are a few proper apps that are cloud-based in some way or another that require installers.

- Skype (obviously :) http://www.skype.com
- DropBox: https://www.dropbox.com
- ChatPast Sync Client: https://chatpast.com/

By using GMail, Google Contacts, and Google Calendar, you obviously get portability from the web. But if you have a modern phone or tablet, chances are you can keep them 100% in sync using Google’s Exchange support. I do that for both my iPhone and iPad.

There are also a couple of light-weight apps that I just didn’t want to do without and still have a solid use in a Cloud OS. I needed a good image editing app (for writing blog posts like this) and one for screen captures. So we have

- Paint.NET: http://download.cnet.com/Paint-NET/3000-2192_4-10338146.html
- Window Clippings: http://www.windowclippings.com/

Finally, there are a couple that I use to just keep an eye on my system such as

- Process Explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653

Here is the full list of *everything* I have installed on my system. Just 11 applications installed. Note that I’ve hidden some that Windows installs without asking. For example, using my Microsoft Mouse forces an install of Microsoft’s Intellipoint software when I plug it in. Similarly for the touch-pad on my laptop. To me, those don’t count so I edited them out.

   

Escape from Outlook

You may be thinking that because you still use your company’s email that you’ll need to install Microsoft Office and the 800-lbs gorilla that is Outlook. Chances are you will not need Outlook. I have several “regular” email accounts I have to check that don’t have reasonable web options.

You can setup your GMail account to automatically pull from your other email accounts as if GMail itself was a POP3 client like Outlook. Just go to Gmail > Settings >  Accounts and Import > and configure “Check mail using POP3” as well as “Send mail as”. Here you see my other accounts have recently been checked and received mail.


  

Sometimes having a “Send and Receive” button for these accounts is handy. Visit the Labs section of GMail and install the “Refresh POP accounts” add-on to make the refresh button propagate a refresh off to your other accounts as well.

  
While you are in there, you should turn on the keyboard shortcuts and take 5 minutes to learn the important ones. For example, / will take you to search, i to the inbox, e archives, <shift>-c composes an email in a separate window, etc.

Where Are the Developer Tools?

You probably didn’t see Visual Studio in that list, because it wasn’t there! In the next blog post, I’ll take you through setting up an Visual Studio 2010 instance in the cloud along with all the other developer tools and servers you might need such as SQL Server, and MongoDB, and LINQPad, and so on. Stay tuned.

When I have that post online, I’ll be sure to add a link here. If you just can’t wait, here’s a hint of where we're going next.

[Update: Read Part 2 Now]

Good luck!
@mkennedy

[Note: You can download this post as a PDF.]

Here's an article about Getting Things Done and how I use Backpack from 37Signals to make it happen.

I've recently taken on some roles where I have a lot more loose-ends in my life and adopting Getting Things Done (GTD) has really helped manage everything. I'm also a huge fan of the 37Signals suite of products (Backpack, Basecamp, etc). That should be obvious from what we're doing over at ChatPast with integrating instant messaging and 37Signals applications.

So using 37Signals to implement Getting Things Done (GTD) was the obvious choice for me. After looking at both Backpack and Basecamp, it seems that Backpack is the clear winner for implementing GTD. You'll see why shortly.

There have been some articles already written about Backpack + GTD. Brett Kelly wrote a nice one over at Getting Things Done with 37signals’ Backpack – Why I’m Switching Back. However, while this was helpful in showing me that I *could* implement GTD in Backpack, it didn't show me how to do it.

How should I organize my pages? Do I use reminders, the calendar, both or neither? There will be TODO lists, but how do I correlate them to my actual work? What about check lists? Can I have templates for repeated ones? Answering these questions is the purpose of this article.

A Step-by-step guide to implementing GTD in Backpack

In this article we'll look at the major pieces of GTD and how to fit them into Backpack.

Inboxes - not just email, but all of them (there are many) Check Lists The Next Action List "Open-Loops" (Projects in GTD parlance - not to be confused with Basecamp projects) Reminders - Actions with Dates Waiting-For Lists Reference Material Taking it with you What about Basecamp?

First, A Quick Review on GTD

I'm not going to give you full summary of GTD. If you haven't started using GTD yet, I recommend that you read the book. Don't like reading? Listen to Scott Hanselman interview Lane Newsom about it or watch David Allen speak about it at Google.

Here's enough to start seeing GTD manifest itself in Backpack. First a definition, what is Getting Things Done:

GTD is a work / life management technique which encourages you to make sure you have absolutely zero open items and actions that you are tracking in your head. You get everything that you are working on or waiting for written down. This frees your mind to be fully "in the moment" and focused on what you chose to do now as well as keeps you from forgetting anything you committed to doing.

At the heart of GTD is this decision-tree model:

       
[image credit Graham King]

In our GTD in Backpack world, the 8 items ringing the workflow will mostly land in Backpack. The "Do It" one just gets done and saved nowhere - although I sometimes track these just for my personal record.

GTD in Backpack - The Basic Layout

In Backpack, I have 5 pages dedicated to GTD, which is convenient because as you see, you get 5 for free. (note: if you aren't familiar with Backpack yet, watch these short videos before going on.)

       

You can see that each page corresponded to a major component of GTD. They are further partitioned inside each page - more on that below.

1. Inboxes

With inboxes, I'm not talking about just your email inbox - although that's probably the biggest one for most people. GTD inboxes represent places that stuff that is new shows up and is waiting to be processed through the decision-tree model above.

I think you'd be surprised to see how many inboxes you truly have if you took a full account of them. GTD encourages you to do this for all sorts of things - take a full account and get them out of your mind.

That's one of the great uses of check lists in GTD + Backpack.

2. Check Lists

A check list is any set of steps you need to remember. Sometimes it's for one-off situations, others it's repeating. Processing all the new stuff in your varied inboxes is a repetitive action. So I made a check list for that. Check lists, as you can see below, are TODO lists in backpack. When they are repeated, I make a template like the actual list but with the title prefix [template]. Then I make a copy to "do" the check list each time.

Here is the check list I use several times a week to make sure that I don't forget to "process" one of my inboxes, which then feeds the other 8 types of containers ringing the workflow diagram.

       

You would see a very similar looking TODO list at the bottom of the [GTD] Check Lists page with the title '[template] Processing "in" Checklist'.

3. Next Action Lists

The Next Action Lists are pretty straight forward. As you saw above, there is a page entitled "[GTD] Next Actions". Here I track my next action for each item or project I have open. Again the page is partition internally:

       

Notice that I have a personal and business one (actually several). One rule of GTD is that if your home life is a mess, it'll affect your business life and visa versa. So manage it all here. Note also that you can add comments and attach files to the items to track additional info you might need.

If the next action came from an ongoing project (see below), then I've tried to put the name of the project as a prefix, for example House: in my personal items.

Finally, you'll see that I don't use the GTD contexts as most people might. In my world there is so much overlap for the contexts that they become meaningless. Traditional contexts might include calls, at computer, errands, at office, at home, agendas, read / review. However, I work at home with internet always on and a computer at my desk. I also do reading here. So those contexts basically all mean the same thing and thus I don't use them.

4. Projects

As you saw in the basic structure, projects get their own page (internally partitioned). Thus I have a page entitled '[GTD] Projects'. This one is pretty simple. I keep track of all the 'open-loops' that require 2 or more actions to complete. Each one gets its own TODO list within Backpack and those lists are partitioned by work and personal.

       

5. Reminders

By far, the biggest GTD "bang for the buck" that you get from Backpack come in the form of reminders. However, their effect takes some exploring to see their full potential. First, let's look at the reminders I have upcoming. Usually I have more than this, but they seem to have thinned out for the weekend.

       

In my reminders, you'll see I need to finalize the home inspection details for a house I'm buying on Feb 4th. By putting it into my reminders, I can rest assured that I'll remember it on the day that I can do something about it.

What about calendars? I make extremely heavy use of calendars for GTD. But not within Backpack. I use Google calendar, which then I sync to Outlook, my iPhone, my iPad, and I can also access via the web. If this had been an appointment, I would have skipped the reminders and just put it on my calendar. But it's an action with a date, so it goes in the reminders.

Getting Notified of Reminders

The first place to leverage reminders is to have them show up as text messages on your phone. To do this, go to "My Info" in backpack, in the section "Mobile phone reminders", check the checkbox for "Send text message reminders to my mobile phone".

Next, you'll want to see these reminders in your calendar (but not as appointments!). That's where the crazy URL starting with iCalendar: webcal://domain.backpackit.com/ical/... comes in. You can subscribe to this in Outlook or, as in my case, in Google calendar.

In my calendar, then have all these actions + dates show up alongside my day.

       

I think I had to replace 'webcal:' with 'http:' in order to add this to Google Calendar though.

6. Waiting-For Lists

Waiting lists are for situations where I know I'll need to follow up on something but no necessarily by any particular date - in that case a reminder doesn't make sense for me. But several times a week, I'll scan the list and close those loops if they other person hasn't responded by then. Here's how that looks in Backpack:

       

Note that the dates I manually typed in are not the due dates. They are the creation dates (Backpack doesn't track this). So I know how long I've been waiting.

7. Reference Material

One of the huge advantages that Basecamp has over other systems that are just simple TODO lists is the ability to attach rich reference material and additional information to the TODO item that represents a next action, or waiting item, and so on.

       

You can add comments to any TODO item. For basic information, this can be simple text. For example, if you needed to renew your insurance, you could attach the phone number and policy number to the TODO item so you don't have to look it up.

For non-textual information, you can also attach files (pictures, word docs, whatever) to the TODO item. Note that file attachments are a paid feature, so you can't use this option if you're on the free version of Backpack.

8. Taking it with you

One of the key tenants of GTD is that you should have your info with you nearly all the time. You might be thinking that using Backpack means you will only be able to get to it when you're on the web.

If you have an iPhone, iPad (or even iPod touch), you can always have an offline copy available using an app called Satchel found in the Extras section.

Here's a few screenshots from the iPhone:

Here's a few screenshots from the iPad (click to enlarge):

       
 

       
 

       

9. What about Basecamp?

Some projects are either shared by a team of people or are truly massive with many steps, artifacts, milestones, and files. These projects are better managed outside of Backpack. In that case I use the paid version of Basecamp and simply add a note to the project entry page as you can see here.

       

Conclusion

If you've been thinking of using Backpack for GTD, I hope this has given you something to work with. I certainly don't promise to be a GTD expert. But this setup works well for me and I think it will for you too.

Thanks!
@mkennedy

So maybe I'm late to the party, but I recently started playing with NuGet. It's a killer new way to find, install, maintain, and manage references to open source libraries in Visual Studio 2010. Plenty of people have written about it (Phil Haack and Scott Hanselman for example). Let's just say you should learn about NuGet if you don't know it already.

What I want to talk about is all the cool open source projects I found just by flipping through the pages of the NuGet directory in the Visual Studio "Add Library Package Reference" dialog.

1. RazorEngine at http://razorengine.codeplex.com/

RazorEngine is templating engine built upon Microsoft's Razor parsing technology. The Razor Templating Engine allows you to use Razor syntax to build robust templates. No need to learn a custom clunky API for generating things like HTML and emails and so on. Just use the hot new @Razor syntax from ASP.NET MVC 3.

2. YUI Compressor for .Net at http://yuicompressor.codeplex.com/

YUI Compressor for .Net is is a .NET port of the Yahoo! UI Library's YUI Compressor Java project. Do you have a bunch of CSS and JavaScript files and you want your page to load faster. This is a great way to do it from ASP.NET.

3. 51degrees.mobi at http://51degrees.codeplex.com/

Want to build an ASP.NET MVC website that has both a desktop and mobile version from the same project? 51degrees.mobi Foundation is an ASP.NET open source module which detects mobile devices and browsers, enhancing the information available to ASP.NET. Mobile handsets can optionally be redirected to a home page designed for mobile phones. Smart phone and feature phones are all supported.

4. Lucene.Net at http://lucene.apache.org/lucene.net/

Lucene.Net is a source code, class-per-class, API-per-API and algorithmatic port of the Java Lucene search engine to the C# and .NET platform utilizing Microsoft .NET Framework. Want indexed full-text search from .NET? Here you go.

5. MvcMailer at http://www.codeproject.com/KB/aspnet/MvcMailerNuGet.aspx

Send a professional looking HTML email from your ASP.NET MVC simply by pointing at a particular view.

6. DotNetZip at http://dotnetzip.codeplex.com/

DotNetZip is an easy-to-use, FAST, FREE class library and toolset for manipulating zip files or folders. Zip and Unzip is easy: with DotNetZip, .NET applications written in VB, C# - any .NET language - can easily create, read, extract, or update zip files. For Mono or MS .NET.

7. FileDB at http://filedb.codeplex.com/

Ever wanted to have a single, simple file that you can access as a basic database without the fuss? FileDB is a free, fast, lightweight C# (v3.5) DLL project to store, retrieve and delete files using a single archive file as a container on disk. It's ideal for storing files (all kind, all sizes) without databases and keeping them organized on a single disk file.

8. Chargify.NET at http://chargify.codeplex.com/

If you need to do recurring billing, like I need to over at http://chatpast.com, then Chargify.NET is super handy. It's comprehensive C# API wrapper library for accessing http://chargify.com, using XML or JSON to read/write.

9. Facebook C# SDK at http://facebooksdk.codeplex.com/

Facebook is all the rage these days, and python, well... Here's a .NET SDK for Facebook. The Facebook C# SDK helps .Net developers build web, desktop, Silverlight, and Windows Phone 7 applications that integrate with Facebook.

10. OpenSearch Toolkit at http://opensearchtoolkit.codeplex.com/

OpenSearch is a great way to "light-up" the search features of your site. For example, if you visit http://chatpast.com with Firefox and pull down the search engine selector, you'll see "Add ChatPast Search". Do the same in Chrome and then type chatpast.com in the omnibox and hit tab then you're searching directly into your chat history (even when you're not on chatpast.com). So obviously OpenSearch is awesome. This toolkit makes that easier. The OpenSearch Toolkit provides ASP.NET developers with drop-in support for OpenSearch. This lets you quickly and easily publish valid search suggestions to all the major browsers.

11. NoRM at http://normproject.org/

LINQ, NoSQL and MongoDB, what more could you want? NoRM is how you make it happen. NoRM is a .Net library for connecting to the document-oriented database, MongoDB.

11.5. Simple MVVM for WPF, SL, and WP7 at http://simplemvvmtoolkit.codeplex.com/ (added after original post, hence the .5)

Simple MVVM Toolkit makes it easier to develop Silverlight, WPF and WP7 applications using the Model-View-ViewModel design pattern. The purpose of the toolkit is to provide a simple framework and set of tools for getting up to speed quickly with Silverlight applications based on the MVVM design pattern. The emphasis is on simplicity, but it should contain everything you need to implement MVVM for real-world line of business applications.


Keep in mind, that to get the latest version of any of these projects, you can just use the Visual Studio "Add Library Package Reference" dialog because of all the NuGet goodness. No need to download and manage these project references at a low level.

Enjoy!
@mkennedy

This article is a follow up one I wrote last week entitled “The NoSQL Movement, LINQ, and MongoDB - Oh My!”. In that article I introduced the NoSQL movement, MongoDB, and showed you how to program against it in .NET using LINQ and NoRM.

I highlighted two cornerstone reasons why you might ditch your SQL Server for the NoSQL world of MongoDB. Those were

1. Ease-of-use and deployment
2. Performance

For ease-of-use, you’ll want to read the original article.

This article is about the performance argument for MongoDB over SQL Server (or MySql or Oracle). In the first article, I threw out a potentially controversial graph showing MongoDB performing 100 *times* better than SQL Server for inserts.

“A potentially controversial graph showing MongoDB performing 100 times better than SQL Server”

We’ll see source code, downloadable and executable examples and you can verify all of this for yourselves. But first, here’s a new twist on an old proverb:

“Data is money”

If your application is data intensive and stores lots of data, queries lots of data, and generally lives and breathes by its data, then you’d better do that efficiently or have resources (i.e. money) to burn.

Let’s imagine you’re creating a website that is for-pay and data intensive. If you were to attempt to plan out your operating costs per user to help guide the pricing of your product then the cost of storing, querying, and managing your data will likely be a significant part of that calculation.

If there is a database that is 100 times faster than SQL Server, free, easy to administer and you program it with LINQ just as you would with SQL Server then that is a very compelling choice.

When you have such a database, it means you can run your system on commodity hardware rather than high-end servers. It means you can have fewer servers to maintain and purchase or lease. It means you can charge a lot less per user of your application and get the same revenue. Think about it.

“It means you can charge a lot less per user of your application and get the same revenue. Think about it.”

One more story before we see the statistics. Kristina Chodorow from 10Gen gave a talk a few weeks ago at San Francisco’s MySQL Meetup entitled “Dropping ACID with MongoDB”. You can watch the recording here:

http://www.ustream.tv/recorded/6146875

[The audio and video isn’t too hot, but the content is. Skip the first minute without audio.]

During this talk, Kristina describes SourceForge’s experience moving from MySql to MongoDB. On MySql, SourceForge was reaching its limits of performance at its current user load. Using some of the easy scale-out options in MongoDB, they fully replaced MySQL and found MongoDB could handle the current user load easily. In fact, after some testing, they found their site can now handle 100 times the number of users it currently supports.

Not convinced of this NoSQL thing yet? Fair enough. Here are some graphs, some stats, and some code.

The scenario:

Model a data intensive web application aiming to support as many concurrent users as possible. There will be users from the web application itself. But there will also be users from an API and external applications. Users will interact with the data by having nearly as many inserts as they do queries. Their inserts are all small pieces of data and are all independent of each other.

Let me just get this out of the way and I mean the following in the nicest of ways: I don’t care about your scenario or use-case. The scenario above is what I’m trying to model. I’m not trying to do bulk-inserts or loading large files into databases or anything like that. MongoDB may be great for these. SQL Server may have specialized features around your use-case, etc. They don’t apply in my scenario. So please don’t wonder why I’m not using bulk inserts or anything like that in the examples below.

Insert Speed Comparison

It’s the inserts where the differences are most obvious between MongoDB and SQL Server.

These inserts were performed by inserting 50,000 independent objects using NoRM for MongoDB and LINQ to SQL for SQL Server 2008. Here are the data models:

MongoDB basic class

SQL Server basic class

I ran five concurrent clients hammering the databases with inserts. Here’s the screenshots for running against MongoDB and against SQL Server. Let’s zoom into the most important result with the output from one of five concurrent clients:

MongoDB:

SQL Server:

That’s right. It’s 2 seconds verses 3 1/2 minutes!

Now to be fair, this was using LINQ to SQL on the SQL side which is slow on the inserts. After discussing these results with some friends, I re-ran the tests using raw ADO.NET style programming and saw a 1.5x-3x performance improvement for SQL. That still leaves MongoDB 30x-50x faster than SQL.

Query Speed Comparison

Now let’s see about getting the data out using the same objects above on the indexed Id field for each database.

Here MongoDB still kicks some SQL butt with almost 3x performance. If we were to leverage the mad scale-out options that MongoDB affords then we could kick that up to many times more.

“If we were to leverage the mad scale-out options that MongoDB affords then we could kick that up to many times more.”

Complex Data and the Real World

Feel like that was an overly simplified example? Here’s some real world data with foreign keys and joins. Below is the complex data model.

MongoDB:

SQL Server:

It shouldn’t surprise you that MongoDB does even better here without its joins.

The Hardware

All of these tests were run on a Lenovo T61 on Windows 7 64-bit with a dual-core 2.8 GHz processor using the 64-bit versions of both SQL Server 2008 Standard and MongoDB 1.4.1. You can even see a picture of the computer here: http://twitpic.com/hywa8

Your Turn

If you want to see the entire set of data above as an Excel spreadsheet, you can download that here:

http://www.michaelckennedy.com/Downloads/sql-vs-mongo.xlsx

You can also download the sample code. Before you do, realize I haven’t done a bunch of work to make it super easy to run. But you should be able to figure it out. Just turn the knobs on the PerfConstants class for the number of inserts and queries. Then comment or uncomment sections of the code in the clients for your scenarios.

The expected use is that you’ll start the launcher application then use it to launch five concurrent clients at exactly the same time.

Download Sample:

http://www.michaelckennedy.com/Samples/SpeedOfSqlVsMongoDBAnddotNetSample.zip

Got feedback? Write a comment or contact me on Twitter: @mkennedy or find me in any of these other ways.

Thanks!

Some thanks are in order for all the help I got bouncing around ideas as well as trying different scenarios. Thanks to

Eric Cain @arcain
Jim Lehmer@dullroar
Karl Seguin @karlseguin

In this article I'm going to give you a simple, step-by-step overview of how to create a Windows 2008 server image in Amazon's Elastic Cloud Compute (EC2) infrastructure. Now I must admit I'd rather have found a good tutorial on The Internets or even in a book. Feel free to send me any I missed. My experience is they are either dated or about Linux and so on...

First, briefly why does one care about EC2? Well maybe you are buying into the whole cloud computing story which lets you cheaply out-source your computer hardware for amazingly cheap prices (staring around $0.20 / hour for a dedicated machine). That's a great reason and Microsoft and Google have interesting plays there too.

Personally I just want a simpler way to create virtual machines. We'll have full admin access over remote desktop to our system to install whatever we want. I'm putting Visual Studio 2010 Beta on mine to play around with that software without 'polluting' my real system.

Here we go. If you don't delay I suspect this would take you about 20 minutes from start to login! Subsequent virtual machines are much faster to create and launch because the can be based on pre-configured images.

1. Create an Account

Register for an Amazon Web Services account at http://aws.amazon.com/.

2. Enable EC2 Features

Enable Elastic Compute Cloud for your AWS account at http://aws.amazon.com/ec2/.

3. Launch a New Instance

Use the AWS Management Console to launch and manage your virtual images: http://aws.amazon.com/ec2/home. As the console says, choose "Launch Instance" under the "Getting Started" section. You will be presented with a list of pre-configured images. We'll start with a stock Amazon Windows 2008 server image.

4. Choose a Base Image

Now you'll be presented with a list of pre-configured virtual disk images. This time we'll setup a 64-bit Windows 2008 Server (Data Center Edition). Just choose "select" out of the list below:

5. Use the Request Instances Wizard

Use the Request Instances Wizard to configure the newly created instance which includes configuring the security, choosing an encryption key, opening ports in the firewall, and kicking off the new instance. Below you'll see the encryption key step - be sure to download the key pair as you'll need it for retrieving the administrator password.

6. Launch!

Here's what you can expect for the review screen of the Request Instances Wizard. Press launch and you're almost there.

7. Launching... (AKA Wait 5 Minutes)

After you launch you're instance you'll get a confirmation screen to show you it's being prepared and allow you to configure durable storage and IP addresses (both entirely optional).

8. Back to the Management Console

Now if you choose "View your instances..." you'll see that your instance is being prepared - it has a yellow pending status. This screen doesn't always refresh on its own so use the refresh button in the upper right of the console (rather than your browser's refresh button).

9. Running!

After a few minutes your instance with the yellow icon will turn green and be in the running state. Note that at first this really means booting up so you can't get to it right away. Give it another minute or two...

10. Login Part 1: Getting the Credentials

Now you'll want to login. Of course, the system was created with an administrator account which has a strong password. You'll need to retrieve that password using the "Instance Actions -> Get Windows Admin Password" option.

11. Login Part 2: A Little Hasty

You're probably excited to get this thing running and if you try right away you'll get another message telling you to be patient and try again in a few minutes. Just keep trying.

12. Login Part 3: Using Private Key

Eventual the new system is up and running and you can get the password. The first step here is to pass in your encryption key from the wizard step before.

13. Login Part 4: Administrator Account and Password

Pass in the encryption keys and you'll see the username and password (don't get excited, I already changed the password!).

14. Login Part 5: Finding the Machine Address

When your instance starts, it'll be given an Internet visible DNS name that you can use to connect via Remote Desktop. You'll find it in several places. One of them is highlighted below. Note that this address changes as you start and stop your instance.

15. Connected!

Now just fire up Remote Desktop, use the Administrator account and password from step 13 to log in. Now you have full access to your Windows 2008 machine. You can do with it what you will, install software, start serving web pages, etc.

16. A Word of Caution

If your intent is to run a web server, then let it run. But if you are just using this for your own purposes and don't need it when you're not logged in to the machine, be sure to return to the Management Console and stop the instance. You can alternatively do that by choosing "Shutdown" instead of logging out of your Remote Desktop instance.

I hope you found this walk-through helpful. I just learned most of this myself so I figured I'd blog it and everyone can learn from it.

Cheers!
Michael

Rob Barry and Jack Vaughan interviewed me for their article on SearchCloudComputing.com entitled

   Azure cloud on horizon: The devil is in the data architecture details

Here's a small excerpt. If you're interested in Windows Azure and Cloud Computing, read on...

Microsoft did a good job when they designed Azure, according to Kennedy. "The company encourages you to build scalable reliable systems by basically making it really hard to do the stuff that makes systems unreliable," he said.

There are many developers curious about cloud computing, but most are being rather cautious. Directions on Microsoft's Sanfilippo said he's talked to more developers that are concerned about building on top of their existing work than re-coding everything to work in the cloud.

"There's still an education bit that has to happen about what kind of applications are appropriate for Azure. But I think there's a lot of curiosity about Azure," Kennedy said.

Still, he continued, "I don't know many projects that are betting the bank on Azure yet."

Note: This is a little dated as it was publish in July 2009 - some how I missed the original publication - but it's still an interesting read.

Thanks Rob and Jack for the article and conversation.

I recently wrote another article for DevelopMentor's Developments newsletter (not subscribed yet? see top-right of this page). This one is entitled

   10 Features in .NET 4.0 that made Me Smile

Read it on the DevelopMentor website: http://www.develop.com/tenfeaturesdotnet4

I am republishing it below for you all to enjoy on your RSS readers.

Cheers,
Michael

---

10 Features in .NET 4.0 that made Me Smile

I have been reviewing some of our upcoming classes at DevelopMentor this week. One of those classes, What’s New in .NET 4.0, left me excited for things to come. There are a bunch of small but wonderful features discussed in that class. I thought I’d take this opportunity to write a few of them up and share the joy. I bet some of them make you smile too.

1. The Parallel Extensions for The .NET Framework will be built into mscorlib.dll.
   
   The fact that PFx will be part of the core .NET library says a lot about how much faith and support it’s getting within Microsoft. BTW, here are some really great demos for PFx in .NET 4.0.
   
   
2. PFx introduces a new threading construct: Barrier<T>.
   
   Barrier lets you define rendezvous points in your code where multiple concurrent operations can automatically sync-up. Here’s an example.
   
   
3. Code contracts.
   
   Code contracts allow you to assert truths about your code as if you are writing a unit test. But these assertions live within your production code and are both verified by the compiler as well as the runtime. Here’s the original research project that lead to this feature on Microsoft Research.
   
   
4. The WPF and Silverlight designers mostly work.
   
   Now this shouldn’t be a point to make me smile or get excited about, but it is. The pain and suffering around the Visual Studio support for WPF and Silverlight designers has been so bad that a mostly-working, and sometimes truly innovative design-time experience within Visual Studio gives me real hope for these technologies. I’m actually excited about them now.
   
   
5. Support for the MVVM pattern across both WPF and Silverlight.
   
   Speaking of that XAML stuff, if you write WPF or Silverlight code and don’t know MVVM, stop reading this article and learn about it here. I’ll wait. Ok, now you too should be excited to hear that there is improved support for MVVM across Silverlight and WPF in a unified way. Smiles baby!
   
   
6. WF (Windows Workflow Foundation) has an AsyncCodeActivity class.
   
   While WF has traditionally supported both synchronous (activities that execute immediately) and asynchronous activities (for long running senarios where the workflow becomes idle and is unloaded from memory [click here for more details]), there has been an unserved middle ground. If you want to use threading in your activity and allow the workflow to go idle without it being unloaded from memory you were basically out of luck. This is the problem solved by the AsyncCodeActivity. WF 4 now has a class which has a BeginExecute / EndExecute pair of methods which much more closely models the regular .NET async design patterns.
   
   
7. WF has a rehostable designer (really, they mean it this time).
   
   There are some great uses for giving regular users a WF designer experience with the right granularity of activities. Now it’s much easier. Here’s an app that rehosts the designer:
   
   
   
   
8.  Configuration-free WCF Hosting.
   
   Hosting WCF services is now like hosting ASMX web services if you like the defaults. Just throw out a service + contract + address and it’s up and running. That’ll save a bunch of <system.serviceModel> configuration goo. Smiles!
   
   
9. No more *.svc in our RESTful urls in IIS.
   
   With the ASP.NET routing framework and WCF REST introduced in .NET 3.5, we can create beautiful, expressive Uri’s for our websites. For example:
   
      http://lookatthiswith.me/watch/intro
   
   But this falls apart with WCF REST when we host it in IIS. Our service Uri’s look like this:
   
      http://lookatthiswith.me/services/lookieservice.svc/lookup/json/cf7
   
   And now we have this ugly .svc part-way through our Uri! Ick. Well, in .NET 4 that Uri is much more customizable and the .svc is gone. Smiles!
   
   
10. ASP.NET MVC has wicked JavaScript support.
    
    JQuery is there by default. That’s awesome. But there is also a class similar to the Html class (for HTML helpers) called Ajax. This static class has functions like Ajax.ActionLink and effectively brings the functionality of UpdatePanel to MVC!

Well there you have it. 10 awesome things in .NET 4 that made me smile this week. I hope you find some to be welcome additions yourself! If you want to learn more about .NET 4.0, check out our recorded webcasts here: http://www.develop.com/dotnet4webcasts. Also have a look at my article from last month Six Things That’ll Surprise You About .NET 4.0. Finally, if you have some training funds laying around, I’d love to spend a week talking about these ideas with you in our What’s New in .NET 4.0 What’s New in .NET 4.0 class.

Michael Kennedy is an instructor for DevelopMentor where he specializes in core .NET technologies as well as agile and TDD development methodologies. Keep up with Michael via his Web site and blog at http://www.michaelckennedy.net or on Twitter: @mkennedy.

I recently wrote an article for DevelopMentor’s Developments entitled

   “Six Things That’ll Surprise You About .NET 4.0”

You can read the entire article (republished just below this introduction) or if you’d rather see it as a quick set of 6 sides, you can see those here:

     

---

Six Things That’ll Surprise You About .NET 4.0

I recently wrote an article for DevelopMentor's Developments newsletter entitled Building a Twitter Application in .NET.

You can read it at the DevelopMentor website:

   http://www.develop.com/twitternetapps

I've republished here for my readers. Enjoy!

---

Building a Twitter Application in .NET

by Michael Kennedy (@mkennedy)
http://www.michaelckennedy.net

Twitter has become one of the web's hottest properties. It is a central part of mainstream news programs such as CNN's Anderson Cooper 360, congressional debates, and talk shows. In fact, it grew at a rate of 1400% this past year [bit.ly/jG9BG].

If your company wants to interact with your customers in a modern and engaging experience, you need to be on Twitter. In fact, if you have customers that really like or dislike you, they are probably talking about you on Twitter. You should be part of that conversation.

In this article, we will explore how to build a rich interactive experience on Twitter that goes beyond just creating a new Twitter account. We will build a .NET application that uses the Twitter API (a free service) alongside other cool technologies such as the WCF REST Starter Kit [http://bit.ly/v8mBb] and LINQ to fully leverage the Twitter experience.

Whether you want to build a community around your brand or you want to build the next real-time, social community website like .NET Developer Buzz [http://dotnet.ubbuzz.com/], this article will cover the technologies required to get the job done.

If you want to download the sample application to follow along, you can get it here:
   http://www.michaelckennedy.com/Samples/SampleStatusUpdater.zip

While you're at it, be sure to follow DevelopMentor on Twitter:
    http://twitter.com/DevelopMentor_

Let's Start Small

I will show you how to fully leverage the Twitter API, but many tasks can be accomplished using simpler tools and you may be better starting there. Let's look at a few things that we can do without the Twitter API.

Case 1. You want to display your latest Twitter messages on your website.

Your tweets[1], as they're called, can be consumed as a simple RSS feed. So you may want to simply pull this feed into your website rather than digging into the Twitter API and consuming custom XML or JSON formats. An example of consuming Twitter in this fashion can be found on my website's front page [http://www.michaelckennedy.net/].

To get your RSS feed, just visit your profile page and get the RSS tweet link, e.g. "RSS feed of mkennedy's tweets" [http://bit.ly/guhZU]

[1] Tweet - these are what the individual messages sent on Twitter are called. If the name sounds weird, I'm sure you'll get used to it. Remember that there was a time when Google was just a noun.

Case 2. You want to watch and manage multiple accounts at the same time.

Most Twitter clients only support a single user. But there are a couple of good tools that allow you to manage multiple accounts. My current favorite is one called bDule and you can get for free at http://www.sobees.com/bdule.

Beyond the Small

There are times when you want to do more than simply syndicate your Twitter stream. Let me give you an example. At DevelopMentor, we have had many instructors on Twitter talking about their own interests. But we didn't have a corporate Twitter presence. We decided to create our corporate presence by pulling all our instructors individual tweets and rebroadcasting them from our DevelopMentor Twitter account: @DevelopMentor_

We wanted to keep a sense of the original instructor who wrote the message, so we append on an attribution. For example:

"Software Transactional Memory is released! (via @mkennedy)

and we wanted to do this in a flexible way. In short, we needed more functionality than Twitter provides.

There are actually three services that do this sort of thing and they looked promising.

   http://www.connecttweet.com/
   http://www.grouptweet.com/
   http://cotweet.com/

But in the end, nothing completely matched our requirements. So we decided to write our own application to publish everyone's tweets under the DevelopMentor banner. There are few simple steps involved as well as a lot of details we won't go into yet.

1.  Take a list of Twitter accounts and download everyone's statuses.
2. Determine which messages we haven't seen before.
3.  Publish these new statuses under our corporate account.
   

We can actually implement a simplistic version of this by continuing to use the public RSS feeds of the individual accounts in conjunction with a very handy Twitter API wrapper called TwitterooCore which you can find at http://rareedge.com/twitteroo/blog/.

The Twitter API

In practice, there is simply data missing from the RSS feed that we require as well as features missing from the Twitteroo Core that move us deeper into the Twitter API. One thing you may well miss is the ability to tie together conversations. For example if Bob says "hello" and Jerry says in reply "@bob Back at you!" Twitter tracks that Jerry replied to Bob and publishes this link in the stream. To get access to these types of features and many other optimizations, you'll need to use the REST-based Twitter API.

The Twitter API is documented at http://apiwiki.Twitter.com/

There you can do things like get the users tweets, if their tweets are public, by requesting:

http://twitter.com/statuses/user_timeline.xml?screen_name=mkennedy

What you get back to dependent on the requested format. Here we're asking for XML (user_timeline.xml) but we can also get JSON, RSS, or ATOM.

Similarly, we can update our status by making a POST request to:

http://twitter.com/statuses/update.xml

Here again we have the four possible formats: XML, JSON, RSS, ATOM. However this time we're using a form post for the request to update the user status which is then in-turn returned as XML.

Great, we have this cool REST API based on loosely-typed GETs and POSTs. Should we program against it using fundamental .NET types such as WebRequest, WebClient, and similar classes? We could. But WebRequest is so .NET 1.0 (circa 2001). There is a much newer API on the verge of release from Microsoft: The REST Starter Kit [http://bit.ly/1IF3Ji].

While this toolkit is generally geared towards building to RESTful WCF services, there's also a great set of classes for building REST clients. We will use these classes to write our application. Let's take a look at how we can use the Twitter API to write are simple application.

The REST Starter Kit

Part 1 - Getting the users of tweets.

We need to download the users messages as XML and convert them to .NET objects that we can consume our application. This is really straightforward because the REST Starter Kit as a cool feature to Visual Studio: "Paste XML as Types". This feature will take an XML file and auto-generate types based on the inferred XML schema. In our case, the XML file we will use is returned from the user timeline.

http://twitter.com/statuses/user_timeline.xml?screen_name=mkennedy

Open a link in a web browser, choose view source, and copy some XML. Then go to Visual Studio, choose edit, Paste XML as Types. Of course you must have the REST Starter Kit installed for this to work.[2]

[2] In my experience I ran into some errors deserializing the response from Twitter. See the The Real World Intrudes section at the end of the article if you run into difficulties.

After generating our status related types, we can use the HttpClient class to download the statuses.

Listing 1.

private static void GetStatuses(IEnumerable userNames)
{
    XmlSerializer serl = new XmlSerializer(typeof (statuses));
    serl.UnknownElement += delegate { };

    foreach (string name in userNames)
    {
        string url = string.Format(
            "http://twitter.com/statuses/user_timeline.xml?screen_name={0}",
            name);

        HttpClient client = new HttpClient(url);

        HttpResponseMessage response = client.Get();
        response.EnsureStatusIsSuccessful();

        string contents = response.Content.ReadAsString();
        Stream stream = new MemoryStream(Encoding.UTF8.GetBytes(contents));
        statuses userStatuses = (statuses) serl.Deserialize(stream);

        if ( userStatuses.status.Length > 0 )
        {
            Console.WriteLine( "@{0}'s latest tweet: {1}", name,userStatuses.status[0].text );
        }
        Console.WriteLine();
    }
}

There are two main things happening in this code. We're using the HttpClient class to download the web content associated with the given user's timeline. Then we're using the XmlSerializer in conjunction with the auto-generated XML-serializable types from the "Paste XML as Types" command. That's all there is to it. The REST Starter Kit does most of the work for us.

Part 2 - Checking for New Messages

Now that we have all the statuses, we need to find the ones that we haven't broadcasted from our main account and send them along to step 3. We won't go into detail on how to track that. But you can imagine a simple database that facilitates a LINQ to SQL query like this:

Listing 2.

public StatusUpdate[] FindRebroadcastableStatuses(StatusUpdate[] updates)
{
    return
        (from up in updates
         let neverPosted = up.User.LastPostBroadcasted == null
         let afterPostDate = up.Time > up.User.LastPostBroadcasted
         where neverPosted || afterPostDate
         select up).ToArray();
}

Part 3 - Publishing the Statuses

Now that we've gathered the statuses of our various users, it's time to rebroadcast them to our community. We're going back to the Twitter API to update our status. Again will use the HttpClient class and following the RESTful principle of using POST to add new items to a given URI. We will do an HTTP POST to our status to add a new message to the account.

Listing 3.

private void UpdateStatus(string newStatus, long? replyToId)
{
    string url = "https://twitter.com/statuses/update.xml";

    HttpClient client = new HttpClient( url );
    client.TransportSettings.Credentials = 
        new NetworkCredential( twitterUser, password );

    HttpUrlEncodedForm form = new HttpUrlEncodedForm();
    form.Add( "status", newStatus );
    if ( replyToId != null )
    {
        form.Add( "in_reply_to_status_id", replyToId.ToString() );
    }

    HttpContent content = form.CreateHttpContent();

    HttpResponseMessage message = client.Post( "", content );
    message.EnsureStatusIsSuccessful();
}

This time we create an HTML form using the HttpUrlEncodedForm class. We set the status field to our new status. If this status as a response to a previous tweet then we add the in_reply_to_status_id so Twitter knows to add a "in reply to..." tag to the tweet.

And there you have it. Working with the Twitter REST API is really quite straightforward. If you use the REST Starter Kit it's downright easy.

The Real World Intrudes

But wait a minute. This is reality and building bulletproof applications is never that simple. There are at least five significant gotchas you have to address in practice when working with Twitter and the REST Starter Kit.

1. There will be times when Twitter is unavailable. You have to be ready for crashes and other types of randomness.

Twitter is one of those sites that can barely handle the traffic it is receiving. With its 1400% growth, this isn't getting much better. So you must program defensively and assume that many of your API requests will fail.

2. The Twitter API is a free service. By default, you are limited to a small number of requests per hour. Many of the limits are around 150 API calls / hour. You may need to carefully design your application to work within the limits. Some applications simply need more data than this permits. For example, .NET Dev Buzz [http://dotnet.ubbuzz.com/] has to track thousands of users. In that case, you will have to get your application white listed with Twitter. You can do that here:

http://twitter.com/help/request_whitelisting

3. The date-time format used by the Twitter API is not directly parseable in .NET. The format returned is in the format "Fri Feb 01 18:18:08 +0000 2008". But if we change this to "Fri Feb 01 +0000 2008 18:18:08" it is parsable. So you might need to adjust these date-time values.

4. You will get 417 error codes when you try to talk to Twitter using the default configuration. The fix is not immediately apparent, but it is very simple. So if you see the following error:

System.ArgumentOutOfRangeException: ExpectationFailed (417) is not one of the following: OK (200), Created (201), Accepted (202), NonAuthoritativeInformation (203), NoContent (204), ResetContent (205), PartialContent (206) at Microsoft.Http.HttpMessageExtensions.EnsuRESTatusIs()

Just to set the following property: ServicePointManager.Expect100Continue = false;

5. "Paste XML as Types" doesn't entirely work. For some reason, certain messages from Twitter are not deserializable to the types generated with this command. My experience was that I didn't actually care about the data causing the problem. So I just removed that part of the generated type. You may have to subscribe to the XmlSerializer error events to prevent exceptions.

Conclusions

I hope you now have a greater appreciation for what you can do with Twitter and how it can help you build your brand or build engaging applications. We've used the REST Starter Kit to make it easy to consume the Twitter API. You've even seen some of the odd things that can go wrong and how to fix them. Don't forget to download the sample application here:

http://www.michaelckennedy.com/Samples/SampleStatusUpdater.zip

Now get out there and build something cool.

[Update: If you are using ASP.NET 4 and .NET 4, Microsoft has added direct, built-in support into the Page class (the foundational class for WebForms pages). See Scott Guthrie's post on this topic: URL Routing with ASP.NET 4 Web Forms (VS 2010 and .NET 4.0 Series).]

I'm a huge fan of ASP.NET Routing. It gained popularity as the part of ASP.NET MVC which channels requests for a given URL to the right controller action. In a wise move, Microsoft moved the routing infrastructure out of ASP.NET MVC and into its own assembly with the release of .NET 3.5 SP1.

With ASP.NET Routing you can construct search engine optimized and human friendly URLs such as these:

Here part of the URL (tag or user) selects the page and part of the URL (everything or codinghorror) are effectively query parameters to the page.

This is well documented in the ASP.NET MVC world running on your server - you can't get anything done without it in MVC. But what about Windows Azure? What if you don't want ASP.NET MVC? What if you're a traditional type of person and want all the goodness that comes with what is now called ASP.NET WebForms (aka "normal ASP.NET")?

In this brief post, I'll cover how to use ASP.NET routing and ASP.NET WebForms in Azure. The sample project can be downloaded if you want to follow along. Phil Haack has written a good post on using routing alongside ASP.NET WebForms so I won't cover too much background information.

How does this change for Azure?

The short answer is that it doesn't. If you get routing working for IIS 7 in your web app, you can effectively deploy it to Azure. But the steps always felt convoluted to me when reading others' write-ups on this. So let's run through converting a Windows Azure Web Role (essentially a "stock" ASP.NET WebForms app) to use routing in Azure.

First you'll need the Azure SDK and Visual Studio tools:

1. Next, create a new solution in Visual Studio by choosing Cloud Service->Web and Worker Cloud Service.
   
   
2. Add a new Global.asax file to your web role project.
   
   
3. Add a reference to System.Web.Routing and System.Web.Abstractions in your web role project.
   
   
4. Define a custom class that derives from IRouteHandler which will map URL parameters into the HttpContext for use in your pages:
   

   internal class CustomRoute : IRouteHandler
   {
       public CustomRoute(string virtualPath)
       {
           VirtualPath = virtualPath;
       }
   
       public string VirtualPath { get; private set; }
   
       public IHttpHandler GetHttpHandler(RequestContext requestContext)
       {
           foreach ( var aux in requestContext.RouteData.Values )
           {
               HttpContext.Current.Items[aux.Key] = aux.Value;
           }
   
           return BuildManager.CreateInstanceFromVirtualPath(
                      VirtualPath, typeof( Page ) ) as IHttpHandler;
       }
   }

5. Register these routes in the Application_Start method of your Global.asax:
   

   protected void Application_Start(object sender, EventArgs e)
   {
       RouteTable.Routes.Add( "ShowName",
              new Route(
               "naming/show/{name}",
               new CustomRoute( "~/ShowName.aspx" )
               ) );
   
       RouteTable.Routes.Add( "CreateAccount",
              new Route(
               "account/begin",
               new CustomRoute( "~/Account.aspx" )
               ) );
   
       RouteTable.Routes.Add( "Home",
              new Route(
               "home",
               new CustomRoute( "~/Default.aspx" )
               ) );
   }

   Now if you run your app, you might expect the routing infrastructure to work. Inside the ASP.NET Dev Server (aka cassini) this will likely work. But in the Azure Development Fabric you'll see this:
   
   
   
   The problem is you need to tell IIS 7.5 to get out of the way and let the request get to ASP.NET.
   
   
6. We'll define a class to short-circuit the IIS validation
   

   class Iis7RoutingHandler : UrlRoutingHandler
   {
       protected override void VerifyAndProcessRequest(
           IHttpHandler httpHandler, HttpContextBase httpContext)
       {
       }
   }

7. Modify the web.config by adding a handler and module to the system.webServer section:
   

   
   
   ...
     
   
   
     
   
   
   

   /li>
8. Finally, we need to recover the data passed to the page. For example, in the sample project we have:
   
   route: /naming/show/{name}
   example: /naming/show/michael-kennedy
   
   How will our page access the value of name? Recall that our custom route stashes the values in HttpContext.Current.Items. We'll just pull them back out as follows in our Page_Load method of our ASPX class:
   

   LabelName.Text = (string)HttpContext.Current.Items["name"];
   

That's it! You can see our routes working in our WebForms app running in Azure (well, technically the screenshot is the dev fabric - but it works in the cloud as well):

Download the source and try it for yourself: AzureRoutingSample.zip (136 KB)

I recently wrote an article for DevelopMentor's Developments newsletter entitled Azure Storage. Read it at the DevelopMentor website here:

http://www.develop.com/content/newsletters/aprilazure

I've republished here for my readers. Enjoy!

---

Developments: Azure Storage

by Michael Kennedy

[Listen to this article as a podcast: Azure-Storage-Article-Kennedy.mp3]

October 27th 2008, Los Angeles CA - It's 9 AM and Microsoft is hosting PDC (their most forward looking developer conference). Ray Ozzie and company are introducing Windows Azure: A new platform which is their first foray into the nascent world of large-scale utility computing. This scalable and reliable platform-as-a-service functionality is commonly referred to as "Cloud Computing" because it runs somewhere out there on the Internet.

Computing platforms that rival the reliability of the utility grids (e.g. electric and gas) which we daily take for granted have long been the stuff of dreams.

A few companies have realized this dream - Google and Amazon come to mind as a couple of the rare exceptions who have accomplished this goal. These companies' web properties seem to handle unbounded amounts of traffic with zero down time. The data centers, redundancies, software engineering and operations know-how required to make this happen are exceedingly expensive. Some reports have Google spending over $2.4 billion (that's 2,400 million dollars) on data centers in 2007 alone.

Prior to large-scale cloud computing efforts (circa 2005), most of us could only dream of such scalability and reliability. Today we have at least three highly reputable companies offering some kind of pay as you go cloud computing platform - Microsoft, Amazon, and Google.

Microsoft's Azure is a new comer to the industry. But for .NET developers, it is not to be ignored. Azure allows you to use your existing skills to build essentially the same .NET applications you are familiar with and "deploy them to the cloud."

These scalable, reliable, and geographically-replicated applications that run on Azure depend on data of course. Virtually all applications we write will be nothing without their underlying data. But if we simply use the tried and true methods of data storage such as the file system or a (single) database server our data is not all that scalable or reliable. Because we cannot have a scalable and reliable application without data, we need a new mechanism for storing and accessing data from our Azure applications.

Enter Azure Storage

Azure storage is the storage component of the Azure platform. It is actually three data services in one:

• Blob Storage - stores unstructured data essentially as a file, limited to 50 GB of data per blob.
• Table Storage - stores structured data that is somewhat like a database. For full database capabilities there is a high level feature called Sql Data Services (SDS).
• Queues - provides interprocess communication functionality between various web and worker roles in your hosted services or even applications running outside of Azure. Queues can pass small xml or binary messages - less than 64 kb per message.

In this article, we will cover just the basics of the three storage services of Windows Azure. I want to give you a sense for what it's like to program against Azure Storage. At the base level all access to Azure Storage uses pure REST APIs. This means that you can access it from any HTTP enabled platform / language. For example, to download the blob data called "config.xml" in the container called "settings" for the Azure project "kennedy" you would simply issue a GET to the Uri:

   http://kennedy.blob.core.windows.net/settings/config.xml

To save data in a blob you do HTTP POSTs and PUTs in a similar fashion. However, real life is full of edge cases, error handling, security, and serialization which makes the pure HTTP model error prone. Thus, a sample library serves as the de facto .NET API to Azure Storage and ships with the Azure SDK. It is called StorageClient and can be found in default installs here:

   C:\Program Files\Windows Azure SDK\v1.0\samples\StorageClient

We will examine working with each of the storage services from the perspective of the StorageClient library - but keep in mind that ultimately this library is a wrapper around a basic and open RESTful API.

Setting The Stage: The Sample Application

To explore Azure Storage I have written a simple photo sharing distributed application. These set of applications allow users to upload photos to a photo sharing site. These photos must be reviewed and approved by moderators of the site. Once approved, the general public can view and interact with the photos. For a concrete example, you could imagine writing a distributed version of the wallpaper sharing site InterfaceLift and deploying it on Azure in this fashion.

You can download the sample application and follow along if you want to see the full source code and try it out yourself. Just be sure to start the Development Storage utility that comes with the Azure SDK before running the application.

Our distributed application consists of three parts.

A typical use case might be as follows (see diagram below).

1. We upload a photo submission with our uploader application. The photo is uploaded to Azure blob storage and a message is sent via an Azure Message Queue to all available reviewer applications. Additional information about the submitter is associated with the photo in Azure table storage.
2. The reviewer application watches the message queue for new messages. When one arrives, the photo is added to a list of pending submissions. The reviewer can either reject (delete) the submission or approve it - move it to a permanent blob storage location where it will be publicly viewable.
3. Users visit our website and can view all approved photos. This list will change in real-time because it is driven by the reviewer application. The web application simply pulls all photos from the approved photo container in Azure blob storage.

Saving Data: Creating Azure Blobs

To save data to Azure Blob Storage, you must realize blob storage follows the ACE pattern (Authority, Container, Entity) to describe a blob. Authority is simply your Azure solution name. Containers are analogous to folders. And entities are analogous to files.

The listing below is essentially the code that runs when the uploader application uploads a pending image submission to blob storage.

Listing 1:

Sending Notifications: Azure Queuing

In addition to uploading the image to the pending images container in blob storage, we will send a message to a message queue to notify any active or future reviewers of the new submission.

Listing 2.

Saving (More) Data: Structured Storage and Azure Tables

Finally, for the upload application, we must also save some information about the contributor. In Azure Storage we have two reasonable places to store this information.

First, we could save this information directly in blob storage as meta-data associated with the blob itself. This is straightforward and easy. But there is a big limitation: information in this meta-data is not queryable. Suppose I want get all images associated with a single contributor. There is no way in Azure Blob Storage to say give me all the blobs with this filter on the meta-data. You would have to pull the properties of every blob and do the comparison client-side. That's tantamount to filling a DataSet with "SELECT * FROM PendingImages" and it's a bad idea.

Instead we will use the third type of Azure Storage: Azure Table Storage. Table Storage allows us to store data with up to 256 properties and query this data as if it were a database. It is exactly what we need for the contributor information. However, you must realize this is not a database. A better mental picture is a durable collection of Dictionary object (as in Dictionary from System.Collections.Generics) with querying built on top. I say this because there is no schema or relational constructs in Azure Table Storage. If you need that, then you'll want Sql Data Services - a service on top of the core Azure platform.

The code to add an entry to Azure Table Storage does not fit into a single method as it's driven through the interaction of several classes we must define. Azure Table Storage can be accessed via ADO.NET Data Services (client-side) and this is the method we will use.

First we'll define a client-side schema for our entry by creating a class called Contributor which derives from the class TableStorageEntity (from the StorageClient library).

Listing 3.

Additionally we must define the tables and queries available to ADO.NET Data Services by created a class derived from TableStorageDataServiceContext and we do that below. We simply have one table called Contributors.

Listing 4.

With those two items in place, we can insert a "row" into Azure Table Storage as follows:

Listing 5.

As for querying Azure Table Storage that is very straight-forward. Because we are using ADO.NET Data Services, querying can be done via LINQ as in "from c in svc.Contributors select c.Name". Ultimately ADO.NET Data Services is also built on a RESTful API so this translates to the underlying HTTP REST calls. Alternatively, you can use that REST API directly from .NET or any other platform.

Waiting on Queues: The Reviewer Application's Code

Next, let's look at how we monitor and pull messages from Azure Queuing. Ultimately we must poll the queue using a RESTful HTTP request. But the StorageClient resurfaces this to us as simple events.

Listing 6.

We won't cover how we move a blob from the pendingImages blob container to the approvedImages blob container which happens when a reviewer approves an image. You can look at the sample to see how that is done.

Ultimately It's About the Website

Finally, let's look at the web application that actually displays the approved images. We don't do anything fancy such as paging or error handling that you'd see in a real application. But this will give you a good idea how to work with the blob data as a collection.

Here we'll create a BlobStorage object and access the BlobContainer approvedImageContainer as we have been in most of the listings. But then instead of saving or reading blobs, we use the ListBlobs method to simply list all the approve images in that container. In order to show the images on our webpage, we just use the BlobProperties.Uri and directly reference that in our HTML. Our ASP.NET application does not touch the data. Rather the consumers (IE, Firefox, Chrome, etc) of the HTML pull the image data directly from blob storage as they would from any web server.

Listing 7.

Now you have a good idea of the concepts and motivation behind Azure Storage. You have seen some typical usages of each of the three storage features: blob storage, table storage, and queuing. Our samples made use of the sample storage API library called StorageClient. Underlying this library we saw that Azure Storage is entirely accessed via RESTful APIs.

Want to get started? Visit http://www.azure.com and choose "Try It Now" to register for a CTP Azure account. You'll need to download the various SDK's listed on that same page. They will install the Visual Studio projects required for working with Azure as well as the Development Storage and Development Fabric so you can develop and debug your applications before deploying them to the cloud.

If you want some intensive, expert-lead training on Azure and associated .NET 4.0 topics be sure to contact DevelopMentor. Or call 800.699.1932 to find out what classes we have available today.

I'm pleased to announce that MSDN Magazine just published my Windows Workflow article entitled

    "ASP.NET WORKFLOW: Web Apps That Support Long-Running Operations"

I hope you find it useful and interesting.