[note: This screencast has been adapted from my earlier blog post.]

      

In Part 1 of my Building a Cloud OS for .NET Developers series, I talked about setting up a pure cloud OS focused on developers. But the one crucial aspect I left out was the developer tools.

In this second installment, we’ll be covering exactly that. How do we manage having Visual Studio and associated tools and servers universally accessible in the cloud, even on mobile devices such as iPads?

We’ll start with the king of the cloud:

    Amazon Elastic Compute Cloud (Amazon EC2)
    http://aws.amazon.com/ec2/

At EC2 we can create a variety of Windows instances, get full admin access via remote desktop, and install anything we want (Visual Studio, SQL Server, MongoDB, etc). We can also run these for just 30 minutes here, 2 hours there and we’ll only be charge for the time we use it.

We are going to setup a Windows Server in EC2 which we can access from our Cloud OS, or any other machine or mobile device, and deck it out with all the developer tools we want. You have to choose your server type based on what you’re planning on doing as well as how much you want to pay. Here are the available options and pricing. You can roughly expect to pay $0.50/hr for a good workstation, or $0.12/hr for an acceptable but small one.

Amazon EC2 Instance Types: http://aws.amazon.com/ec2/instance-types/
Amazon EC2 Pricing: http://aws.amazon.com/ec2/pricing/

I’m currently using a Micro Instance for an SVN server which is always running as well as High-Memory Extra Large Instance for my development machine that I start and stop on-demand when I need to get some programming done. Here are the specs for the developer workstation:

High-Memory Extra Large Instance

17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform
I/O Performance: Moderate
API name: m2.xlarge

You have to admit, that’s a pretty killer workstation. It’s a overkill on the memory, but has better processors than the cheaper ones which also have decent memory.

The price of the dev machine is $0.69 / hour. I’ve also built some tools which help shut down the machine in case I get distracted. So far I’ve been using it pretty heavily this past few weeks and have spent probably $13 on it. That’s not bad for the possibility of being anywhere in the world (with an Internet connection) and getting my workstation running in an instant. OK, maybe it takes longer than an instant. To be fair, it usually it’s 10-15 seconds :).

Take the Free Lunch

One thing you should do straight away if you’re going to try this is sign up for the EC2 Free Tier:

AWS Free Usage Tier
http://aws.amazon.com/free/

This won’t cover the EC2 CPU time (that’s Linux only), but it will cover storage, bandwidth, S3 snapshots, etc. You also get a free Linux machine to play with if you like.

Setting up Windows in the Cloud

OK, so if you’re chosen an instance type and are ready to start, you should do the EC2 walk-through I put together here to get your machine up and running and login for the first time.

Building Windows Machines in Amazon EC2
http://www.michaelckennedy.net/blog/2010/01/31/BuildingWindowsMachinesInAmazonEC2.aspx

I’d guess that takes about 15 minutes. When you’re done, come back here...

Installing Software in EC2 and Windows

So you’ve logged in and you want to install Visual Studio, SQL Server, and other MSDN goodies. The easiest way to get them is to use the ISO DVD images from the MSDN download site. In order to mount them on your EC2 machine (which has no DVD drive), you should install

Virtual CloneDrive
http://www.slysoft.com/en/virtual-clonedrive.html

which lets you mount ISO images as if they were proper DVD drives. Then you can download and install all the MSDN files you need. A good place to start is with Visual Studio:

Microsoft Visual Studio 2010 Premium Trial - ISO
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=f81412a2-d48e-4040-9b32-27eaf771c5db&displaylang=en

One word of warning here: I’ve noticed that Chrome can freak out and fail to download very large files inside your EC2 instance. Don’t know why, but it does. So be sure to use IE or FireFox to do the downloads for the ISOs.

Syncing your Files Between your Local Machine and Dev EC2 Machine

There are two techniques we can use to sync files, depending on what category of files we’re sharing and how. For the source files, you can always setup an SVN Server (that’s what my micro instance is about). I’m a fan of Visual SVN. It’s free and easy to setup and admin.

VISUALSVN SERVER
http://www.visualsvn.com/server/

Then on your EC2 dev machine (and your others you may have) setup:

TortoiseSVN
http://tortoisesvn.net/

and

AnkhSVN for Visual Studio Integration
http://ankhsvn.open.collab.net/

Then you have super easy, reliable, and secure sharing of your source files.

    

For everything else, you can just setup a Dropbox share between your Cloud OS and you EC2 Dev Machine:


      


      

If you don’t have Dropbox and want to try it, use this link to create an account

http://db.tt/Qo7m9PA

and you’ll get me some extra free space which I always appreciate. :) Then if you have a file on your local computer you want accessible on your cloud dev machine, just drop it in your Dropbox folder on the originating computer and it’ll be there as soon as it uploads.

Waste Not, Want Not

Now here is a problem you will encounter:

You’re working on your dev machine in EC2, something comes up and you get distracted and forget about it for the rest of the day. When you come back the next day you see your EC2 system is still running and you’ve wasted $12 by leaving it running idle overnight. That sucks. If it happens on Friday, it might run the whole weekend which is even worse.

To avoid this, I wrote a little tray utility that watches for extended idle times and will shutdown your EC2 instance if it’s idle for too long.
     

Below is how you configure it. Simple, right?

    
Add it to your startup menu to make sure it starts when you log in.

     

You can download it here. I may blog about this app in greater detail later, for now feel free to use it.

Shutdown on Idle
http://www.michaelckennedy.com/Downloads/Tools/ShutdownAfterIdle.zip

Reliable Access to EC2

One annoyance in EC2 is that every time you turn off your EC2 computer, it will take on a new random public DNS name. If you really want to access to your EC2 machine from anywhere, you probably want a reliable DNS name / IP address. That’s where Elastic IPs come into play in EC2:

Feature Guide: Amazon EC2 Elastic IP Addresses
http://aws.amazon.com/articles/1346

If you create an elastic IP, then you can map it to a sub-domain in your domain and always use that. For example I could register it for cloud-developer.michaelckennedy.net. Then as long as I have a way to start my machine and re-associate the IP, all my config on my various machines (such as remembered RDP settings and passwords and such) will “stick”.

Be aware that while your EC2 machine is off, you’ll be charged a small fee (few dollars / month) for the address. You can decide whether it’s worth the trouble to save the few dollars.

Mobile / iPad Access

I’m sure when you think of accessing your EC2 dev machines, you’re thinking of doing that form Windows or maybe a Mac (using  http://www.microsoft.com/mac/remote-desktop-client). But the iPad (and even the iPhone) has apps to allow you to use Remote Desktop to EC2. For example, here’s editing the code for http://chatpast.com in Visual Studio via my iPad:


      
                                    (click for full size image)

And you can even bring up the keyboard or attach a Bluetooth keyboard.

      
                                    (click for full size image)

The app I was using is called “Jump”, and you can find it here:

Jump for iPad
http://jumpdesktop.com/

Summary

I hope that gives you some inspiration to try a new way to work in the cloud. You saw that we can create fairly serious EC2 Windows computers which can serve as developer workstations. We can share files via source control and Dropbox and we can access the computers from Windows, from Macs, and even mobile devices like an iPad.

Good luck!
@mkennedy

[Update: Read Part 2...]

Let’s talk about cloud operating systems. This article explores the options and potential of moving entirely “To The Cloud” for developers who normally demand significant offline power from their applications (IDEs, compilers, debuggers, etc). We’ll focus on .NET / Visual Studio developers, but I’m sure you can adapt this to your technology of choice.

I’ve been fascinated with what Google is doing with Chrome OS. I think finding a way to fully “live in the cloud” has a lot of promise. However, in practice Chrome OS is entirely unappealing to me. It’s just a single browser window, maximized. Yuck. I don’t mind doing most things in the browser, but I’d like multiple non-maximized browser windows and a desktop to organize things like shortcuts. That’s OK though because Chromebooks aren’t the only option. We have decent operating systems right now that can function largely in the same way if we set them up with discipline.

What we’re going to do is take a pristine Windows 7 instance and deck it out for the cloud. Even with all the choices these days, Windows 7 is still my favorite OS. I do have a Mac, several iOS devices, and I could always install Linux. Nevertheless, it’s Windows 7 for me so that’s where we’ll start.

Creating a Fresh Windows 7 Instance - Gently

I want to start from the very beginning. I don’t want old apps I’ve already installed encouraging me to “cheat” and work outside of the cloud more than necessary. At the same time I need to be able to fire up Visual Studio and do some work with an unreliable or nonexistent Internet connection So I’m keeping my current “full” Windows system intact. We’ll install a secondary instance of Windows using the awesome Boot to VHD feature in Windows 7 Ultimate. If you haven’t heard of it, I did a video walk-through here:

   Boot to VHD Screencast:
   http://www.michaelckennedy.net/blog/2009/10/20/BootToVHDScreencast.aspx

Scott Hanselman’s write up is very helpful too.

   Less Virtual, More Machine - Windows 7 and the magic of Boot to VHD:
   http://www.hanselman.com/blog/LessVirtualMoreMachineWindows7AndTheMagicOfBootToVHD.aspx

This process takes about 15 minutes. Once you’ve got everything setup, I recommend you activate and then snapshot that VHD file (from your main OS) so you can always get back to “clean”. If you’re really living the cloud lifestyle, than paving your system is much easier than before. I always keep a secondary partition around for data files anyway so that can be shared across OSes.

Now that you’ve got a truly fresh Windows 7 copy, it’s time to install just a few apps needed to set you free (and a few you’d rather not live without).

Installing “The Cloud”

While your fresh copy of Windows 7 does have IE 8 installed, that browser is entirely not up to the task of being where you spend 90% of your time. I strongly encourage you to give the latest Chrome beta a shot as your main browser. The “application shortcuts” feature of Chrome makes it much more immersive than anything IE 9 or FireFox 4 are doing (yes, I know they have pinned sites, that’s not even close).

   Chrome Beta
   http://www.google.com/intl/en/landing/chrome/beta/

Got Chrome installed and set as your default browser? Good. Don’t forget to install all your Windows updates and Windows 7 SP1 while you’re at it.

What about Visual Studio?

You may be thinking, “I *NEED* Visual Studio 2010”. Yes, we do need it for sure, but stick with me here. We are NOT installing that beast on our fresh Windows 7 system. Remember, this is your Cloud OS. I even named mine Cloud OS in my boot loader to keep that focused in my mind.

      

We’ll talk about Visual Studio soon.

Installing Your Web Applications

You may think that you do not need to install your web apps cloud apps. There are a few installers we’ll be running such as DropBox and ChatPast, but even the plain vanilla websites such as GMail are better if you install them. That’s why Chrome is way better than the other browsers.

Let’s suppose you’ll want to use Google Docs to work with documents, presentations, Excel worksheets and so on (you will want this!). Visit https://docs.google.com, then choose “Create application shortcut” in Chrome (see image):

   

This not only gives you start menu items and desktop shortcuts, but it makes your taskbar truly useful and your web apps behave as regular Windows apps.

  

In addition, you get the true feel of your websites being applications without all the browser toolbars and other junk around it. For example, here’s how this article (barely written) looks in Google Docs right now. Notice, how it looks like an app rather than a tab buried in a busy browser. It remembers its window size and position. In short, it’s more application-like.



What Cloud Apps Should I Install?

OK, so I hope I convinced you that treating your web apps as true applications is worthwhile. Now which ones do I install? Here are a few I recommend. You may have your own favorite sites. We’ll also need a few apps that do cloudy stuff but run MSI installers on our system.

- GMail: https://mail.google.com/ (obviously :) )
- Google Docs: https://docs.google.com (good Microsoft Office replacement)
- Google Calendar: https://www.google.com/calendar/
- Google Contacts: http://mail.google.com/mail/contacts/u/0/ui/ContactManager
- 37 Signals LaunchPad: https://launchpad.37signals.com (Great web apps for projects)
- Amazon Web Service: https://console.aws.amazon.com/ec2/ (You’ll see why soon)
- Twitter: http://twitter.com/
- Office Online: http://office.live.com (for when Google docs fail you)
- Pandora: http://www.pandora.com/

Here are a few proper apps that are cloud-based in some way or another that require installers.

- Skype (obviously :) http://www.skype.com
- DropBox: https://www.dropbox.com
- ChatPast Sync Client: https://chatpast.com/

By using GMail, Google Contacts, and Google Calendar, you obviously get portability from the web. But if you have a modern phone or tablet, chances are you can keep them 100% in sync using Google’s Exchange support. I do that for both my iPhone and iPad.

There are also a couple of light-weight apps that I just didn’t want to do without and still have a solid use in a Cloud OS. I needed a good image editing app (for writing blog posts like this) and one for screen captures. So we have

- Paint.NET: http://download.cnet.com/Paint-NET/3000-2192_4-10338146.html
- Window Clippings: http://www.windowclippings.com/

Finally, there are a couple that I use to just keep an eye on my system such as

- Process Explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653

Here is the full list of *everything* I have installed on my system. Just 11 applications installed. Note that I’ve hidden some that Windows installs without asking. For example, using my Microsoft Mouse forces an install of Microsoft’s Intellipoint software when I plug it in. Similarly for the touch-pad on my laptop. To me, those don’t count so I edited them out.

   

Escape from Outlook

You may be thinking that because you still use your company’s email that you’ll need to install Microsoft Office and the 800-lbs gorilla that is Outlook. Chances are you will not need Outlook. I have several “regular” email accounts I have to check that don’t have reasonable web options.

You can setup your GMail account to automatically pull from your other email accounts as if GMail itself was a POP3 client like Outlook. Just go to Gmail > Settings >  Accounts and Import > and configure “Check mail using POP3” as well as “Send mail as”. Here you see my other accounts have recently been checked and received mail.


  

Sometimes having a “Send and Receive” button for these accounts is handy. Visit the Labs section of GMail and install the “Refresh POP accounts” add-on to make the refresh button propagate a refresh off to your other accounts as well.

  
While you are in there, you should turn on the keyboard shortcuts and take 5 minutes to learn the important ones. For example, / will take you to search, i to the inbox, e archives, <shift>-c composes an email in a separate window, etc.

Where Are the Developer Tools?

You probably didn’t see Visual Studio in that list, because it wasn’t there! In the next blog post, I’ll take you through setting up an Visual Studio 2010 instance in the cloud along with all the other developer tools and servers you might need such as SQL Server, and MongoDB, and LINQPad, and so on. Stay tuned.

When I have that post online, I’ll be sure to add a link here. If you just can’t wait, here’s a hint of where we're going next.

[Update: Read Part 2 Now]

Good luck!
@mkennedy

This article is a follow up one I wrote last week entitled “The NoSQL Movement, LINQ, and MongoDB - Oh My!”. In that article I introduced the NoSQL movement, MongoDB, and showed you how to program against it in .NET using LINQ and NoRM.

I highlighted two cornerstone reasons why you might ditch your SQL Server for the NoSQL world of MongoDB. Those were

1. Ease-of-use and deployment
2. Performance

For ease-of-use, you’ll want to read the original article.

This article is about the performance argument for MongoDB over SQL Server (or MySql or Oracle). In the first article, I threw out a potentially controversial graph showing MongoDB performing 100 *times* better than SQL Server for inserts.

“A potentially controversial graph showing MongoDB performing 100 times better than SQL Server”

We’ll see source code, downloadable and executable examples and you can verify all of this for yourselves. But first, here’s a new twist on an old proverb:

“Data is money”

If your application is data intensive and stores lots of data, queries lots of data, and generally lives and breathes by its data, then you’d better do that efficiently or have resources (i.e. money) to burn.

Let’s imagine you’re creating a website that is for-pay and data intensive. If you were to attempt to plan out your operating costs per user to help guide the pricing of your product then the cost of storing, querying, and managing your data will likely be a significant part of that calculation.

If there is a database that is 100 times faster than SQL Server, free, easy to administer and you program it with LINQ just as you would with SQL Server then that is a very compelling choice.

When you have such a database, it means you can run your system on commodity hardware rather than high-end servers. It means you can have fewer servers to maintain and purchase or lease. It means you can charge a lot less per user of your application and get the same revenue. Think about it.

“It means you can charge a lot less per user of your application and get the same revenue. Think about it.”

One more story before we see the statistics. Kristina Chodorow from 10Gen gave a talk a few weeks ago at San Francisco’s MySQL Meetup entitled “Dropping ACID with MongoDB”. You can watch the recording here:

http://www.ustream.tv/recorded/6146875

[The audio and video isn’t too hot, but the content is. Skip the first minute without audio.]

During this talk, Kristina describes SourceForge’s experience moving from MySql to MongoDB. On MySql, SourceForge was reaching its limits of performance at its current user load. Using some of the easy scale-out options in MongoDB, they fully replaced MySQL and found MongoDB could handle the current user load easily. In fact, after some testing, they found their site can now handle 100 times the number of users it currently supports.

Not convinced of this NoSQL thing yet? Fair enough. Here are some graphs, some stats, and some code.

The scenario:

Model a data intensive web application aiming to support as many concurrent users as possible. There will be users from the web application itself. But there will also be users from an API and external applications. Users will interact with the data by having nearly as many inserts as they do queries. Their inserts are all small pieces of data and are all independent of each other.

Let me just get this out of the way and I mean the following in the nicest of ways: I don’t care about your scenario or use-case. The scenario above is what I’m trying to model. I’m not trying to do bulk-inserts or loading large files into databases or anything like that. MongoDB may be great for these. SQL Server may have specialized features around your use-case, etc. They don’t apply in my scenario. So please don’t wonder why I’m not using bulk inserts or anything like that in the examples below.

Insert Speed Comparison

It’s the inserts where the differences are most obvious between MongoDB and SQL Server.

These inserts were performed by inserting 50,000 independent objects using NoRM for MongoDB and LINQ to SQL for SQL Server 2008. Here are the data models:

MongoDB basic class

SQL Server basic class

I ran five concurrent clients hammering the databases with inserts. Here’s the screenshots for running against MongoDB and against SQL Server. Let’s zoom into the most important result with the output from one of five concurrent clients:

MongoDB:

SQL Server:

That’s right. It’s 2 seconds verses 3 1/2 minutes!

Now to be fair, this was using LINQ to SQL on the SQL side which is slow on the inserts. After discussing these results with some friends, I re-ran the tests using raw ADO.NET style programming and saw a 1.5x-3x performance improvement for SQL. That still leaves MongoDB 30x-50x faster than SQL.

Query Speed Comparison

Now let’s see about getting the data out using the same objects above on the indexed Id field for each database.

Here MongoDB still kicks some SQL butt with almost 3x performance. If we were to leverage the mad scale-out options that MongoDB affords then we could kick that up to many times more.

“If we were to leverage the mad scale-out options that MongoDB affords then we could kick that up to many times more.”

Complex Data and the Real World

Feel like that was an overly simplified example? Here’s some real world data with foreign keys and joins. Below is the complex data model.

MongoDB:

SQL Server:

It shouldn’t surprise you that MongoDB does even better here without its joins.

The Hardware

All of these tests were run on a Lenovo T61 on Windows 7 64-bit with a dual-core 2.8 GHz processor using the 64-bit versions of both SQL Server 2008 Standard and MongoDB 1.4.1. You can even see a picture of the computer here: http://twitpic.com/hywa8

Your Turn

If you want to see the entire set of data above as an Excel spreadsheet, you can download that here:

http://www.michaelckennedy.com/Downloads/sql-vs-mongo.xlsx

You can also download the sample code. Before you do, realize I haven’t done a bunch of work to make it super easy to run. But you should be able to figure it out. Just turn the knobs on the PerfConstants class for the number of inserts and queries. Then comment or uncomment sections of the code in the clients for your scenarios.

The expected use is that you’ll start the launcher application then use it to launch five concurrent clients at exactly the same time.

Download Sample:

http://www.michaelckennedy.com/Samples/SpeedOfSqlVsMongoDBAnddotNetSample.zip

Got feedback? Write a comment or contact me on Twitter: @mkennedy or find me in any of these other ways.

Thanks!

Some thanks are in order for all the help I got bouncing around ideas as well as trying different scenarios. Thanks to

Eric Cain @arcain
Jim Lehmer@dullroar
Karl Seguin @karlseguin

I'm working on a fantastic website that I hope will have significant impact when it's ready. I'm planning on launching in roughly one month.

I came across what I think is an awesome technique for seeing how your web page will look as you edit it. This is WAY beyond WYSIWIG:

1. Load the page you're working on in ALL the browser you care about. I'm using Chrome 4, FireFox 3.6, and IE 8.
2. If you have the monitor space, cascade these browsers side-by-side.
3. Add a meta-refresh tag to the header of that HTML file you're working on (or which consumes the CSS you're building)
   
      <meta http-equiv="refresh" content="5" />
   
   
4. Now here's the sweet part:
   
   Edit the page in Visual Studio, notepad, whatever. When you press save all browsers reload their view in a few seconds!
   
   
5. Now you get real WYSIWIG on real browsers.

That's it. The technique is totally low tech and would have worked for years. But I found it really helpful. Hope you do too.

Be sure to keep watching here. I promise a cool site will be announced soon!

Cheers!
Michael
@mkennedy

I recently did a webcast for DevelopMentor on using the routing framework introduced in ASP.NET MVC within ASP.NET WebForms based applications to build more modern websites without a major rewrite of existing web applications. The talk was called "Building Modern Websites with ASP.NET WebForms".

Here's all the related downloads. We had some microphone troubles so I want to apologize in advance for the sub-optimal sound quality.

   

     

You can also download the slides and peepleocity.com sample website built during the presentation.

I recently wrote an article for DevelopMentor's Developments newsletter entitled Building a Twitter Application in .NET.

You can read it at the DevelopMentor website:

   http://www.develop.com/twitternetapps

I've republished here for my readers. Enjoy!

---

Building a Twitter Application in .NET

by Michael Kennedy (@mkennedy)
http://www.michaelckennedy.net

Twitter has become one of the web's hottest properties. It is a central part of mainstream news programs such as CNN's Anderson Cooper 360, congressional debates, and talk shows. In fact, it grew at a rate of 1400% this past year [bit.ly/jG9BG].

If your company wants to interact with your customers in a modern and engaging experience, you need to be on Twitter. In fact, if you have customers that really like or dislike you, they are probably talking about you on Twitter. You should be part of that conversation.

In this article, we will explore how to build a rich interactive experience on Twitter that goes beyond just creating a new Twitter account. We will build a .NET application that uses the Twitter API (a free service) alongside other cool technologies such as the WCF REST Starter Kit [http://bit.ly/v8mBb] and LINQ to fully leverage the Twitter experience.

Whether you want to build a community around your brand or you want to build the next real-time, social community website like .NET Developer Buzz [http://dotnet.ubbuzz.com/], this article will cover the technologies required to get the job done.

If you want to download the sample application to follow along, you can get it here:
   http://www.michaelckennedy.com/Samples/SampleStatusUpdater.zip

While you're at it, be sure to follow DevelopMentor on Twitter:
    http://twitter.com/DevelopMentor_

Let's Start Small

I will show you how to fully leverage the Twitter API, but many tasks can be accomplished using simpler tools and you may be better starting there. Let's look at a few things that we can do without the Twitter API.

Case 1. You want to display your latest Twitter messages on your website.

Your tweets[1], as they're called, can be consumed as a simple RSS feed. So you may want to simply pull this feed into your website rather than digging into the Twitter API and consuming custom XML or JSON formats. An example of consuming Twitter in this fashion can be found on my website's front page [http://www.michaelckennedy.net/].

To get your RSS feed, just visit your profile page and get the RSS tweet link, e.g. "RSS feed of mkennedy's tweets" [http://bit.ly/guhZU]

[1] Tweet - these are what the individual messages sent on Twitter are called. If the name sounds weird, I'm sure you'll get used to it. Remember that there was a time when Google was just a noun.

Case 2. You want to watch and manage multiple accounts at the same time.

Most Twitter clients only support a single user. But there are a couple of good tools that allow you to manage multiple accounts. My current favorite is one called bDule and you can get for free at http://www.sobees.com/bdule.

Beyond the Small

There are times when you want to do more than simply syndicate your Twitter stream. Let me give you an example. At DevelopMentor, we have had many instructors on Twitter talking about their own interests. But we didn't have a corporate Twitter presence. We decided to create our corporate presence by pulling all our instructors individual tweets and rebroadcasting them from our DevelopMentor Twitter account: @DevelopMentor_

We wanted to keep a sense of the original instructor who wrote the message, so we append on an attribution. For example:

"Software Transactional Memory is released! (via @mkennedy)

and we wanted to do this in a flexible way. In short, we needed more functionality than Twitter provides.

There are actually three services that do this sort of thing and they looked promising.

   http://www.connecttweet.com/
   http://www.grouptweet.com/
   http://cotweet.com/

But in the end, nothing completely matched our requirements. So we decided to write our own application to publish everyone's tweets under the DevelopMentor banner. There are few simple steps involved as well as a lot of details we won't go into yet.

1.  Take a list of Twitter accounts and download everyone's statuses.
2. Determine which messages we haven't seen before.
3.  Publish these new statuses under our corporate account.
   

We can actually implement a simplistic version of this by continuing to use the public RSS feeds of the individual accounts in conjunction with a very handy Twitter API wrapper called TwitterooCore which you can find at http://rareedge.com/twitteroo/blog/.

The Twitter API

In practice, there is simply data missing from the RSS feed that we require as well as features missing from the Twitteroo Core that move us deeper into the Twitter API. One thing you may well miss is the ability to tie together conversations. For example if Bob says "hello" and Jerry says in reply "@bob Back at you!" Twitter tracks that Jerry replied to Bob and publishes this link in the stream. To get access to these types of features and many other optimizations, you'll need to use the REST-based Twitter API.

The Twitter API is documented at http://apiwiki.Twitter.com/

There you can do things like get the users tweets, if their tweets are public, by requesting:

http://twitter.com/statuses/user_timeline.xml?screen_name=mkennedy

What you get back to dependent on the requested format. Here we're asking for XML (user_timeline.xml) but we can also get JSON, RSS, or ATOM.

Similarly, we can update our status by making a POST request to:

http://twitter.com/statuses/update.xml

Here again we have the four possible formats: XML, JSON, RSS, ATOM. However this time we're using a form post for the request to update the user status which is then in-turn returned as XML.

Great, we have this cool REST API based on loosely-typed GETs and POSTs. Should we program against it using fundamental .NET types such as WebRequest, WebClient, and similar classes? We could. But WebRequest is so .NET 1.0 (circa 2001). There is a much newer API on the verge of release from Microsoft: The REST Starter Kit [http://bit.ly/1IF3Ji].

While this toolkit is generally geared towards building to RESTful WCF services, there's also a great set of classes for building REST clients. We will use these classes to write our application. Let's take a look at how we can use the Twitter API to write are simple application.

The REST Starter Kit

Part 1 - Getting the users of tweets.

We need to download the users messages as XML and convert them to .NET objects that we can consume our application. This is really straightforward because the REST Starter Kit as a cool feature to Visual Studio: "Paste XML as Types". This feature will take an XML file and auto-generate types based on the inferred XML schema. In our case, the XML file we will use is returned from the user timeline.

http://twitter.com/statuses/user_timeline.xml?screen_name=mkennedy

Open a link in a web browser, choose view source, and copy some XML. Then go to Visual Studio, choose edit, Paste XML as Types. Of course you must have the REST Starter Kit installed for this to work.[2]

[2] In my experience I ran into some errors deserializing the response from Twitter. See the The Real World Intrudes section at the end of the article if you run into difficulties.

After generating our status related types, we can use the HttpClient class to download the statuses.

Listing 1.

private static void GetStatuses(IEnumerable userNames)
{
    XmlSerializer serl = new XmlSerializer(typeof (statuses));
    serl.UnknownElement += delegate { };

    foreach (string name in userNames)
    {
        string url = string.Format(
            "http://twitter.com/statuses/user_timeline.xml?screen_name={0}",
            name);

        HttpClient client = new HttpClient(url);

        HttpResponseMessage response = client.Get();
        response.EnsureStatusIsSuccessful();

        string contents = response.Content.ReadAsString();
        Stream stream = new MemoryStream(Encoding.UTF8.GetBytes(contents));
        statuses userStatuses = (statuses) serl.Deserialize(stream);

        if ( userStatuses.status.Length > 0 )
        {
            Console.WriteLine( "@{0}'s latest tweet: {1}", name,userStatuses.status[0].text );
        }
        Console.WriteLine();
    }
}

There are two main things happening in this code. We're using the HttpClient class to download the web content associated with the given user's timeline. Then we're using the XmlSerializer in conjunction with the auto-generated XML-serializable types from the "Paste XML as Types" command. That's all there is to it. The REST Starter Kit does most of the work for us.

Part 2 - Checking for New Messages

Now that we have all the statuses, we need to find the ones that we haven't broadcasted from our main account and send them along to step 3. We won't go into detail on how to track that. But you can imagine a simple database that facilitates a LINQ to SQL query like this:

Listing 2.

public StatusUpdate[] FindRebroadcastableStatuses(StatusUpdate[] updates)
{
    return
        (from up in updates
         let neverPosted = up.User.LastPostBroadcasted == null
         let afterPostDate = up.Time > up.User.LastPostBroadcasted
         where neverPosted || afterPostDate
         select up).ToArray();
}

Part 3 - Publishing the Statuses

Now that we've gathered the statuses of our various users, it's time to rebroadcast them to our community. We're going back to the Twitter API to update our status. Again will use the HttpClient class and following the RESTful principle of using POST to add new items to a given URI. We will do an HTTP POST to our status to add a new message to the account.

Listing 3.

private void UpdateStatus(string newStatus, long? replyToId)
{
    string url = "https://twitter.com/statuses/update.xml";

    HttpClient client = new HttpClient( url );
    client.TransportSettings.Credentials = 
        new NetworkCredential( twitterUser, password );

    HttpUrlEncodedForm form = new HttpUrlEncodedForm();
    form.Add( "status", newStatus );
    if ( replyToId != null )
    {
        form.Add( "in_reply_to_status_id", replyToId.ToString() );
    }

    HttpContent content = form.CreateHttpContent();

    HttpResponseMessage message = client.Post( "", content );
    message.EnsureStatusIsSuccessful();
}

This time we create an HTML form using the HttpUrlEncodedForm class. We set the status field to our new status. If this status as a response to a previous tweet then we add the in_reply_to_status_id so Twitter knows to add a "in reply to..." tag to the tweet.

And there you have it. Working with the Twitter REST API is really quite straightforward. If you use the REST Starter Kit it's downright easy.

The Real World Intrudes

But wait a minute. This is reality and building bulletproof applications is never that simple. There are at least five significant gotchas you have to address in practice when working with Twitter and the REST Starter Kit.

1. There will be times when Twitter is unavailable. You have to be ready for crashes and other types of randomness.

Twitter is one of those sites that can barely handle the traffic it is receiving. With its 1400% growth, this isn't getting much better. So you must program defensively and assume that many of your API requests will fail.

2. The Twitter API is a free service. By default, you are limited to a small number of requests per hour. Many of the limits are around 150 API calls / hour. You may need to carefully design your application to work within the limits. Some applications simply need more data than this permits. For example, .NET Dev Buzz [http://dotnet.ubbuzz.com/] has to track thousands of users. In that case, you will have to get your application white listed with Twitter. You can do that here:

http://twitter.com/help/request_whitelisting

3. The date-time format used by the Twitter API is not directly parseable in .NET. The format returned is in the format "Fri Feb 01 18:18:08 +0000 2008". But if we change this to "Fri Feb 01 +0000 2008 18:18:08" it is parsable. So you might need to adjust these date-time values.

4. You will get 417 error codes when you try to talk to Twitter using the default configuration. The fix is not immediately apparent, but it is very simple. So if you see the following error:

System.ArgumentOutOfRangeException: ExpectationFailed (417) is not one of the following: OK (200), Created (201), Accepted (202), NonAuthoritativeInformation (203), NoContent (204), ResetContent (205), PartialContent (206) at Microsoft.Http.HttpMessageExtensions.EnsuRESTatusIs()

Just to set the following property: ServicePointManager.Expect100Continue = false;

5. "Paste XML as Types" doesn't entirely work. For some reason, certain messages from Twitter are not deserializable to the types generated with this command. My experience was that I didn't actually care about the data causing the problem. So I just removed that part of the generated type. You may have to subscribe to the XmlSerializer error events to prevent exceptions.

Conclusions

I hope you now have a greater appreciation for what you can do with Twitter and how it can help you build your brand or build engaging applications. We've used the REST Starter Kit to make it easy to consume the Twitter API. You've even seen some of the odd things that can go wrong and how to fix them. Don't forget to download the sample application here:

http://www.michaelckennedy.com/Samples/SampleStatusUpdater.zip

Now get out there and build something cool.

You may know that I work for DevelopMentor where I'm an instructor in the .NET curriculum (among other cool things I do there). You probably also know I'm kind of loopy for Twitter as evidenced by my Twitter page and .NET community site driven by Twitter:

   http://twitter.com/mkennedy
   http://dotnet.ubbuzz.com/

Today those two things came together in a big way. Introducing DevelopMentor's Twitter presence:

    @DevelopMentor_
   http://twitter.com/developmentor_

I encourage you get out there and follow us! Here you will see all the combined Twitter messages of most of the DevelopMentor instructors as well as a couple of messages from DevelopMentor itself. You'll get the chance to keep on top of the world as viewed by some of the smartest people I've had the chance to work with: the DM instructors! You'll see which instructor posted any given message with an attribution at the end (either "via @marksm" or ^MS depending on the available space).

I'm excited about this not just because I think it is cool and useful (and has to do with Twitter :) ), but over the last few days I'm the guy who wrote the back-end systems to make this all go. It was a short but fun project. It's a real testament to .NET that this was mostly written an hour.

After you follow @DevelopMentor_ you might want to interact with some of the instructors directly. Here's our Twitter accounts in a single place for your reference:

http://twitter.com/bmaso
http://twitter.com/danamiga
http://twitter.com/jason_diamond
http://twitter.com/markblomsma
http://twitter.com/marksm
http://twitter.com/mauricedb
http://twitter.com/mkennedy
http://twitter.com/richardblewett
http://twitter.com/tonysneed
http://twitter.com/wallacekelly

I hope you enjoy the conversations we're bound to have on Twitter. Come be part of it.

Signed: @mkennedy

[Update: If you are using ASP.NET 4 and .NET 4, Microsoft has added direct, built-in support into the Page class (the foundational class for WebForms pages). See Scott Guthrie's post on this topic: URL Routing with ASP.NET 4 Web Forms (VS 2010 and .NET 4.0 Series).]

I'm a huge fan of ASP.NET Routing. It gained popularity as the part of ASP.NET MVC which channels requests for a given URL to the right controller action. In a wise move, Microsoft moved the routing infrastructure out of ASP.NET MVC and into its own assembly with the release of .NET 3.5 SP1.

With ASP.NET Routing you can construct search engine optimized and human friendly URLs such as these:

Here part of the URL (tag or user) selects the page and part of the URL (everything or codinghorror) are effectively query parameters to the page.

This is well documented in the ASP.NET MVC world running on your server - you can't get anything done without it in MVC. But what about Windows Azure? What if you don't want ASP.NET MVC? What if you're a traditional type of person and want all the goodness that comes with what is now called ASP.NET WebForms (aka "normal ASP.NET")?

In this brief post, I'll cover how to use ASP.NET routing and ASP.NET WebForms in Azure. The sample project can be downloaded if you want to follow along. Phil Haack has written a good post on using routing alongside ASP.NET WebForms so I won't cover too much background information.

How does this change for Azure?

The short answer is that it doesn't. If you get routing working for IIS 7 in your web app, you can effectively deploy it to Azure. But the steps always felt convoluted to me when reading others' write-ups on this. So let's run through converting a Windows Azure Web Role (essentially a "stock" ASP.NET WebForms app) to use routing in Azure.

First you'll need the Azure SDK and Visual Studio tools:

1. Next, create a new solution in Visual Studio by choosing Cloud Service->Web and Worker Cloud Service.
   
   
2. Add a new Global.asax file to your web role project.
   
   
3. Add a reference to System.Web.Routing and System.Web.Abstractions in your web role project.
   
   
4. Define a custom class that derives from IRouteHandler which will map URL parameters into the HttpContext for use in your pages:
   

   internal class CustomRoute : IRouteHandler
   {
       public CustomRoute(string virtualPath)
       {
           VirtualPath = virtualPath;
       }
   
       public string VirtualPath { get; private set; }
   
       public IHttpHandler GetHttpHandler(RequestContext requestContext)
       {
           foreach ( var aux in requestContext.RouteData.Values )
           {
               HttpContext.Current.Items[aux.Key] = aux.Value;
           }
   
           return BuildManager.CreateInstanceFromVirtualPath(
                      VirtualPath, typeof( Page ) ) as IHttpHandler;
       }
   }

5. Register these routes in the Application_Start method of your Global.asax:
   

   protected void Application_Start(object sender, EventArgs e)
   {
       RouteTable.Routes.Add( "ShowName",
              new Route(
               "naming/show/{name}",
               new CustomRoute( "~/ShowName.aspx" )
               ) );
   
       RouteTable.Routes.Add( "CreateAccount",
              new Route(
               "account/begin",
               new CustomRoute( "~/Account.aspx" )
               ) );
   
       RouteTable.Routes.Add( "Home",
              new Route(
               "home",
               new CustomRoute( "~/Default.aspx" )
               ) );
   }

   Now if you run your app, you might expect the routing infrastructure to work. Inside the ASP.NET Dev Server (aka cassini) this will likely work. But in the Azure Development Fabric you'll see this:
   
   
   
   The problem is you need to tell IIS 7.5 to get out of the way and let the request get to ASP.NET.
   
   
6. We'll define a class to short-circuit the IIS validation
   

   class Iis7RoutingHandler : UrlRoutingHandler
   {
       protected override void VerifyAndProcessRequest(
           IHttpHandler httpHandler, HttpContextBase httpContext)
       {
       }
   }

7. Modify the web.config by adding a handler and module to the system.webServer section:
   

   
   
   ...
     
   
   
     
   
   
   

   /li>
8. Finally, we need to recover the data passed to the page. For example, in the sample project we have:
   
   route: /naming/show/{name}
   example: /naming/show/michael-kennedy
   
   How will our page access the value of name? Recall that our custom route stashes the values in HttpContext.Current.Items. We'll just pull them back out as follows in our Page_Load method of our ASPX class:
   

   LabelName.Text = (string)HttpContext.Current.Items["name"];
   

That's it! You can see our routes working in our WebForms app running in Azure (well, technically the screenshot is the dev fabric - but it works in the cloud as well):

Download the source and try it for yourself: AzureRoutingSample.zip (136 KB)

I recently got the chance to record a screencast discussing REST-oriented web services in WCF. If you're interested in WCF you should definitely check it out because WCF and REST make an awesome combination.
   WCF-REST-Kennedy-Peepleocity.wmv 35 MB (WMV HD)

   WCF-REST-Kennedy-Peepleocity.mp4 96 MB (MP4 / iPad)

I cover building WCF services using REST princples, the WebGet and WebInvoke attributes, working with the SyndicationFeed & Rss20FeedFormatter classes, and configuration-free WCF hosting in IIS.

You can also download the source code of the project built in the screencast.

Finally, if you're willing to do without video you can download just the audio as an MP3.

Any of my technically savvy friends know that I'm a big fan of Channel 9. If you want the raw, inside view of Microsoft's developer world, it's a great place to start. That's why it was a big honor for me that my .NET Developer community website

   http://dotnet.ubbuzz.com

was featured on "This Week on Channel 9" last week.


  (see segment 7:45 - 10:25)

Thank you Dan Fernandez for covering our site! If you're unfamiliar with .NET Dev Buzz, then what are you waiting for? You'd better check it out!

I just launched the beta version of an entirely differnet kind of .NET community website:

.NET Dev Buzz
Bringing you absolutely fresh conversations about .NET and the Microsoft developer community.

http://dotnet.ubbuzz.com/

I'll follow this up with some highlights of a bunch of interesting .NET concepts being used here.

For starters, consider this. This is a website built in ASP.NET web forms, but there is no view state, no form posts, and no (visible) aspx files. It's fully RESTful.



If you like it, then