Hello there! It looks like this might be your first time to my website. You should...
Subscribe to my
RSS feed
Follow me
on Twitter.
Check out
.NET Dev Buzz

Portland CodeCamp Downloads: Demos and Slides

Monday, May 24, 2010 9:54:13 AM (Pacific Standard Time, UTC-08:00)

Thanks to everyone who came to my two sessions at the Portland CodeCamp this weekend.

The NoSQL Movement, LINQ, and MongoDB - Oh My!

Panel Discussion: NoSQL vs. RDBMS

You can download the slides, demo code, and the mongoctx code snippet for Visual Studio here:

NoSQL + MongoDB + LINQ:
http://www.michaelckennedy.com/Talks/Downloads/PortlandCodeCamp/Kennedy-PdxCodeCamp-2010-MongoDB-NoRM-LINQ.zip

NoSQL vs. RDBMS Panel:
http://www.michaelckennedy.com/Talks/Downloads/PortlandCodeCamp/Kennedy-PdxCodeCamp-2010-NoSQL-vs-RDBMS-Panel.zip

If you missed the sessions and want to see some MongoDB + LINQ or if you would like to recommend this session to a friend, I'll be doing a webcast on a very similar talk tomorrow:

NoSQL + MongoDB + LINQ Webcast

Finally, if you attended any of my sessions, please rate them here:

http://speakerrate.com/mkennedy

Cheers!
@mkennedy

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

The NoSQL Movement, LINQ, and MongoDB - Oh My!

Thursday, April 22, 2010 1:01:01 PM (Pacific Standard Time, UTC-08:00)

Maybe you’ve heard people talking about ditching their SQL Servers and other RDBMS entirely. There is a movement out in the software development world called the "No SQL" movement and it’s taking the web application world by storm.

“Insanity!” you may cry, “for where will people put their data if not in a database? Flat files? Tell me we aren’t going back to flat files.”

No, but in the relational model, something does has to give. The NoSQL movement is about re-evaluating the constraints and scalability of data storage systems in the light of the way modern web applications generate and consume data.

The outcry about flat files above is meant to highlight an assumption developers often have about building data-driven applications: Data goes in the database (SQL Server, Oracle, or MySql). Just maybe, if we are really cutting-edge, we might consider storing our data in the cloud, but the choices generally stop there.

The NoSQL movement asks the question:

“Is the relational database (RDBMS) always the right tool for data storage and data access?”

Starting from an RDBMS is virtually an axiom of software development. However, those of us who are excited about NoSQL believe that relational databases are not always the answer. I think this highlights one of the reasons this NoSQL thing is called a movement. People are realizing they have a choice where they thought they had none.

The converse is, of course, also true. The NoSQL databases are also not always the right choice either. If you look carefully however, you will find that they are a good choice much of the time. Don’t take my word on it. Ask Facebook, Twitter, Digg, SourceForge, WebEx, Reddit and a bunch of other companies here and here that are using NoSQL databases.

This move towards NoSQL is driven by pressure from two angles in the web application world:

  • Ease-of-use and deployment
  • Performance - especially when there are many writers as compared to the number of readers (think Twitter or Facebook).

Choosing NoSQL for Ease-of-Use and Deployment

I cover the programming model in detail as well as introduce the actual database server below. For some vague motivation, let me just give you a quick look at how you define the data model and maintain it.

  1. Define your classes in C# (largely) without regard to putting them in a database. Related classes? Easy - one has a collection of the others.
  2. Create a simple DataContext-like class which exposes each top-level type that is to be stored in the database. This is only a few lines of code per collection (think of this as a table).
  3. Interact with the database using LINQ. This creates the collections (think tables), sets the schema, etc.
  4. Maintain the database and evolve it by maintaining your classes from step 1. *

Why, in the name of all that is right, do we have to model our system twice? Once in the database and once, in parallel, in code? With NoSQL, you have one place to do that - in your C# classes.

* You may have to run a transformation tool if you’re making radical data changes, but that’s true in SQL systems as well.

Choosing NoSQL for Performance

When the number of concurrent clients using your application - and thus your database - is reasonably small (let’s say 500 users as a baseline) RDBMS can work great. But what if that number grows? And if you are writing a web app, you definitely want that number to grow. At 50,000 users, can you still run on a single instance of SQL Server or MySql? How powerful does your hardware have to be to handle that? What about at 500,000 or 5,000,000 users, still good?

I’m sure there are some of you out there thinking, “What a minute now! There are plenty of systems with tons of users built upon relational databases.”

It’s true, there are. But how much expensive hardware and software do these require? How easy is it to leverage *commodity* hardware and free software? A basic SQL Server cluster might run you $100,000 just to get it up and running on decent hardware. Rather than leveraging crazy scaling-up options, the NoSQL databases let you scale-out. They make this possible (dare I say easy?) by dropping the relational aspects of a database. Some NoSQL systems such as MongoDB get even better scalability by loosening some of the durability guarantees – which they backfill somewhat with redundancy (more on MongoDB shortly).

“Ok, ok. So it’s cheaper and simpler,” you say. “How much faster than the finely tune system that is SQL Server 2008 can these open source NoSQL systems be?”

The answer is: MUCH MUCH FASTER. Here’s a simple comparison of running a bunch of concurrent inserts into SQL Server 2008 and MongoDB on the same computer.

Looks like under heavy load, I’d say it’s about 100 times faster. I’m sure there going to be tons of second guessing this graph and so on. Hold your comments please! I’ll be posting a full performance comparison with source code soon. Let me just say that I think the comparison was fair - I’ll back that up in a later post.

NoSQL and a New Programming Model

If we do not have joins and primary / foreign key relationships, how do we associate related data? In NoSQL, there is a way to mimic foreign keys for certain relationships. However the main answer is that you do not disassociate your data in the first place.

I’m sure that you’ve all heard of the object-relational impedance mismatch. A large part of that mismatch comes from the fact that we normalize the data in our database to the extreme and then use joins to reassemble that data. Not only does that cause this so-called impedance mismatch, but those joins can be really slow and they can be the death of any scale-out solution. The key to many of the NoSQL databases’ scalability is that they do not use joins. You simply save large swaths of your data as a single blob (which in MongoDB’s case, is still deeply queriable).

Shortly we’ll look at an example where we build out a disconnected, offline RSS reader that uses MongoDB and LINQ to store its data. But just think about how you might structure your data storage if you could save entire object graphs and still query them? Your "row" might be a Blog object which has an array of BlogEntries which contain the entry text, link, date, etc. Then your *entire* query to pull all the details of a single blog would hit a single “table” in the database. That might look like this query which has one result:

var blog = 
       (from b in ctx.Blogs 
       where b.Id == requestedBlogId 
       select b).FirstOrDefault();

There are no joins or anything like that because you’re saving objects not columns and those objects contain their collections already (e.g. RssEntries). There is an important distinction to make here. These NoSQL databases generally are *not* the same as object databases. They are what are known as document databases. There’s actually a big difference between the two.

Introducing MongoDB

The NoSQL database we are using in this example is MongoDB. This is free, open-source database which runs on Windows, Linux, and Mac OS X systems. You can access it from many platforms including .NET, Ruby, Java, PHP, and so on.

We’ll be using .NET and C# of course. You have several options when choosing how to access MongoDB from .NET but generally that means using LINQ and a light-weight object-mapper on top of MongoDB itself. Note that common terminology might categorize the object mapper that moves objects into and out of the database as an ORM. While that’s OK, there is technically no "R" in this ORM because MongoDB is not relational. Hence I’m calling simply an Object-Mapper (OM).

In MongoDB nomenclature, theses libraries are called drivers. My favorite .NET driver is called NoRM. It’s being actively developed and was created by Karl Seguin, Andrew Theken, Rob Conery, James Avery, and Jason Alexander. You can find NoRM on GitHub and discuss it in its related Google Group.

If you want to learn more about MongoDB you should listen to these Podcast interviews:

Michael Dirolf also has a great book in the works. You can catch a preview of it on Safari Books Online. Here’s the amazon page:

MongoDB: The Definitive Guide.

NoSQL in Action

Let’s write some code. The first step typically in a data-driven application is to spec out the database. Then we’d use LINQ to SQL or Entity Framework to generate the ORM classes. MongoDB is different. MongoDB has no schema or rather its schema is flexible and defined via usage rather than being predefined in the database. So our first step is to define the classes we’d be storing in the DB via NoRM.

We’re going to define 3 classes: Blog, RssEntry, and RssDetail. The Blog object will contain a collection of RssEntry objects. In practice you might just go with the Blog and RssEntry classes. But I wanted to model both the embedded case (Blog + RssEntry) and the loosely defined foreign key style relationship that mimic joins (RssEntry + RssDetail). That way we can demonstrate both use-cases.

Here’s a taste of the Blog class:

public class Blog
{
	public ObjectId _id { get; set; }
	public string Name { get; set; }
	public string Url { get; set; }
	public string RssUrl { get; set; }
	public List<RssEntry> Entries { get; set; }
      // ...
}

Notice that it contains a collection (List<T> really) of RssEntry objects. That’s the relationship supported by nesting. The Blog class just has this collection as part of its data model.

The RssEntry class has the summary info for a blog entry:

public class RssEntry
{
	public ObjectId _id { get; set; }
		
	public Guid UniqueId { get; set; }
	public DateTime PostedDate { get; set; }
	public string Title { get; set; }
	public string RssGuid { get; set; }
}

And the larger data is stored in the RssDetails class (for example the text of the post):

public class RssDetails
{
	public ObjectId _id { get; set; }

	// this is kinda like the foreign key.
	public Guid RssEntryId { get; set; }

	public List<string> Categories { get; set; }
	public string Link { get; set; }
	public string Text { get; set; }
	// ...
}

Let’s see how we insert an entire set of Blog data into the database. We begin by generating the objects (Blog, RssEntry, etc) in memory and then serializing them via NoRM to MongoDB much as you would in LINQ to SQL. The difference is this will actually generate the collections (analogous to tables) if they don’t already exist and it will define the implicit schema to match our objects:

void SaveBlogToMongoDb(
	string rssUrl, XElement root, RssDataContext ctx)
{
	Blog blog = new Blog();
	blog.RssUrl = rssUrl;
	blog.Name = GetBlogName(root);
	blog.Url = GetBlogUrl(root);

	blog.Entries = ParseEntries(root);
	IEnumerable<RssDetails> details 
		= GetDetails(blog.Entries, root);
			
	foreach (RssDetails detail in details)
	{
		ctx.Add(detail);
	}

	ctx.Add(blog);
}

Here we are using a class called RssDataContext which we wrote manually. It is very similar to what LINQ to SQL and Entity Framework use to do the object-relational mapping. Want to do a query? Do you know LINQ? Well then you’re all set:

var results = 
    from b in ctx.Blog 
    where b.Name.Contains( "MongoDB" ) 
    select b;

How do you add a new entry to an existing blog and update it in the database?

void AddEntry(Blog blog, RssEntry entry)
{
	blog.Entries.Add(entry);
	ctx.Save(blog);
}

We leverage the fact that the blog.Entries collection is a List and just add to it. Then save will update the record in the DB.

All this works great and is highly performant. But do be careful as not all the LINQ operations are fully implemented yet in NoRM and some (like join) may never be added because MongoDB doesn’t support it.

To get started, download MongoDB the tools and server here:

http://www.mongodb.org

You unzip the zip file and run the mongod.exe program. Be sure that you have created the C:\data\db folder. It appears at first that you have to run MongoDB in a console window. But you can register it as a Windows Service:

Here’s some helpful advice on installing MongoDB as a Windows Service (there is a small bug you have to work around):

http://www.deltasdevelopers.com/post/Running-MongoDB-as-a-Windows-Service.aspx

There’s also a management console (and I mean "console"):

It’s a little different. You’ll get used to it. The means of interaction with the server is through JavaScript rather than T-SQL and the storage format is a binary form of JSON as you can see.

For a project I’m working on I’ve built a Windows Forms UI that lets me manage the database easily by just adding an object data source and doing some drag-drop magic in Visual Studio. Generally I look down upon that sort of development, but for an admin tool it’s just fine.

Now It’s Your Turn!

Try it out for yourself. Download MongoDB and the NoRM driver and build some apps. You may also want to check out the source code for my demo app:

Download Sample: RssMongoSample-Kennedy.zip

Got feedback? Write a comment or contact me on Twitter where I'm @mkennedy or find me in any of these other ways.

Recommended Reading:

Here are some other blogs on this subject.


 
Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Join me in Boston to Talk about .NET!

Thursday, March 11, 2010 1:09:49 PM (Pacific Standard Time, UTC-08:00)
I'll be in Boston, MA on March 22 to teach an open enrollment course for DevelopMentor. If you want to learn about WCF, WPF, Silverlight, LINQ, Entity Framework, and more there is still time to sign up (note the date may no longer appear on the public calendar).

   http://www.develop.com/course/new-net3-net35-linq

Mention this blog post and you'll receive a discount as well.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

ASP.NET WebForms + Routing Video and Downloads

Wednesday, December 09, 2009 11:38:45 AM (Pacific Standard Time, UTC-08:00)

I recently did a webcast for DevelopMentor on using the routing framework introduced in ASP.NET MVC within ASP.NET WebForms based applications to build more modern websites without a major rewrite of existing web applications. The talk was called "Building Modern Websites with ASP.NET WebForms".

Here's all the related downloads. We had some microphone troubles so I want to apologize in advance for the sub-optimal sound quality.

    Watch streaming video (WMV HQ)

   Watch streaming video (WMV HQ)    Download WMV Video Listen to MP3 Streaming Download MP3

You can also download the slides and peepleocity.com sample website built during the presentation.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Webcast: Building Modern Apps in ASP.NET WebForms

Thursday, November 05, 2009 8:20:00 PM (Pacific Standard Time, UTC-08:00)

At DevelopMentor we have been running a bunch of free webcasts. Last month it was TDD and Agile. This month we are running 4 webcasts celebrating the announcements around .NET 4.0, Visual Studio 2010, and PDC 2009.

Join me Monday, November 23rd and register here:

     http://bit.ly/aspwebforms

We’ll talk about integrating ASP.NET’s routing infrastructure into existing an ASP.NET WebForms application. This allows you to build SEO websites with URLs like

      http://dotnet.ubbuzz.com/tag/.NET+4.0

while still taking advantage of all the productivity features of WebForms such as post-backs, controls, UpdatePanel, and so on.

We have room for a couple hundred more attendees so please register and be part of the fun. I promise lots of demos and somedisdainful comments about PowerPoint!

Share it with your friends (social, virtual, real, and other types) using the widgets below!

ASP.NET MVC: What’s that, you’d rather hear about ASP.NET MVC, not this creaky old WebForms stuff? That’s Brock Allen’s talk: http://bit.ly/intromvc

WF 4: Is WF 4 and visual programming your thing? Check out Maurice de Beijer’s WF 4 talk: http://bit.ly/meetwf4

New Parallel Extensions your thing: Check out Andy Clymer’s PFX talk. (link to follow soon).

Cheers,
Michael

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

TDD Space Invaders Video and Downloads

Wednesday, November 04, 2009 11:17:00 PM (Pacific Standard Time, UTC-08:00)

Recently Llewellyn Falco and I did a webcast for DevelopMentor where we demonstrated some TDD techniques and introduced Approval Tests. We let the audience choose our project and they chose Space Invaders. It was all great fun. Now the videos and MP3 streams are online and available for download.

Watch streaming video (WMV HQ)
Watch streaming video (WMV HQ)   Download WMV Video Listen to MP3 Streaming Download MP3

Be sure to check out the write-up we did afterward where we talked about the tools and gave you a chance to try it for yourself:

   TDD Space Invaders Write-up

You can also watch two other, higher level agile webcasts by Bill Nazzaro here:

   Agile Webcasts at DevelopMentor

Cheers!
Michael

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

TDD Invades Space Invaders

Wednesday, October 28, 2009 1:13:21 PM (Pacific Standard Time, UTC-08:00)
A joint post by Llewellyn Falco and Michael Kennedy

[Update: Get the videos and additional downloads for this webcast.]

As a follow-up to our "Avoiding 5 Common Pitfalls in Unit Testing" article we did a webcast where we took a problem from the audience and solved it live and unrehearsed on stage. These kinds of performances are always a risk but that's part of what makes them fun.

Of course, the question is could we have done it better? Here's your chance to try it for yourself (details below).

The Problem:

Our viewers chose to have us build the game Space Invaders. The first thing we had to do to sketch out a basic scenario we could implement. We started with a picture to remind what Space Invaders even was:



This was too big of a scenario for us to tackle in the allotted 40 minutes for programming. So then we started by creating a simpler scenario which we sketched out on the "whiteboard":


               Click for full size image.


Flushing Out the Scenario
:
In doing this, a couple of things were revealed about the game.

First, we wanted to make the tank and aliens all be the same size so we could put them on a grid. But then we saw that our bullet wouldn't fit that story, so we introduced the idea of relative sizes. We also realized that even though we drew the block, it was too complex for the first scenario and it would have to wait.

Notice that as we started writing the scenario in English, there are mistakes, irrelevancies, and problems with the order. This is OK. The thing to remember is that all of this was done for the sole purpose of creating a recipe for a scenario we could test. That scenario is the following:
[TestMethod]
public void TestSimpleKill()
{
   // 1. Create a 15x10 board.
   // 2. Place a 3x2 tank at 1x8.
   // 3. Place a 2x2 alien at 7x3 heading west.
   // 4. tank shoots
   // 5. advance 4 turns
   // 6. not won
   // 7. advance
   // 8. win
}

Now that we had the recipe, we could go about writing the code.

Here's your chance to play at home!

  1. Set your timer to 40 minutes.
  2. Create a new test project.
  3. Paste that method above.
  4. Translate the comments into code.
If you believe there's a better process, we invite you to try that as well.

We made it to step 4 during our presentation (download code below) and estimate another 15 minutes would have had the whole scenario done, tested, and well-factored.

Stories vs. Requirements (stories win):

We'd like to point out a couple of things about the story. First, it was quick to write the story. We did it in 5 minutes. Second, it translates well to code because it has behavior and objects working together. Let's compare that to the requirements that this story flushed-out.
  • Need a board
  • Boards should have width & height
  • Boards contain game objects
  • Game objects have a witdth, height
  • Game objects have the ability to move each turn
  • Aliens move either left or right each turn
  • Bullets move either up or down each turn
  • Bullets are 1x1
  • Tanks are 3x2
  • Aliens are 2x2
  • The game is not won until all the aliens are killed
  • The game is won if alll the aliens are killed
  • An Alien is killed if it is hit by a bullet
  • Tanks can fire
  • Firing with a tank creates a bullet going up from the space directly above the tank

Now we want to point out that this requirements doc is much hard to understand than our story. For example, if you were to add more requirements (e.g. an alien also shoots) is that easy to determine whether we have complete requirements? It also takes much more effort to create and especially to tell if it is complete. People aren't made to handle requirement documents well but we are story-telling machines. We embrace this in our coding techniques.

We'd also like to mention some of the tools discussed at the end.

For remote collaboration we use:

Skype (audio / video)
VNC for screen keyboard sharing
RDP (windows remote desktop) -- requires Windows 2003/2008 server for pairing.

Source Control:
TortoiseSVN
TortoiseGit

Developer Tools:
Resharper
CodeRush

Testing Tools:
MsTest (in Visual Studio Professional and up)
NUnit
NCover
TortioseDiff
Approvals Tests
Approvals Tests CodeRush add-in
Rhino Mock
TypeMock

If you try this scenario yourself, please leave a comment about your experience.

Download the code and slides from the webcast here:

   Code: TddWithLlewellynAndMichael.zip
   Slides dmtdd.pdf

Cheers -- Michael and Llewellyn

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Attend My Live, Free TDD Webinar with Llewellyn Falco and Myself at 10am Tomorrow!

Monday, October 26, 2009 3:29:46 PM (Pacific Standard Time, UTC-08:00)

[Update: See the follow-up post here: TDD Invades Space Invaders]

Tuesday, October 27, 2009 at 10am Pacific time Llewellyn Falco and I will be giving a live, unscripted, and no safety-net demonstration of Test Driven Development (TDD) as part of the DevelopMentor webinar series (this particular series is a 3-part series on Agile development).

We already have a bunch of attendees registered. But we have room for as many of you who are interested in agile and TDD. Sign up here:

   http://bit.ly/dm-tdd-m-and-l

In addition to core TDD techniques, you will see how an amazing technique and set of tools designed by Llewellyn called Approval Tests makes writing tests as simple as verifying an image or text file. Tired of writing 50 lines of test code for every 50 lines of production code but you still want the power of TDD? You need to learn more about Approvals and we'll demo that live tomorrow!

I hope to see you all online. Feel free to help me get the word out by tweeting this or shouting it (see icons below).

Cheers, Michael.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Article: Azure Storage

Wednesday, April 08, 2009 12:03:23 PM (Pacific Standard Time, UTC-08:00)

I recently wrote an article for DevelopMentor's Developments newsletter entitled Azure Storage. Read it at the DevelopMentor website here:

http://www.develop.com/content/newsletters/aprilazure

I've republished here for my readers. Enjoy!


Developments: Azure Storage

by Michael Kennedy

[Listen to this article as a podcast: Azure-Storage-Article-Kennedy.mp3]

October 27th 2008, Los Angeles CA - It's 9 AM and Microsoft is hosting PDC (their most forward looking developer conference). Ray Ozzie and company are introducing Windows Azure: A new platform which is their first foray into the nascent world of large-scale utility computing. This scalable and reliable platform-as-a-service functionality is commonly referred to as "Cloud Computing" because it runs somewhere out there on the Internet.

Computing platforms that rival the reliability of the utility grids (e.g. electric and gas) which we daily take for granted have long been the stuff of dreams.

A few companies have realized this dream - Google and Amazon come to mind as a couple of the rare exceptions who have accomplished this goal. These companies' web properties seem to handle unbounded amounts of traffic with zero down time. The data centers, redundancies, software engineering and operations know-how required to make this happen are exceedingly expensive. Some reports have Google spending over $2.4 billion (that's 2,400 million dollars) on data centers in 2007 alone.

Prior to large-scale cloud computing efforts (circa 2005), most of us could only dream of such scalability and reliability. Today we have at least three highly reputable companies offering some kind of pay as you go cloud computing platform - Microsoft, Amazon, and Google.

Microsoft's Azure is a new comer to the industry. But for .NET developers, it is not to be ignored. Azure allows you to use your existing skills to build essentially the same .NET applications you are familiar with and "deploy them to the cloud."

These scalable, reliable, and geographically-replicated applications that run on Azure depend on data of course. Virtually all applications we write will be nothing without their underlying data. But if we simply use the tried and true methods of data storage such as the file system or a (single) database server our data is not all that scalable or reliable. Because we cannot have a scalable and reliable application without data, we need a new mechanism for storing and accessing data from our Azure applications.

Enter Azure Storage

Azure storage is the storage component of the Azure platform. It is actually three data services in one:

  • Blob Storage - stores unstructured data essentially as a file, limited to 50 GB of data per blob.
  • Table Storage - stores structured data that is somewhat like a database. For full database capabilities there is a high level feature called Sql Data Services (SDS).
  • Queues - provides interprocess communication functionality between various web and worker roles in your hosted services or even applications running outside of Azure. Queues can pass small xml or binary messages - less than 64 kb per message.

In this article, we will cover just the basics of the three storage services of Windows Azure. I want to give you a sense for what it's like to program against Azure Storage. At the base level all access to Azure Storage uses pure REST APIs. This means that you can access it from any HTTP enabled platform / language. For example, to download the blob data called "config.xml" in the container called "settings" for the Azure project "kennedy" you would simply issue a GET to the Uri:

   http://kennedy.blob.core.windows.net/settings/config.xml

To save data in a blob you do HTTP POSTs and PUTs in a similar fashion. However, real life is full of edge cases, error handling, security, and serialization which makes the pure HTTP model error prone. Thus, a sample library serves as the de facto .NET API to Azure Storage and ships with the Azure SDK. It is called StorageClient and can be found in default installs here:

   C:\Program Files\Windows Azure SDK\v1.0\samples\StorageClient

We will examine working with each of the storage services from the perspective of the StorageClient library - but keep in mind that ultimately this library is a wrapper around a basic and open RESTful API.

Setting The Stage: The Sample Application

To explore Azure Storage I have written a simple photo sharing distributed application. These set of applications allow users to upload photos to a photo sharing site. These photos must be reviewed and approved by moderators of the site. Once approved, the general public can view and interact with the photos. For a concrete example, you could imagine writing a distributed version of the wallpaper sharing site InterfaceLift and deploying it on Azure in this fashion.

You can download the sample application and follow along if you want to see the full source code and try it out yourself. Just be sure to start the Development Storage utility that comes with the Azure SDK before running the application.

Our distributed application consists of three parts.

  • The Uploader: A Windows Forms application that lets contributors upload images to the site.
  • The Reviewer: A Windows Forms application that lets moderators view image submissions and either approve or reject them.
  • The Website: An ASP.NET website for viewing the photos - this is our public facing application.

A typical use case might be as follows (see diagram below).

  1. We upload a photo submission with our uploader application. The photo is uploaded to Azure blob storage and a message is sent via an Azure Message Queue to all available reviewer applications. Additional information about the submitter is associated with the photo in Azure table storage.
  2. The reviewer application watches the message queue for new messages. When one arrives, the photo is added to a list of pending submissions. The reviewer can either reject (delete) the submission or approve it - move it to a permanent blob storage location where it will be publicly viewable.
  3. Users visit our website and can view all approved photos. This list will change in real-time because it is driven by the reviewer application. The web application simply pulls all photos from the approved photo container in Azure blob storage.

Saving Data: Creating Azure Blobs

To save data to Azure Blob Storage, you must realize blob storage follows the ACE pattern (Authority, Container, Entity) to describe a blob. Authority is simply your Azure solution name. Containers are analogous to folders. And entities are analogous to files.

The listing below is essentially the code that runs when the uploader application uploads a pending image submission to blob storage.

Listing 1:

Sending Notifications: Azure Queuing

In addition to uploading the image to the pending images container in blob storage, we will send a message to a message queue to notify any active or future reviewers of the new submission.

Listing 2.

Saving (More) Data: Structured Storage and Azure Tables

Finally, for the upload application, we must also save some information about the contributor. In Azure Storage we have two reasonable places to store this information.

First, we could save this information directly in blob storage as meta-data associated with the blob itself. This is straightforward and easy. But there is a big limitation: information in this meta-data is not queryable. Suppose I want get all images associated with a single contributor. There is no way in Azure Blob Storage to say give me all the blobs with this filter on the meta-data. You would have to pull the properties of every blob and do the comparison client-side. That's tantamount to filling a DataSet with "SELECT * FROM PendingImages" and it's a bad idea.

Instead we will use the third type of Azure Storage: Azure Table Storage. Table Storage allows us to store data with up to 256 properties and query this data as if it were a database. It is exactly what we need for the contributor information. However, you must realize this is not a database. A better mental picture is a durable collection of Dictionary object (as in Dictionary from System.Collections.Generics) with querying built on top. I say this because there is no schema or relational constructs in Azure Table Storage. If you need that, then you'll want Sql Data Services - a service on top of the core Azure platform.

The code to add an entry to Azure Table Storage does not fit into a single method as it's driven through the interaction of several classes we must define. Azure Table Storage can be accessed via ADO.NET Data Services (client-side) and this is the method we will use.

First we'll define a client-side schema for our entry by creating a class called Contributor which derives from the class TableStorageEntity (from the StorageClient library).

Listing 3.

Additionally we must define the tables and queries available to ADO.NET Data Services by created a class derived from TableStorageDataServiceContext and we do that below. We simply have one table called Contributors.

Listing 4.

With those two items in place, we can insert a "row" into Azure Table Storage as follows:

Listing 5.

As for querying Azure Table Storage that is very straight-forward. Because we are using ADO.NET Data Services, querying can be done via LINQ as in "from c in svc.Contributors select c.Name". Ultimately ADO.NET Data Services is also built on a RESTful API so this translates to the underlying HTTP REST calls. Alternatively, you can use that REST API directly from .NET or any other platform.

Waiting on Queues: The Reviewer Application's Code

Next, let's look at how we monitor and pull messages from Azure Queuing. Ultimately we must poll the queue using a RESTful HTTP request. But the StorageClient resurfaces this to us as simple events.

Listing 6.

We won't cover how we move a blob from the pendingImages blob container to the approvedImages blob container which happens when a reviewer approves an image. You can look at the sample to see how that is done.

Ultimately It's About the Website

Finally, let's look at the web application that actually displays the approved images. We don't do anything fancy such as paging or error handling that you'd see in a real application. But this will give you a good idea how to work with the blob data as a collection.

Here we'll create a BlobStorage object and access the BlobContainer approvedImageContainer as we have been in most of the listings. But then instead of saving or reading blobs, we use the ListBlobs method to simply list all the approve images in that container. In order to show the images on our webpage, we just use the BlobProperties.Uri and directly reference that in our HTML. Our ASP.NET application does not touch the data. Rather the consumers (IE, Firefox, Chrome, etc) of the HTML pull the image data directly from blob storage as they would from any web server.

Listing 7.

Now you have a good idea of the concepts and motivation behind Azure Storage. You have seen some typical usages of each of the three storage features: blob storage, table storage, and queuing. Our samples made use of the sample storage API library called StorageClient. Underlying this library we saw that Azure Storage is entirely accessed via RESTful APIs.

Want to get started? Visit http://www.azure.com and choose "Try It Now" to register for a CTP Azure account. You'll need to download the various SDK's listed on that same page. They will install the Visual Studio projects required for working with Azure as well as the Development Storage and Development Fabric so you can develop and debug your applications before deploying them to the cloud.

If you want some intensive, expert-lead training on Azure and associated .NET 4.0 topics be sure to contact DevelopMentor. Or call 800.699.1932 to find out what classes we have available today.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

RESTful Web Services with WCF Screencast

Tuesday, March 24, 2009 11:02:03 AM (Pacific Standard Time, UTC-08:00)

I recently got the chance to record a screencast discussing REST-oriented web services in WCF. If you're interested in WCF you should definitely check it out because WCF and REST make an awesome combination. kennedy-wcf-rest-video-screenshot.png
   WCF-REST-Kennedy-Peepleocity.wmv 35 MB (WMV HD)

I cover building WCF services using REST princples, the WebGet and WebInvoke attributes, working with the SyndicationFeed & Rss20FeedFormatter classes, and configuration-free WCF hosting in IIS.

You can also download the source code of the project built in the screencast.

Finally, if you're willing to do without video you can download just the audio as an MP3.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Debugging the Future (Advanced .NET Debugging) Video Presentation

Tuesday, March 03, 2009 3:14:04 PM (Pacific Standard Time, UTC-08:00)
My esteemed colleague, friend, and fellow instructor at DevelopMentor Jason Whittington gave a great presentation on advanced .NET debugging recently at the Oklahoma City Developer's Group. They luckily recorded it on video and published it on their website so that it may "live on in the Google".

   Debugging The Future: The Video by Jason Whittington

 
kick it on DotNetKicks.com

If you want to debug .NET applications right down to their memory footprint, this talk is for you.

If you like this type of presentation, be sure to check out the classes we offer at DevelopMentor.
Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

ClickOnce Deployment for Unmanaged Code (C++, VB6, etc)

Monday, August 25, 2008 7:53:30 PM (Pacific Standard Time, UTC-08:00)

ClickOnce is a great deployment model for many Windows applications built with the .NET Framework. Too bad it isn't supported for C++, VB 6, or other technologies. Or is it...

Surprisingly, you can deploy your unmanaged apps with ClickOnce. You just need a tiny .NET app to get it started.

Here's how it works:

  1. Take your existing C++ project.

  2. Add to the solution a .NET console application.

  3. Change the project settings on the console application to "Windows Application".

  4. Write the following code for your Main method of your .NET launcher application:
    static void Main(string[] args)
    {
        try
        {
            Process.Start( "TheRealApp.exe" );
        }
        catch ( Exception x )
        {
            string msg = "Error launch application:\n\n" + x;
            string cap = "Error Launching Application";
            MessageBox.Show( msg, cap, MessageBoxButtons.OK, MessageBoxIcon.Error );
        }
    }
    
  5. Set the build path of your C++ app to be in the main folder for your .NET application.

  6. Add the C++ app and its libraries (if any) as existing items in the .NET app.

  7. Change the build action to "Always Copy" as shown here:




  8. Then you publish your .NET application and when it runs, TADA!, the C++ app is deployed, versioned, and kept up to date as well.



If you want to try it yourself, you can run this sample application here:

   Run Michael's Useless C++ App via ClickOnce...

You can also download the source.



Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Dynamic Data Access with LINQ

Wednesday, July 02, 2008 10:08:00 AM (Pacific Standard Time, UTC-08:00)
When I talk about LINQ people often ask whether it’s possible to have dynamic queries with LINQ. An example of this is presenting the user with a UI that allows them to optionally filter by some criteria or other and sort by some criteria. This was straightforward if you built-up a SQL string in code.

The compiled, static nature of LINQ makes dynamic queries appear difficult at first. They are not and this video and sample application shows you how to accomplish just this.


                        Screen shot of sample application

You can download the source code here:

           Kennedy-Dynamic-Linq.zip (88.34 KB)

And you can watch the screencast video here:

          Kennedy-Dynamic-Linq.wmv (35.7 MB)

Of course, if you like this post, please kick it: kick it on DotNetKicks.com


Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Top 5 Most Important Sessions from Mix 08 (with Video)

Friday, April 04, 2008 6:16:07 AM (Pacific Standard Time, UTC-08:00)
You may have heard that the sessions for Mix 08 were posted online and you can view or download them until your heart is content.

Mix 08 is becoming an increasingly important event in the .NET space. I spent a fair amount of time watching the sessions in the evenings. I was teaching a class and couldn't attend myself.

So who wouldn't want to curl up with a cold beer, a warm laptop, and a bunch of .NET presentations? That's what I want to know. But, maybe you don't have time to go through all 89 sessions? Don't despair, I'll distill them down for you.

BTW, I'd love to link directly to them, but SilverLight and Flash are broken web metaphors (you can't link their content) so you'll have to find them at http://sessions.visitmix.com/.

Without further ado, I present to you my top 5 most important sessions from Mix 08:

  1. T22 - Developing ASP.NET Applications Using the Model View Controller Pattern
    Scott Hanselman

  2. T01 - Creating a RESTful API with Windows Communication Foundation
    Aaron Sloman and Haider Sabri

  3. UX03 - The Back of the Napkin: Solving Design Problems (and Selling Your Solutions) with Pictures
    Dan Roam

  4. T11 - What's New in Windows Presentation Foundation 3.5
    Rob Relyea

  5. T26 - Building Applicaitons and Services with.NET Framework 3.5
    Justin Smith
Of course the keynotes were great too - but they don't really belong in this category.

So get out there and learn something! :) And kick it if you like it: kick it on DotNetKicks.com

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

.NET 3.5 Brings Major (Undocumented) Changes to ThreadPool

Wednesday, February 06, 2008 9:33:32 PM (Pacific Standard Time, UTC-08:00)

It was all going so smoothly. Jason Whittington, Mark Smith and I were teaching the big DevelopMentor event here in Los Angeles (Guerrilla.NET) when my presentation on the ThreadPool took a nose dive. It started with a great joke involving Wilson (the volleyball from Cast Away).

Wilson and I built an application to compute a multiplication table where each computation was (artificially) slow. To speed it up we threw it at the thread pool using delegate.BeginInvoke. We figured that the ThreadPool would allocate 25 or so threads and the table would display quickly. Here's the expected output - pretty much the same thing we've seen since about .NET 1.0:

    

Each color represents the thread that did that computation.

For the last 7 years, the behavior has been that as the ThreadPool was overloaded, it would steadily start up new threads at the rate of one every 500 milliseconds until it hits its upper limit (typically). Using Performance Monitor (perfmon) we can watch the thread pool adding threads. It usually looks something like this:

    

Much to our surprise we saw completely different behavior. The thread pool added the first 15 or so threads quickly (as expected) but then stalled. New threads were not created every 500ms, instead they were added at increasingly long intervals. My demo took almost twice as long to run as it had the last time I did this demo a few months ago.

Jason, Mark, and I took this code, ported it back to .NET 1.1 and ran it side-by-side with .NET 3.5 and here's what we saw (blue = 3.5, red = 1.1):

    

As of .NET 3.5 the upper limit of the ThreadPool was increased: Knew that.

But, it appears that v3.5 of the CLR changes the policy for adding threads to the thread pool. Rather than adding threads regularly when under load the thread pool uses a logarithmic backoff. My colleague Jason Whittington remarked that this behavior looked similar to the behavior of the thread pool in "Rotor" (the shared-source version of the CLR). We speculated that this backoff algorithm makes sense given the new 250-thread per CPU maximum - it would take a long time to reach that if the runtime waits longer and longer to start a new thread. The 250-thread limits makes it less likely that your application will deadlock the thread pool, and the exponential backoff algorithm keeps the thread pool from creating too many threads too quickly.

Why should you care? Usually you won't, but it could have dramatic impact if you count on that behavior. For example, in our contrived case, .NET 1.1 ran about twice as fast as .NET 3.5 ( ! ):

    

Here's the program and source code (trimmed down to run in both .NET 1.1 and 3.5).

    Math.zip (6.38 KB)

Needless to say, Wilson and I felt kinda stood up.

      kick it on DotNetKicks.com

Please see the follow up post on Breaking changes and now there is screencast, movie version as well.


Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Agile Object Oriented Design Talk at Central Jersey Group Tonight

Thursday, October 19, 2006 5:42:00 AM (Pacific Standard Time, UTC-08:00)

I'll be speaking tonight at The Central New Jersey .NET User Group (http://www.njdotnet.net) on one of my favorite topics: Five Fundamental Object Oriented Design Principles for Agile Development.

Thanks to Jason Beres (http://www.geekswithblogs.net/jberes) for inviting me to speak at his user group and welcoming me to New Jersey.

Once you've attended the talk, you might be interested in downloading the slides and sample code:

    http://www.MichaelCKennedy.net/Talks/Downloads/NJdotNet/AgileDesign.zip

You also might want to look into some of the tools I was using in Visual Studio. See an older post on my tricked out Visual Studio.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Great Meeting at OCVBUG and OC C# Last Night

Wednesday, May 10, 2006 4:31:51 PM (Pacific Standard Time, UTC-08:00)
Thanks to Reza Madani and Mike Vincent for having me speak at the combined meeting of OC VBUG and OC C# last night. It was a packed house which was wonderful, but I felt bad for folks who had to stand in the back for 2 hours.

The audience participation was great. Thanks to everyone who asked questions or had comments.

You can download the slides and samples from my website here:

http://www.michaelckennedy.net/Talks/Downloads/OCVBandCSharp/AgileDesign.zip

Many attendees also noticed the tricked-out Visual Studio I was using during the presentation. That was from CodeRush and Refactor! both highly recommended. You can check those out here:

http://www.devexpress.com/Products/NET/CodeRush/
http://www.devexpress.com/Products/NET/Refactor/

and DevExpress has some very interesting screen-casts demos here:

http://www.devexpress.com/Products/NET/CodeRush/Training.xml

Also I was using the new Vista programming font Consolas, which you can download here:

Consolas Font Pack for Microsoft Visual Studio 2005


Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

OOD at San Diego .NET Developers Group

Tuesday, May 02, 2006 4:28:23 PM (Pacific Standard Time, UTC-08:00)
Thanks to David McCater for hosting my Agile Object Oriented Design talk at San Diego .NET Developers Group on May 2nd. I met some great guys and thoroughly enjoyed talking about agile and OOD topics with everyone.

You can download the slides at:

http://www.michaelckennedy.net/talks/downloads/sddotnetdev/agiledesign.zip

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Back to Basics: Agile Object-Oriented Design

Monday, April 10, 2006 7:06:40 AM (Pacific Standard Time, UTC-08:00)

Dave, over at extremeplanner.com, wrote a nice summary of the talk I gave at XPSD. I completely agree with him that "a review of these fundamentals [is] refreshing." As you can see, I'm doing my part to spread the word:

Many pundits and authors in the software industry are busy promoting the next big product/feature/etc that they seem to forget that these things only provide second order benefits as compared to the more "fundamental" ideas such as good object oriented design. For example, looking through some of the user groups' upcoming meetings you'll find topics like:
  • Looking at some Cool New Features of ASP.NET 2.0: Managing Membership and Role Management
  • Leveraging the New Windows Mobile APIs
  • Visual Studio Team System
  • Closer to the Data: Using Managed Code in the Database with SQL Server 2005

Lots of neat, whiz-bang features, but I would contend that 90% of the audience would be much better off if they had heard a good talk about solid OOD or TDD or this list goes on. It's been my experience that much of these old (5-15 years), tried and true fundamentals are unfamiliar to most developers.

So here is my call to action:

If you are a speaker or author, reach for something big for your next topic. Yes it's easier to write/talk about the details of the next version of feature X. But you'll do your audience a service if you reach higher, think bigger, and go for something "fundamental."

If you are attending user groups and code camps, ask for and expect something more than a talk on the of latest version of product X.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Upcoming Talk in San Diego (Tentative)

Wednesday, April 05, 2006 10:39:00 AM (Pacific Standard Time, UTC-08:00)

I am tenatively scheduled to speak in San Diego, CA at the “San Diego .NET Developers Group” (http://sddotnetdg.org/). We haven’t arranged a date yet, but I think it will be around late spring / early summer.

This time I will again be presenting a talk on object oriented design and agile development, probably a slight variation on the general topics from my other talks.

I’ll provide more details as they become available.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

Upcoming Talk in Orange County

Sunday, April 02, 2006 10:02:33 PM (Pacific Standard Time, UTC-08:00)

I’m very pleased to announce that I’ll be speaking in Orange County, CA on May 9th from 6pm onward.

I’ll will be presenting a talk on object oriented design and agile development at a joint meeting of the “Orange Country C# Developers Group” (http://sddotnetdg.org/) and the “Orange County VB.NET User Group” (http://www.ocvbug.org/).

I encourage anyone in the Anaheim / L.A. area interested in the interplay between object oriented design and agile software development to attend.

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

XPSD Object Oriented Design Talk

Saturday, April 01, 2006 9:45:57 PM (Pacific Standard Time, UTC-08:00)

[NOTE: This recent post refers to a much older (March 3, 2006) event.]

I want thank Carlton Nettleton and June Clarke for inviting me to present my “Five Fundamental Object Oriented Design Principles for Agile Development” talk at XPSD this past March. The audience participation was great and I really learned a lot myself!

You can download the slides and code samples here:

http://michaelckennedy.net/Talks/Downloads/XPSD/AgileDesign.zip

 

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post

January SoCal Code Camp Talk

Saturday, April 01, 2006 9:40:58 PM (Pacific Standard Time, UTC-08:00)

[NOTE: This recent post refers to a much older (Jan 21, 2006) event.]

I want to thank everyone who attended my talk at the first Southern California Code Camp. I really enjoyed speaking there and I got the sense that many of you got something from the talk. In case you weren’t there and are interested, here’s the abstract:

Five Fundamental Object Oriented Design Principles for Agile Development

This session will present five object oriented design principles that facilitate agile software development. These general design principles promote the creation of testable, maintainable, and reusable software. The principles include the Open Closed Principle and the Liskov Substitution Principle. The interaction between Agile Development and these principles will be demonstrated using several code samples.

 You can download the slides and code samples here:

http://michaelckennedy.net/Talks/Downloads/SoCalCodeCamp/AgileDesign.zip

 

Tweet this Follow me on Twitter Post this to dotnetshoutout.com Digg this Submit this to Stumbleupon email this post


Just a site note: I'm doing my part to rid the world of IE 6. Visit this site with IE 6 and you'll get a shameful message telling you to "Stop Living in the Past".