The importance of idempotent services

Introduction

We all know the importance of performing a unit of work (UoW) in a transaction scope so that if anything goes wrong we can rollback the actions that have taken place and be sure that the state is exactly how it was before the UoW took place, this is were we can use a messaging framework (i.e. MSMQ) that can support transactional capabilities using a transaction coordinator (i.e. MSDTC) so that when we receive a message perform some work, but when saving to the DB there is an issue (i.e. Network problem, transaction deadlock victim) we know that the message received will be put back on the queue and any messages that have been attempted to be sent will be returned (to prevent duplicates):

public void DoWork()
{
    using (var scope = new TransactionScope())
    {
        var command = MsmqInboundWork.NextMessage();
        // *** processing here ***
        DbGateway.Save(workItem);
        MsmqOutboundResult.SendMessage(new ResultMessage { CompletedAt = DateTime.UtcNow; });

        scope.Complete();
    }
} 

If this happens we have options (i.e. retry for a limited amount of goes and then put the message into an error queue) however we don’t always have the option of using a reliable/durable/transactional transport layer such as messaging, for instance if we are integrating with a 3rd party over the internet they are going to provide usually a HTTP based transport layer such as SOAP/REST or like in my case recently integrating with an internal legacy system that only provides a HTTP transport layer (SOAP webservices was the only option in my case).

This post will demonstrate how we can work with services exposed on unreliable transport layer to perform actions but still not end up with duplicates by making the operations idempotent.

What is an idempotent operation?

This basically boils down to being able to call a function with the same arguments and have the same result occur, so if we have a method:

public int Add(int first, int second)
{
    return first + second;
} 

This method is idempotent because if I called it multiple times with Add(2,2) it will always return 4 and there are no side effects, compare this to the next method (demo only!):

public void DebitAccount(string accountNumber, decimal amount)
{
    var account = FindAccount(accountNumber);
    account.Balance -= amount;
    // account saved away
} 

If I call this multiple times like this DebitAccount("12345678", 100.00m) the client of that account is going to be more out of pocket each time a call is made.

How does this relate to my webservice method?

You may be thinking to yourself we have a system running that exposes a webservice with a method to perform a similar action to the example above or that you make a call to a 3rd party to perform an action (i.e. send an SMS) and so far we haven’t had any problems so what’s the big deal!? You’ve been lucky so far but remember the first fallacy of distributed computing

The network is reliable

Once you get network issues this is where everything goes wrong, this may not be a big deal you may decide to retry the call and the user ends up with another SMS sent to their mobile however if you’re performing a journal on an account this is a big problem.

If we look at a typical call to a webservice, in this case we are going to use the example of a façade service of a legacy banking system:

Account Service
fig 1 – If all goes to plan

In this case everything has gone ok and the client has received a response back from the account service so we know that the account has been debited correctly, what if we have the following:

Account Service
fig 2 – Network issue causes loss of response to client

Now we have a problem because the client has no way of knowing whether the debit was performed or not, in the example above the journal has taken place but the machine running the account service could have a problem and not get round to telling the banking system to perform a journal.

Fixing the problem

We established in the last section that the client has no way of knowing whether the debit took place if we do not receive a response from the account service so the only thing we can do would be assume that it did not take place and retry the service call given the scenario from fig 2 this would be very bad as the debit did take place so we would duplicate the debiting of the account.

So we have a few options of how we can deal with this issue:

  1. We don’t perform retries on the client side automatically instead we handle the situation manually and could perform a compensating action before attempting a retry (i.e. perform a credit journal)
  2. We check on the account service whether we have already had a debit for the chosen account and amount in a specific time frame and if so we ignore the call (this would not work in this example as we may have a genuine need to debit the account twice within a specific time frame)
  3. Have the client send something unique (i.e. GUID) for the current debit that you want to take place this can then be used by the account service to check if we have already performed the journal associated with this request.

The last 2 options are to make the account service idempotent my recommendation would be to use the last option and this is the option I will demonstrate for the rest of the post.

Ok so the first change we should make is to add a unique id to the request that gets sent, it is the responsibility of the client application to associate this unique id with the UoW being carried out (i.e. if the client application was allowing an overdrawn account be settled then the UoW would be a settlement and we can then associate this id to the settlement) so the client needs be changed to store this unique id.

The next change is for the account service to perform a check to make sure that we have not already performed a journal for the id passed to it, this is accomplished by storing each id against the result (in this case the journal reference) once the journal has taken place and querying it as the first task, if we find the id then we simply return the journal reference in the response back to the client.

If the id is not found we must not have performed a journal so we need to make the call to the banking system as we were before and then store away the id and the journal reference and return the response back to the client.

Now that this is in place we can have the client perform retries (within a certain threshold) if we receive no response back from the account service safe in the knowledge that we won’t get duplicate debits from the account, here is a sequence diagram to outline the new strategy:

New idempotent account service

Ah the joys of distributed architecture!

Advertisements

Adding Controls to a Server Control

Usually looks like this:

 
var row = new TableRow(); 
var cell = new TableCell()
    { 
         Height = Unit.Pixel(30) 
    }); 
cell.Controls.Add( new Button() 
    { 
        Text = "Go",
        CommandName = "Go",
        ID = "btnGo" 
    }); 
row.Controls.Add(cell); 

How about this instead:

 
var row = new TableRow(); 
var cell = row.AppendControl(x => 
                  { 
                      x.Height = Unit.Pixel(30); 
                   });
cell.AppendControl(x => 
    { 
        x.Text = "Go";
        x.CommandName = "Go";
        x.ID = "btnGo"; 
    });
 

Implemented with this:

 
public static ControlExtensions 
{ 
    public static TControl AppendControl(this TControl control, Action initialization) 
        where TControl : Control, new() 
    { 
        var controlToAppend = Activator.CreateInstance();
        intialization(controlToAppend);
        control.Controls.Add(controlToAppend); 
        return controlToAppend; 
    } 
} 

I think this reads better especially if you end up appending lots of child controls, thoughts or any other solutions?

Using Windsor for Controller Creation in ASP.NET MVC

Given it’s reached v1.0 I finally decided to do some work with the ASP.NET MVC framework and what better way than to take the Issue Tracker which currently runs on Rhino Igloo and refactor it to run on ASP.NET MVC, so expect the next few posts to be related to my findings of how to get things done with it.

First thing is that Rhino Igloo out of the box handles controller creation and uses Castle Windsor to do this ASP.NET MVC by default will use it’s own controller builder, because of this it expects your controllers to have a default no-arg constructor this is no good as I have controllers that need their dependencies passed in, so we need to replace the default controller builder with a version that we can hook up to Castle Windsor.

Now I could go and write my own version this is because the framework is very flexible and was built to allow you to plugin your own  objects, in this case an object that implements IControllerFactory will suffice however why re-invent the wheel when the guy’s over at MvcContrib have done most of the work for us.

Getting the MvcContrib Objects

Head over to to the MvcContrib project on codeplex and grab the extras release, now copy the MvcContrib.dll and MvcContrib.Castle.dll into your external references folder and add a reference where you will be doing the app startup.

Setting the Controller Factory

Inside your global.asax or wherever you do app startup you need to add the following code:

ControllerBuilder.Current.SetControllerFactory(
                new WindsorControllerFactory(this.Container));

Notice that I’m passing the current instance of the windsor container which I get for free because I’m using the UnitOfWorkApplication base class from Rhino.Commons, you will probably have some custom way of managing your container, in which case it just needs to be passed in to the constructor of WindsorControllerFactory.

Wiring up the Controllers

Next we need to make Windsor aware of the controllers, I use Binsor for my Windsor configuration this gives me a nice way to autowire new controllers to Windsor without making any changes:

for type in AllTypes("IssueTracker.Controllers"):
    continue unless typeof(IController).IsAssignableFrom(type)
    component type.Name, type:
        @lifestyle="transient"

Gotta love the conditional expression support Boo has to offer it almost reads like a sentence 🙂 You could also use the XML configuration or programmatic configuration to configure your controllers, and that’s all there is to it!

Kudos to the ASP.NET team for having the foresight to allow these kind of extensions to be plugged in so easily and also to the MvcContrib contributer’s for creating this and support for other IoC containers out there.

Gridview binded to LinqDataSource not showing association values

I’m working on a report screen for IssueTracker.NET and because this is going to be used purely for reporting on data this is a perfect match for leveraging the GridView & a LinqDataSource  to bind to, this will then give me dynamic filtering and paging out of the box splendid!

However after I had added the GridView & LinqDataSource wired the all up and ran the page there was a problem in that my association objects weren’t being displayed, hmm.. so I had code that looked like this:


    
        
    

Where User is a property on the current binded object my association object, so I went and re-checked the Linq to Sql generated code and sure enough the User property existed and was ok, so I re-ran with the debugger for the line above the Eval("User") returned null… then it struck me Lazy Loading perhaps the Eval method under the hood may not trigger a lazy loaded property to be retrieved (and in which case we don’t want lazy loading on this property anyway to cut down on DB trips).

So after some looking around it turns out you can control creating and assigning the DbContext object to the GridView via the ContextCreating event, so I subscribed to this and added the following:

protected void IssuesDataSource_ContextCreating(object sender, LinqDataSourceContextEventArgs e)
{
    var db = new IssueTrackerDbContextDataContext();
    db.ObjectTrackingEnabled = false;
    var loadOptions = new DataLoadOptions();
    loadOptions.LoadWith(x => x.User);
    db.LoadOptions = loadOptions;
    e.ObjectInstance = db;
}

Here is where you create your own custom DbContext object and set it how you want it, the important part for the above was the definition of the LoadOptions, don’t forget to assign it to the GridView via the e.ObjectInstance property.

Once I had this in place I did a re-run and I had my value displayed, so hopefully this will help people in the same boat.

Spaghetti code? No I dont think so

I just wrote this while working on the IssueTracker.NET project:


<tr>
<th>
        - </th>
</tr>
<tr>
<td>
       </td>
</tr>

Now some will look at this and be thinking eurgh!… why is he writing classic ASP style spaghetti code and that’s the point, since when did writing code in the page == spaghetti code? I believe that spaghetti code originated because of the shortcomings of ASP 3.0 not allowing us to separate UI concerns from other concerns, but we don’t have this problem in ASP.NET we can quite nicely have our seperate objects in other layers deal with persistence, domain etc… and the code on our pages can deal just with what they are supposed to the UI.

This is why when I look at the code above I don’t think eurgh! because it’s not dealing with anything other than concerns of the UI and in the case above displaying any comments assigned  to the current issue and in my opinion it reads quite well.

Finding Web Controls by Type

If someone runs into the same problem I was having which was trying to find controls of a certain type they can use this extension method which extends the Control object:

public static class ControlExtensions
{
    public static ICollection FindControlsByType(this Control control)
        where T : Control
    {
        var coll = new List();
        if (control == null) return coll;

        FindControlsByType(control.Controls, coll);
        return coll;
    }

    private static void FindControlsByType(ControlCollection controls, ICollection coll)
        where T : Control
    {
        foreach (Control current in controls)
        {
            if (current.HasControls())
                FindControlsByType(current.Controls, coll);

            if (current is T)
                coll.Add(current as T);
        }
    }
}

Remember as with all extension methods if you are using them in an inherited class you will need to put this before the name of the method otherwise it won’t be seen in intellisense.

ASP.NET Ajax weirdness

I just ran into a strange issue when trying to serialize one of my objects over a webservice that’s called using Ajax, the weirdness isn’t what you need to resolve the issue but instead how it manifests itself in such a convoluted way and goes like this:

public class Foo
{
    public Foo(Bar bar)
    {
    }

    // some properties here...
}

Say I have the above class and then the following in my webservice:

[WebMethod]
[GenerateScriptType(typeof(Foo))]
public Foo RetrieveFoo(int id)
{
// perform retrival here...
}

Now if I have some javascript on the client that tries to make a call to this webservice it will fail because the javascript proxy to the webservice just doesn’t get created so it ends up being null, this is not good cos’ if you have been making various changes you have no way of knowing why the proxy class isn’t being generated so you have to spend time tracking down what could be causing the issue.

Turns out it’s because my Foo class needs to have a public empty constructor but it took a while to track this down, couldn’t a better way of alerting the issue to the developer been accomplished.