Compose & Sequence created using recursion & imperative

Introduction

As I’m making my way further down the rabbit hole of the functional programming paradigm, I have found myself at the topic of recursion and wanted to do a post demonstrating how we can take an imperative operation and turn it into a recursive one but also using a real world example of something useful, hence why I decided that implementing compose & sequence[1] would be a good demonstration, first let’s get acquainted with these 2 functions.

Brief Introduction to Compose & Sequence

In functional programming you typically end up with functions calling other functions that then may also be called by another function[2], so if we had functions a, b and wanted to join them into a function called c:

var c = a(b);

This becomes harder to read as we have more nested calls, ideally we would like to pass the results of one function into another this is where compose and sequence come in, compose works from right to left and sequence left to right:

var c = compose(b, a);
// or
var c = sequence(a, b);

Typically compose is used as a function that takes 2 functions like the above in which case the implementation is trivial:

var compose = function (a, b) {
  return function () {
    return a(b.apply(b, arguments));  
  };
};

It returns a closure and applies on the second function using the arguments passed in and then uses the result of this call to pass to the first function and return the results. We can execute like this:

var print = function (val) { console.log(val); };
var double = function (val) { return val * 2; };

var display = compose(print, double);
display(10);

However in JS we can do much better than this because of its varargs support via the special arguments variable[3] so what I want to do next is go through how we can implement the compose & sequence functions using both an imperative approach or by using recursion.

Imperative Approach

First let’s start by creating the sequence function:

var sequence = function (/* funs */) {
  var funs = Array.prototype.slice.call(arguments);
  var len = funs.length;
     
  return function (/* args */) {  
    var args = funs[0].apply(funs[0], arguments);
    for (var i=1; i < len; i++) {
      args = funs[i](args); 
    }
        
    return args;
  };
};
  1. The sequence function does not specify the arguments explicitly as instead we are going to take multiple functions[4]
  2. The functions supplied are extracted from the arguments variable
  3. A closure is then declared that will be returned to the caller
  4. The argumentsfrom this closure are then used to call the first function supplied
  5. The results of this function call are stored in args variable
  6. The remaining functions passed are then called using a for loop each time the args variable is being mutated to contain the results of each function call
  7. Once the functions have all been executed the value stored in args is returned

The only difference between compose and sequence is that compose runs from right to left so if we can take all the existing code and just make this one bit of behaviour configurable we will save ourselves having to duplicate code:

var stitch = function (reverse) {
  return function (/* funs */) {
    var funs = Array.prototype.slice.call(arguments);
    var len = funs.length;
    
    if (reverse) funs.reverse();
    
    return function (/* args */) {  
      var args = funs[0].apply(funs[0], arguments);
      for (var i=1; i < len; i++) {
        args = funs[i](args); 
      }
    
      return args;
    };
  };  
};

This uses much of the same code as before but this function now takes a reverse boolean that when set to true will perform a reverse on the functions passed in[5], and returns another closure (to take advantage of partial application[6]), we can now implement compose & sequence using this new function:

var compose = stitch(true);
var sequence = stitch(false);

Recursive Approach

In order to understand recursion, you must understand recursion

– Hunter, David (2011)

One section mentioned in Functional Javascript regarding recursion was the rules of thumb (Touretzky 1990) for every recursive function:

  • Know when to stop
  • Decide how to take one step
  • Break the problem into that step and a smaller problem

I will show how each relates to the recursive function of sequence & compose, now first let’s create a sequence function:

var sequence = function (/* funs */) {
  var funs = Array.prototype.slice.call(arguments);
  var recurse = function recurse(funs/*, args */) {
    var args = Array.prototype.slice.call(arguments, 1);  
    if (_.isEmpty(funs))
      return args;
        
      var fun = _.first(funs);
      recurse(_.rest(funs), fun.apply(fun, args));
  };
    
  return recurse.bind(null, funs);
};

First thing to note is that I’m taking advantage of underscore.js[7] this takes away a lot of the menial work.

  1. Like the imperative version we are taking any number of functions so we use the arguments variable to capture the arguments
  2. Next we declare an internal function called recurse, this is the function that will perform the recursion, this function accepts the array of functions as it’s first argument to make it more explicit about what it takes
  3. It first pulls out the arguments supplied, ignoring the first element because this will be the array functions
  4. A check is performed to see if we have removed all the functions from the array, if this is the case we return arguments that were passed in from the last call
  5. Otherwise we use the _.first function which as you might guess returns the first element when passed an array
  6. The we call ourselves, the first parameter we use _.rest which returns an array with the first element removed (we captured this above). The next parameter is calling the current function using the arguments we were supplied.
  7. It then returns a partially applied function[8] bound to the recurse function with the captured functions so it is ready to be executed by the caller

This can then be used in the same way as the imperative version:

var print = function (val) { console.log(val); };
var double = function (val) { return val * 2; };

var display = sequence(print, double);
display(10);

Here is a diagram to help illustrate how the recurse function works using the example above:

And so how does this fit in with the recursion rules of thumb I mentioned previously:

When to stop
_.isEmpty(funs)
Take one step
recurse(_.first(funs)…
Smaller problem
recurse(_rest.(funs)…

One thing to note is that this recursive function is written to enable tail recursion[9] however JS does not have support for this unlike full functional programming languages such as Scheme.

The next step is to enable compose, we can do this by abstracting the order in which the functions are picked:

var stitch = function (take, remove) {
  return function (/* funs */) {
    var funs = Array.prototype.slice.call(arguments);
    var recurse = function recurse(funs/*, args */) {
      var args = Array.prototype.slice.call(arguments, 1);  
      if (_.isEmpty(funs))
        return args;
        
        var fun = take(funs);
        recurse(remove(funs), fun.apply(fun, args));
    };
            
    return recurse.bind(null, funs);
  };
};

Similar to the imperative version we have created a new function called stitch, only this time we accept 2 functions one that will decide how to take a function from the array and another that will decide how to remove a function from the array, now using our good friend partial application we can declare both sequence and compose:

var compose = stitch(_.last, _.initial);
var sequence = stitch(_.first, _.rest);

The compose simply passes _.last (retrieves the last element when passed an array) and _.initial (returns a new array with the last element removed).

Summary

I have demonstrated a couple of ways of how these 2 functions can be implemented there are a whole host of other ways they could be implemented (or you could just use _.compose!) hopefully this will be helpful at this post has certainly helped me get a better understanding of how to apply recursion.


  1. Also known as pipeline in other languages
  2. Functional Composition
  3. The arguments variable
  4. I like to add a comment in the function parentheses to denote the arguments we are expecting just to make it clearer that we are expecting arguments (a handy trick picked up from Functional Javascript)
  5. I know from a micro performance point of view this isn’t as efficient as looping backwards but we are talking a tiny amount of functions that are going to be passed in so I don’t think this is going to be the bottleneck of your application.
  6. You could also change the signature to be function (reverse, /* funs */) and then use bind var compose = stitch.bind(null, true); but I think this nicely separates the configurable part from the actual usage part.
  7. This is JS library that provides functional style functions such as map, reduce etc… just a shame about the argument ordering as it means partial application cannot be applied out of the box.
  8. For more info partial application check out this post
  9. Tail Recursion

The difference between partial application & currying

As I’m working my way through Functional Javascript[1] (currently on chapter 5) I only now feel as though I understand what separates partial application from currying, I believe the main reason is that coming from an OOP background I think it takes quite some time to get used to functions playing such a critical role and the idea of functions taking other functions as parameters and returning functions from functions[2] is a large step away from objects being the focus point they are in OOP.

Once you get passed this first hurdle it then becomes easier to breakdown a Higher-Order Function and to be able to start see the signature[3] and how you can use either partial application or currying to created configured re-usable functions.

To illustrate how we can use currying to create re-usable functions I will need a volunteer function[4]:

var changeCasing = function (items, prop, upper) {
  var results = [];
  var len = items.length;
  for (var i = 0; i < len; i++) {
    var item = items[i];
    var val = item[prop];
    item[prop] = (upper) ? val.toUpperCase() : val.toLowerCase();
    results.push(item);
  }
  return results;
};

If this is not making you scream out use map or at least forEach you may want to come back after learning more about functional programming, aside from the fact this function is not utilizing any functional programming we can still wrap it up to use currying then with some tweaks partial application.

First we need some data to supply to it:

var data = [
    {id: 1, name: 'joe'},
    {id: 2, name: 'bob'},
    {id: 3, name: 'jill'},
];

So if I want to transform the name of the user into upper case as is the case here, it’s simply:

var upperCaseNames = changeCasing(data, 'name', true);

However we have a lot of repeating parameters that are always the same for this data array (the upper bool flag and the property name) we would like to pre-populate these parameters and then just be able to call the function providing the data array.

Currying

We can achieve this by modifying the changeCasing function to this[5]:

var changeCasingCurryable = function (upper) {
  return function (prop) {
    return function (items) {
      var results = [];
      var len = items.length;
      for (var i = 0; i < len; i++) {
        var item = items[i];
        var val = item[prop];
        item[prop] = (upper) ? val.toUpperCase() : val.toLowerCase();
        results.push(item);
      }
      return results;
    }
  }
};

Don’t worry I remember how I felt first time I saw code like this and goes back to my first point, notice however that once we get passed the nested function code the main body has not changed at all and we can call it via:

var upperCaseNames = changeCasingCurryable(true)('name')(data);

There are a few things to note here:

  1. We now have a separate function call for each argument expected
  2. I modified the changeCasing function so that we accept the parameters from right-to-left[6] which is why the upper bool flag is in the first call

Now we can start to reap the benefits of the modified function, let’s got through supplying values for each function call:

var upper = changeCasingCurryable(true);
// now I have a version that will always change the property provided to uppercase
var upperCaseNames = upper('name')(data);

var upperNames = changeCasingCurryable(true)('name');
// now I have a version that will always change the name property to uppercase
var upperCaseNames = upperNames(data);

Each time I’am providing just enough of the values I want to be applied and being handed back a function ready for the next value, once the values are all supplied the body is executed and I get the result back.

If we had to change each function like changeCasing to take advantage of currying this would be a major headache and require a lot of work when working with other libraries luckily we can take advantage of JS’s dynamic nature to build a generic functions that we can apply to a functions we want to curry:

var curry3 = function (fun) {
  return function (third) {
    return function (second) {
      return function (first) {
        return fun(first, second, third);      
      }
    }
  }
};

We can then change the changeCasingCurryable to be:

var changeCasingCurryable = curry3(changeCasing);

This function is explicit about the number of arguments to curry on, you can use the arguments variable to dynamically curry on the remaining arguments however this can cause you issues given the leniency JS has regarding function arguments.[7]

Partial Application

To demonstrate effective use of partial application, I will need to make some adjustments to changeCasing’s signature:

var changeCasing = function (upper, prop, items) {
//...
}

I have swapped the arguments so that the array of items are the last argument, this is the only change we need to make next we need a function we can use to apply partial application:

var partial = function (fun) {
  var partialArgs = Array.prototype.slice.call(arguments, 1);
  return function () {
    var remainingArgs = Array.prototype.slice.call(arguments);
    return fun.apply(fun, partialArgs.concat(remainingArgs));      
  }
};

This function takes the function we want to partially apply and also the parameters we want to fix the returned function then executes the function using the fixed parameters plus the passed into values, this can then be applied to changeCasing:

var changeCasingPartial = partial(changeCasing, true, 'name');
var upperCaseNames = changeCasingPartial(data);

There are a few things worth noting:

  1. Unlike the currying usage we don’t have a function returned for each argument, instead we can fix any number of parameters and the returned function is ready to be executed with the remaining parameters.
  2. Also unlike the currying usage we are dynamically applying the arguments rather than being explicit, this approach works well with JS’s lenient argument approach to function calls as we typically want to provide values for the first few parameters and then be able to provide the remaining at a later point.

Instead of using the custom partial function above you can also make use of the bind function[8] if it is supported in the target runtime, then the changeCasing above would change to:

var changeCasingPartial = changeCasing.bind(null, true, 'name');
var upperCaseNames = changeCasingPartial(data);

Note the context is supplied as the first parameter.

Conclusion

Both currying and partial application are very similar, they both return you back a function that has had one or more values bound to its parameters, however there are some notable differences:

  1. Currying can be performed on each argument in either direction, partial application is only performed left-to-right.
  2. Currying only deals with single argument functions, so we end up with a function for every argument we want to curry on. Partial application can be applied for any number of arguments, so you could provide every expected parameter and then the returned function could be executed with no parameters (although this would be of little use!)

  1. A great book for those wanting to learn about the functional programming paradigm but in a language which you already know
  2. Higher-Order Functions
  3. One way I have found it easier is by using a comment signature notation I have seen being used:
    //+ methodname :: input_type -> return_type
    //+ map :: fun(a -> b) -> [a] -> [b]
    

    I think it is based off other FP languages, to breakdown the map example:

    1. takes a function that expects any type as input and returns any type
    2. next it takes an array of the a type (passed to first function)
    3. finally returns an array of the b type (returned from first function)
  4. I really wanted to try and find a real world example however it always ended up with lots of other techniques detracting away from the main point of the post (I will try and get another post showing real world examples of functional programming in JS)
  5. The reason this is possible in JS is because of closures otherwise the nested function would not be able to capture the argument passed to it’s containing function
  6. Currying can be applied in either direction, I have chosen this direction because partial application always goes left-to-right so it emphasizes one of the key differences
  7. This is a demonstration to illustrate how you can get into trouble when not being explicit with function arguments.
  8. Bind Function

Handling retries part 4 – using both OOP & functional programming

Handling retries part 1 – using inheritance

Handling retries part 2 – using composition

Handling retries part 3 – using functional

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Having the best of both worlds

In the last part we had a couple of issues with using just a functional programming (FP) approach (certainly in C#, full FP languages have dealt with these issues)

Lets deal with the client duplication issue first, ideally what we would like is that we can have the FP retry behaviour but to have it wrapped up in an object that can be re-used:

public class RetryInstance
{    
    public TimeSpan Interval { get; set; }    
    public int Retries { get; set; }    
    public Action&lt;Exception&gt; OnException { get; set;}    
    public Action OnFailure = { get; set; }    
    public void Execute(Action action)    
    {        
        var retryCount = this.Retries;        
        while (retryCount &gt; 0)        
        {            
            try            
            {                
                this.action();                
                break;            
            }            
            catch (Exception ex)            
            {                
                retryCount--;                
                this.OnException(ex);            
            }            
            Thread.Sleep(this.Interval);        
            }        
        this.OnFailure();    
    }
}

This would be used like this:

var retryInstance = new retryInstance()
{    
    Interval = TimeSpan.FromSeconds(30),    
    Retries = 5,    
    OnException = ex =&gt; Log.Error(ex),    
    OnFailure = () =&gt; Log.Fatal(&quot;fail!&quot;)
};
    
retryInstance.Execute(networkFilCopier.DoCopy());
// later in the code
retryInstance.Interval = TimeSpan.FromMinutes(1);
retryInstance.Execute(networkFilCopier.DoCopy());

Here we have moved the FP code into an RetryInstance object we can the use this object to maintain the state of how we want the retries, intervals, callbacks to behave and in the example we can adjust this when we need to without having to duplicate anything. The readability of the code has improved quite a lot as well, but I think we can go one step better by introducing a Builder object to help us build a RetryInstance and also to provide it with suitable defaults:

public class RetryConfiguration
{    
    private TimeSpan _interval;    
    private int _retries;    
    private Action&lt;Exception&gt; _onException;    
    private Action _onFailure;    
    
    public void SetInterval(TimeSpan duration)    
    {        
        _interval = duration;    
    }    
    
    public void SetRetries(int count)    
    {        
        _retries = count;    
    }    
    
    public void WhenExceptionRaised(Action&lt;Exception&gt; handler)    
    {        
        _onException = handler;    
    }    
    
    public void WhenFailed(Action handler)    
    {        
        _onFailure = handler;    
    }    
    
    public RetryInstance Build()    
    {        
        return new RetryInstance()        
        {            
            Retries = _retries,            
            Interval = _interval,            
            OnException = _onException,            
            OnFailure = _onFailure        
        };    
    }
}
    
public static class RetryFactory
{    
    public static RetryInstance New(Action&lt;RetryConfiguration&gt; configuration)    
    {        
        config = new RetryConfiguration();        
        config.SetInterval(TimeSpan.FromSeconds(30));        
        config.SetRetries(5);        
        config.WhenExceptionRaised(_ =&gt;; {}); // no-op        
        config.WhenFailed(() =&gt;; {}); // no-op        
        configuration(config);        
        return config.Build();    
    }
}

This would then be used by client code like this:

RetryFactory.New(cfg =&gt;                
    {                    
        cfg.SetInterval(TimeSpan.FromMinutes(1));                    
        cfg.SetRetries(3);                    
        cfg.WhenExceptionRaised(ex =&gt; Log.Error(ex));                    
        cfg.WhenFailed(() =&gt; Log.Fatal(&quot;fail!&quot;));                
    }).Execute(networkFileCopier.DoCopy());

We have introduced a couple more objects one is a configuration object that exposes a really nice API to setup our RetryInstance object and the other is our builder object/fluent DSL that exposes a static method to create a new RetryInstance and also provides our suitable defaults that we can choose to override.

Summary

We have certainly covered a lot of ground in these series of posts and have ended up with quite a lot of options for how we could solve the issue in the introduction and there are no doubt way more other ways that I couldn’t think of! Some of these may be overkill especially for the simplistic case of retry behaviour we have been looking at as with most design & architecture its always a balance and this goes hand in hand with how the behaviour is going to be used. Here is a breakdown of how I would personally go about deciding how to implement the retry behaviour:

  • If I have one object that has one single use for using retry logic I would probably stick to a simple method call like the pseudo-code in the introduction
  • Once I have another use in a separate object I may decide to use inheritance (if no inheritance hierarchy is present) or move to using composition (decorator pattern if I don’t want the object to manage the retries)
  • If I start to have lots of objects wanting retry behaviour and especially across different projects that’s when I would probably move to using OOP & FP together to provide the callers with a nice API to use and to also reduce the amount of code they need to write in order to use it

I hope that this series has helped to put forward the case that we should be open to various different ways of designing software to solve problems and just because in the past you have always done it one way it doesn’t mean that you should then apply that across the board as each problem tends to unique.

Handling retries part 3 – using functional

Handling retries part 1 – using inheritance

Handling retries part 2 – using composition

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Functional

Sometimes it helps to take a look at different languages and programming styles when facing a problem to see how you would solve the problem and whether you can take any of the techniques and utilise them, this is especially true of functional programming (FP) now that C# has a lot more support of FP constructs (lambdas, generics, tuples etc…).

If we take a look at a javascript example of how we can achieve the retry behavour:

var Retrier = {
    execute: function (action, exception, failure, interval, retries) {
        try
        {
            action();
            return;
        }
        catch (ex)
        {
            retries--;
            exception(ex);
            if (retries > 0) {
              var vals = {
                retries: retries,
                interval: interval,
                action: action,
                exception: exception,
                failure: failure
              };
              window.setTimeout(function() {
                  Retrier.execute(vals.action, vals.exception, vals.failure, vals.interval, vals.retries);
                }, vals.interval);
            } else {
                failure();
            }
        }
    }
};

This would then be used like this:

Retrier.execute(function () { // action
                  networkFileCopier.DoCopy();
                },
                function (ex) { // exception
                  console.log('exception raised! ' + ex);
                },
                function () { // failure
                  console.log('fail!');
                }, 1000, 5);

I’ll be the first to admit that my javascript is not the best as I don’t tend to use it (I have omitted the anonymous function to close the scope), there are a number of major differences we have had to take into account:

  • Javascript does not have a concept of a class and instead just uses objects , therefore we have a simple object to hang the execute method off you can think of it as a static method in C#
  • All of the state is maintained inside the call I could have had the Retrier object have properties and this would work better if we wanted to have a standard number of retries, interval and way of handling errors, instead I have stuck to more of a FP style
  • You generally don’t want to do any sort of blocking in javascript as this would either block the UI thread in the browser or block the processing thread in NodeJS therefore instead we have to use the setTimeout function to tell javascript to call a specific function sometime in the future based on the interval
  • Due to the fact that we have to use setTimeout instead of sleeping the thread for the interval we use a recursive call with the retries value decremented each time, before we can do so we have to setup a closure vals otherwise the variables would be lost as javascript uses function scoping

Whenever using recursion we need to be careful not to end up overflowing the stack but in this case unless your going to retry a task several thousand times this should not be an issue.

So let’s take the above and create a C# equivelant:

public static class Retrier
{
    public static void Execute(Action action, Action exception, Action failure, TimeSpan interval, int retries)
    {
        var retryCount = retries;
        while (retryCount > 0)
        {
            try
            {
                action();
                break;
            }
            catch (Exception ex)
            {
                retryCount--;
                exception(ex);
            }
            Thread.Sleep(interval);
        }
        failure();
    }
}

This would then be used like this:

Retrier.Execute(() => networkFileCopier.DoCopy(),
                ex => Log.Error(ex),
                () => Log.Fatal("fail!"),
                TimeSpan.FromSeconds(30),
                5);

Well we have completely eliminated OOP from the retry behaviour here and instead are left with a single class to hold our Execute method, from the client side they are no longer required to create new objects to hook into the retry behaviour however there are a couple of issues:

  • There is going to be quite a bit of duplication from the client code as each time they need to setup all the callback methods and also assign the interval and retry amount
  • The API for the caller is very obtuse, once you start to have lamdas being passed in to method calls it can start to get difficult to understand (named arguments can help but is generally an indication your API could do with being changed)

In the next part I want to leverage OOP and FP together to see if we can fix the issues above.

Handling retries part 2 – using composition

Handling retries part 1 – using inheritance

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Composition

In chapter 1 of the Design Patterns GoF book there is a section titled Inheritance versus Composition I highly recommend anyone with the book who has not read this to go and take a look as it really distills the problems with relying too heavily on inheritance and even includes there own principle:

Favor object composition over class inheritance

The principle holds up when you see how many of the design patterns use composition as opposed to inheritance, so lets give composition a go:

public class Retrier
{
    protected readonly IRunner _runner;

    public int RetryCount { protected get; set; }
    public TimeSpan Interval { protected get; set; }
    public event EventHandler OnException = {};
    public event EventHandler OnFailure = {};

    public Retrier(IRunner runner)
    {
        _runner = runner;
    }

    public void Execute()
    {
        var retries = RetryCount;
        while (retries > 0)
        {
            try
            {
                _runner.Run();
                break;
            }
            catch (Exception ex)
            {
                retries--;
                OnException(this, ex);
            }
            Thread.Sleep(Interval);
        }
        OnFailure(this, EventArgs.Empty);
    }
}

public interface IRunner
{
    void Run();
}

This would then be used as follows:

public class NetworkFileCopier : IRunner
{
    protected Retrier _retrier;

    public NetworkFileCopier()
    {
        _retrier = new Retrier(this);
        _retrier.Interval = TimeSpan.FromSeconds(30);
        _retrier.OnException += ex => Log.Error(ex);
    }

    public void DoCopy()
    {
        _retrier.Execute();
    }

    public void Run()
    {
        // do file copy here
    }
}

Now we have wrapped up the behaviour inside the Retrier object and we reference it inside NetworkFileCopier, unlike the inheritance version we no longer need to be in an inheritance hierarchy so NetworkFileCopier can inherit from some other base class if it needed to. It does need to implement an interface so that the Retrier object knows what to call when it gets executed however this could be changed so that you pass the Retrier a delegate to call instead.

We still have the issue though that NetworkFileCopier is still having to manage the retry object the next section will remove this in case this is an issue.

Decorator pattern

One way we could split this responsibility out of NetworkFileCopier is to use the Decorator Pattern:

public interface IFileCopier
{
    void DoCopy();
}

public class NetworkFileCopier : IFileCopier
{
    public void DoCopy()
    {
        // do file copy here
    }
}

public class RetryFileCopier : IFileCopier, IRunner
{
    protected readonly IFileCopier _decoratedFileCopier;
    protected Retrier _retrier;

    public RetryFileCopier(IFileCopier decoratedFileCopier)
    {
        _decoratedFileCopier = decoratedFileCopier;
        _retrier = new Retrier(this);
        _retrier.Interval = TimeSpan.FromSeconds(30);
        _retrier.OnException += ex => Log.Error(ex);
    }

    public void DoCopy()
    {
        _retrier.Run();
    }

    public void Run()
    {
        _decoratedFileCopier.DoCopy();
    }
}

This can then be used by client code like this:

var fileCopier = new RetryFileCopier(
                    new NetworkFileCopier());
fileCopier.DoCopy();

The first thing to note is how slimmed down NetworkFileCopier is now its only concern is copying files this means that if we needed to change the retry behaviour we do not need to make any changes to this object a good example of orthogonal code also the client gets to decide if we want the behaviour or not.

I feel that these versions are nicer than the inheritance version we looked at in part 1 however it still feels like we need to perform quite a few tasks (or ceremony) to get this to work:

  • Introduce new interface IRunner so that Retrier can communicate with the method to execute (this can be alleviated by a delegate)
  • Introduce a new interface IFileCopier for the decorator pattern to be utilised
  • Introduce new object RetryFileCopier to wrap up the retry behaviour

In the next part I’m going to be throwing OOP out of the window and looking at how functional programming in C# could potentially save us from some of this overhead.

Handling retries part 1 – using inheritance

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Inheritance

In true OOP fashion many will reach for the tried and tested inheritance model to wrap up the behaviour above inside a base class à la Template Method Pattern:

public abstract class RetryBase
{
    public RetryBase()
    {
        Interval = TimeSpan.FromSeconds(10);
        RetryCount = 5;
    }

    protected TimeSpan Interval
    {
        get; set;
    }
    protected int RetryCount
    {
        get; set;
    }

    public void Execute()
    {
        var retries = retryCount;
        while (retries > 0)
        {
            try
            {
                ExecuteImpl();
                break;
            }
            catch (Exception ex)
            {
                retries--;
                Exception(ex);
            }
            Thread.Sleep(Interval);
        }
        Failure();
    }

    protected abstract void ExecuteImpl();

    protected virtual void Exception(Exception ex)
    {
    }

    protected virtual void Failure()
    {
    }
}

public class NetworkFileCopier : RetryBase
{
    public NetworkFileCopier()
    {
        // override to 30 secs
        Interval = TimeSpan.FromSeconds(30);
    }

    protected override void ExecuteImpl()
    {
        // do file copy here
    }

    // override to provide logging
    protected override void Exception(Exception ex)
    {
        Log.Error(ex);
    }
}

Client usage:

var networkFileCopier = new NetworkFileCopier();
networkFileCopier.Execute();

We now have reusable behaviour for our retry logic and we also have the ability to override the interval & retries and also get hooked into calls when an exception occurs or in case of failure. There are some issues with this approach though:

  • Firstly it requires quite a bit of work to be able to get this behaviour because we need to inherit from a specific class if we had many actions that needed this behaviour this could get tedious
  • Inheritance is static, once the class is compiled into the hierarchy it cannot change its behaviour dynamically (removing retry logic on demand) without extra code hooks this breaks OCP.
  • The NetworkFileCopier is now intrinsically tied to this inheritance hierachy if it already inherited from another base class we would then need that class to inherit from RetryBase or change RetryBase to inherit from the existing base class (yuk!)
  • Before NetworkFileCopier was happily getting on with it’s responsibility of copying a file over the network now it has to worry about retry logic (intervals, retry count, exception handling etc…) this breaks SRP.

WCF MaxReceivedMessageSize Handle with care

I see a number of questions get raised when the following exception is raised from a WCF client/server:

The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.

And a lot of the responses I see are similar the following:

Easy to solve you just need to set the maxReceivedMessageSize to 2147483647

So you make the change and the error goes away, all is good…

I’m not too sure you see the reason WCF has the the maxReceivedMessageSize in the first place is to have a safeguard by restricting the size of the message being transported, if we think about the change we have just made we have gone from allowing up to a 64kb message  to 2gb, now chances are your not going to hit this but even if we get up to 5mb because we are selecting every transaction that a client has ever made and they have been a customer for several years, now the following is put under extra stress:

  • Server has to serialize 5mb of data, this increases memory usage & cpu
  • Any network points now have to cope with 5mb going over the wire
  • Calling application has to deserialize 5mb of data, this also increases memory usage & cpu
  • The data then needs to be displayed to the client if this is a web application that means returning the html for the 5mb amount of data which will probably end up being more due to the presentation html structure as well, this will increase the outbound bandwidth usage significantly

Now looking at the above you may be thinking 5mb is not much and it should be handled easily and you’re probably correct however this is the steps for a single request, once you factor in multiple clients that may have similar amounts of transactions all browsing the site, your going to have major problems especially given the amount of time request may take to come back the the clients browser as the client will usually refresh the page a number of times before giving up.

Looking at the above the are a number of potential problems that could arise:

  • Depending on the memory the server has it may reach its maximum usage if it has to serialize multiple large objects
  • The calling application may also suffer the same fate when deserializing multiple large objects
  • The network may not have been prepared for the amount of traffic now going over it leading to sluggish replies
  • The calling application could run out of threads all waiting for replies coming back from the server (hopefully there will at least be a timeout set so that the thread does not wait indefinitely)
  • The clients browser may not be able to deal with the vast amounts of html being returned or more likely the client will give up before the browser completes rendering it

So had do we solve this? well let’s take a step back, once we start seeing errors about the size of messages we should really be thinking about what it is we are trying to achieve if we look at the example above

As a client I want to be able to view my previous transactions

If we think about the client who could have 5 years worth of transactions who could have thousands of transactions we want to present the transactions that are important to him now, chances are he is interested in his recent transactions so you could have:

  • Last 6 months transactions
  • Last 12 months transactions
  • Transactions made in 2012
  • Transactions made greater than £1000

By partitioning the transactions we have made it easier for the client to locate the transactions they’re interested in and also solved the issue of sending huge data objects over the network.

Now some people may be thinking that this is a cop out and if I were a client I may want to see all my transactions how would you get this to work…

  1. The client would be able to request all transactions
  2. The client is presented with a message to inform them that the results are being collated and he will be sent an email with the results
  3. In the meantime a backend service would deal with this request by querying the relevant database directly (preferably separate to the OLTP database)
  4. It would then build the results into a file (csv, xml, json etc…)
  5. Provide it to the client over a more appropriate channel (ftp, zipped attachment email)
  6. Notify the client via email that their transactions can now be viewed with the details to access the channel above

What I have done above is acknowledge that providing all the users transactions could result in a massive result set and therefore needs to be treated differently to a simple query performed via the website.

The key point I want to make here is that when you get an error raised due to message size it’s usually a dead give-away that you need to re-think what it is your trying to achieve especially if you have already increased the message size once before!