Functional Programming & DSL’s Part 1

Introduction

I have explored DSL’s (Domain Specific Language) many moons ago and at that time I was using Boo (Python based .net language) after reading Ayende’s DSL book, it was a great choice as the language offers a lot of powerful capabilities way beyond C#, in the end I put together a DSL for log4net configuration.

While working my way through FP the topic of creating DSL’s has stood out for me when you start go down the route of FP you essentially build up a DSL around the problems your trying to solve it’s surprising the expressiveness that can be achieved using FP techniques, what I wanted to do was to see how closely I can get to a perfect DSL using FP.

Querying

If we take querying for data as our example[1], first we need some data:

var users = [
  { id: 1, username: 'jbloggs', dob: new Date(1984,1,1), displayName: 'Joe Bloggs', gender: 'm' },  
  { id: 2, username: 'zball', dob: new Date(1964,2,16), displayName: 'Zoe Ball', gender: 'f' },
  { id: 3, username: 'jsmith', dob: new Date(1957,7,23), displayName: 'John Smith', gender: 'm' },
  { id: 4, username: 'jfranco', dob: new Date(1979,3,2), displayName: 'Juan Franco', gender: 'm' },
  { id: 5, username: 'scastle', dob: new Date(1998,10,11), displayName: 'Susan Castle', gender: 'f' }
];

Now if we wanted to query from this data users that were male, over eighteen and only wanted to return the id, dob and gender, the following would be pretty close to a perfect way of expressing this requirement:

var results = query users                
              where all male, overEighteen
              select id, dob, gender             

If we were to express this in an OOP way we would probably be going down the route of a query object:

var results = new UserQuery()
                .gender('m')
                .ageGreaterThan(18)
                .query(users)
                .map(function (user) { return { id: user.id, dob: user.dob, gender: user.gender }; });

And of course you tweak the Query object and add some objects that will help make some of the querying more adaptable and less hardcoded:

var results = new Query()
                .where(new Equal('gender', 'm'))
                .where(new GreaterThan('dob', date.minusYears(18)))
                .restrict(['id', 'dob', 'gender'])
                .query(users);

But there’s a lot of ceremony going on there with objects getting created in a number of places and this is a pretty simple query, let’s see how taking an FP approach pans out, essentially what we have here is a pipeline of actions against a dataset, so lets breakdown each of the actions to see if we can achieve as close to the ideal DSL, first let’s start with select:

//+ select :: [string] -> [a] -> [b]
var select = function (/* props */) {
  var props = _.toArray(arguments);  
  return function (items) {
    return _.map(items, function (item) { 
      return _.pick(item, props); 
    });
  };
};

var selectResult = select('id', 'dob')(users);
console.log(selectResult);

To help with some of functional lifting I’m using underscore _.toArray and _.map don’t need explanation, the _.pick function takes an object and a series of strings for every property to retrieve on the passed object.

Ok this isn’t so bad we know that select is going to be used to specify which properties on the dataset we want returned and being part of the pipeline it is going to be eventually called with the items so I have used currying to set this up. I think we can do better regarding the properties and the moment we are using string literals ideally it would be nice to not have to use strings, we could either specify some variables to act like “constants”:

var id = 'id';
var dob = 'dob';

This works but is incredibly fragile as JS does not support true constants that cannot be mutated, what if we treated them instead of strings but as functions that when executed always returned the same value, we can use the _.partial and _.identity functions to achieve this:

var id = _.partial(_.identity, 'id');
var dob = _.partial(_.identity, 'dob');

id();
//=> id

_.identity will return the exact same value that is passed to it when called, and so I partially apply this function using _.partial specifying the value I always want to return.[2] We now need to change the select function because at the moment it is expecting strings to be passed in not functions:

//+ invoke :: (fun (a) -> b) -> b
var invoke = function (fun) { return fun(arguments); };
var concatResult = _.map([id, dob], invoke);
concatResult()
//=> ["id","dob"]

Here I have defined an simple function that takes a function in and invokes it passing any arguments and returning the result, we can then test this out by using _.map and as we can see correctly returns an array of the results of each function call. Now we can use this new function inside select:

//+ select :: [(fun () -> string)] -> [a] -> [b]
var select = function (/* props */) {
  var props = _.map(_.toArray(arguments), invoke);
  return function (items) {
    return _.map(items, function (item) { 
      return _.pick(item, props); 
    });
  };
};

You can see that props has now been changed so that it is a map for each argument passed in, the next thing we can do is get rid of the need for the anonymous function used in the map the only reason we currently have it there now is to capture the item argument that is then passed to the _.pick function what we can do is get rid of this middle step and have the map pass the item straight through, at present this is tricky because we can’t partially apply this function because it takes the item as the first argument[3], so lets first create a reusable function to allow this:

//+ pick :: [string] -> a -> b
var pick = function (props, item) {
  return _.pick(item, props);  
};

First we create a simple function that swaps the arguments and delegates to the underscore _.pick function.

//+ select :: [(fun () -> string)] -> [a] -> [b]
var select = function (/* props */) {
  var props = _.map(_.toArray(arguments), invoke)
    , picker = _.partial(pick, props);
  return function (items) { 
    return _.map(items, picker); 
  };
};

Next we can now partially apply over the the new pick function supplying the properties we want to pick, this means we can remove the anonymous function, one thing that might stand out is that we have a similar situation with the returned function we are simply using the items argument to then pass onto the _.map function if we could do the same thing we can return a partially applied map, this arguments issue is becoming quite a nuisance instead of having to create a new function from scratch let’s create a reverseArgs[4] function that will return the same function but will reverse it’s arguments when called:

//+ reverseArgs :: fun -> fun -> a
var reverseArgs = function (fun) {
  return function (/* args */) {
    var args = _.toArray(arguments);
    args.reverse();
    return fun.apply(null, args);  
  };
};

//+ map :: (fun a -> b), [a] -> [b]
var map = reverseArgs(_.map);

map follows the same strategy as pick in that it swaps the arguments over and delegates, this now opens the door to allow us to partially apply:

//+ select :: [(fun () -> string)] -> [a] -> [b]
var select = function (/* props */) {
  var props = _.map(_.toArray(arguments), invoke)
    , picker = _.partial(pick, props);
  return _.partial(map, picker);
};

We have managed to remove all the anonymous functions completely and instead are allow functions to call other functions without having to step in between, it now becomes clear what select performs:

  • Takes a number of arguments that are expected to be functions that return string values that will be properties of an object
  • Returns a function that when called with an array items will return an array of objects with only the properties specified populated

We can now start to use this select function:

var selectUsers = select(id, dob);
_.first(selectUsers(users));
//=> {id: 1, dob: Wed Feb 01 1984 00:00:00 GMT+0000 (GMT Standard Time)}

However one thing that is really good about this is that it is in no way tied to a user object, so we can use this for any object type:

var cars = [
    { reg: 'GNS 453A', make: 'Mitsubishi', model: 'Colt Ralliart' },
    { reg: 'HJD 112E', make: 'Ford', model: 'Focus RS' },
    { reg: 'JFS 672F', make: 'Vauxhall', model: 'Corsa VXR' }
];
    
var reg = _.partial(_.identity, 'reg');
var make = _.partial(_.identity, 'make');
    
var selectCars = select(reg, make);
_.first(selectCars(cars));
//=> {reg: "GNS 453A", make: "Mitsubishi"}

As long as I abide by the contract of what select needs it works perfectly for any object. In the next part I will look at where and how we can use it to filter results.


  1. Note that we are using a simple array and an object literal (treated like a map) rather than a user object like we would use in OOP.
  2. I know this is also susceptible to someone changing the variable to another function returning a different value, I have not shown it in the code examples but for all of these they should be inside there own scope which should lessen the likelihood of accidental changes.
  3. The argument ordering is one major annoyance with the underscore.js library as it seems to promote using _.chain based OOP style API rather than partial application & composition FP style, in fact there is a great video on youtube by Brian Lonsdorf which I highly recommend that goes into this in a lot more detail.
  4. We will also be using this for later parts.
Advertisements

Compose & Sequence created using recursion & imperative

Introduction

As I’m making my way further down the rabbit hole of the functional programming paradigm, I have found myself at the topic of recursion and wanted to do a post demonstrating how we can take an imperative operation and turn it into a recursive one but also using a real world example of something useful, hence why I decided that implementing compose & sequence[1] would be a good demonstration, first let’s get acquainted with these 2 functions.

Brief Introduction to Compose & Sequence

In functional programming you typically end up with functions calling other functions that then may also be called by another function[2], so if we had functions a, b and wanted to join them into a function called c:

var c = a(b);

This becomes harder to read as we have more nested calls, ideally we would like to pass the results of one function into another this is where compose and sequence come in, compose works from right to left and sequence left to right:

var c = compose(b, a);
// or
var c = sequence(a, b);

Typically compose is used as a function that takes 2 functions like the above in which case the implementation is trivial:

var compose = function (a, b) {
  return function () {
    return a(b.apply(b, arguments));  
  };
};

It returns a closure and applies on the second function using the arguments passed in and then uses the result of this call to pass to the first function and return the results. We can execute like this:

var print = function (val) { console.log(val); };
var double = function (val) { return val * 2; };

var display = compose(print, double);
display(10);

However in JS we can do much better than this because of its varargs support via the special arguments variable[3] so what I want to do next is go through how we can implement the compose & sequence functions using both an imperative approach or by using recursion.

Imperative Approach

First let’s start by creating the sequence function:

var sequence = function (/* funs */) {
  var funs = Array.prototype.slice.call(arguments);
  var len = funs.length;
     
  return function (/* args */) {  
    var args = funs[0].apply(funs[0], arguments);
    for (var i=1; i < len; i++) {
      args = funs[i](args); 
    }
        
    return args;
  };
};
  1. The sequence function does not specify the arguments explicitly as instead we are going to take multiple functions[4]
  2. The functions supplied are extracted from the arguments variable
  3. A closure is then declared that will be returned to the caller
  4. The argumentsfrom this closure are then used to call the first function supplied
  5. The results of this function call are stored in args variable
  6. The remaining functions passed are then called using a for loop each time the args variable is being mutated to contain the results of each function call
  7. Once the functions have all been executed the value stored in args is returned

The only difference between compose and sequence is that compose runs from right to left so if we can take all the existing code and just make this one bit of behaviour configurable we will save ourselves having to duplicate code:

var stitch = function (reverse) {
  return function (/* funs */) {
    var funs = Array.prototype.slice.call(arguments);
    var len = funs.length;
    
    if (reverse) funs.reverse();
    
    return function (/* args */) {  
      var args = funs[0].apply(funs[0], arguments);
      for (var i=1; i < len; i++) {
        args = funs[i](args); 
      }
    
      return args;
    };
  };  
};

This uses much of the same code as before but this function now takes a reverse boolean that when set to true will perform a reverse on the functions passed in[5], and returns another closure (to take advantage of partial application[6]), we can now implement compose & sequence using this new function:

var compose = stitch(true);
var sequence = stitch(false);

Recursive Approach

In order to understand recursion, you must understand recursion

– Hunter, David (2011)

One section mentioned in Functional Javascript regarding recursion was the rules of thumb (Touretzky 1990) for every recursive function:

  • Know when to stop
  • Decide how to take one step
  • Break the problem into that step and a smaller problem

I will show how each relates to the recursive function of sequence & compose, now first let’s create a sequence function:

var sequence = function (/* funs */) {
  var funs = Array.prototype.slice.call(arguments);
  var recurse = function recurse(funs/*, args */) {
    var args = Array.prototype.slice.call(arguments, 1);  
    if (_.isEmpty(funs))
      return args;
        
      var fun = _.first(funs);
      recurse(_.rest(funs), fun.apply(fun, args));
  };
    
  return recurse.bind(null, funs);
};

First thing to note is that I’m taking advantage of underscore.js[7] this takes away a lot of the menial work.

  1. Like the imperative version we are taking any number of functions so we use the arguments variable to capture the arguments
  2. Next we declare an internal function called recurse, this is the function that will perform the recursion, this function accepts the array of functions as it’s first argument to make it more explicit about what it takes
  3. It first pulls out the arguments supplied, ignoring the first element because this will be the array functions
  4. A check is performed to see if we have removed all the functions from the array, if this is the case we return arguments that were passed in from the last call
  5. Otherwise we use the _.first function which as you might guess returns the first element when passed an array
  6. The we call ourselves, the first parameter we use _.rest which returns an array with the first element removed (we captured this above). The next parameter is calling the current function using the arguments we were supplied.
  7. It then returns a partially applied function[8] bound to the recurse function with the captured functions so it is ready to be executed by the caller

This can then be used in the same way as the imperative version:

var print = function (val) { console.log(val); };
var double = function (val) { return val * 2; };

var display = sequence(print, double);
display(10);

Here is a diagram to help illustrate how the recurse function works using the example above:

And so how does this fit in with the recursion rules of thumb I mentioned previously:

When to stop
_.isEmpty(funs)
Take one step
recurse(_.first(funs)…
Smaller problem
recurse(_rest.(funs)…

One thing to note is that this recursive function is written to enable tail recursion[9] however JS does not have support for this unlike full functional programming languages such as Scheme.

The next step is to enable compose, we can do this by abstracting the order in which the functions are picked:

var stitch = function (take, remove) {
  return function (/* funs */) {
    var funs = Array.prototype.slice.call(arguments);
    var recurse = function recurse(funs/*, args */) {
      var args = Array.prototype.slice.call(arguments, 1);  
      if (_.isEmpty(funs))
        return args;
        
        var fun = take(funs);
        recurse(remove(funs), fun.apply(fun, args));
    };
            
    return recurse.bind(null, funs);
  };
};

Similar to the imperative version we have created a new function called stitch, only this time we accept 2 functions one that will decide how to take a function from the array and another that will decide how to remove a function from the array, now using our good friend partial application we can declare both sequence and compose:

var compose = stitch(_.last, _.initial);
var sequence = stitch(_.first, _.rest);

The compose simply passes _.last (retrieves the last element when passed an array) and _.initial (returns a new array with the last element removed).

Summary

I have demonstrated a couple of ways of how these 2 functions can be implemented there are a whole host of other ways they could be implemented (or you could just use _.compose!) hopefully this will be helpful at this post has certainly helped me get a better understanding of how to apply recursion.


  1. Also known as pipeline in other languages
  2. Functional Composition
  3. The arguments variable
  4. I like to add a comment in the function parentheses to denote the arguments we are expecting just to make it clearer that we are expecting arguments (a handy trick picked up from Functional Javascript)
  5. I know from a micro performance point of view this isn’t as efficient as looping backwards but we are talking a tiny amount of functions that are going to be passed in so I don’t think this is going to be the bottleneck of your application.
  6. You could also change the signature to be function (reverse, /* funs */) and then use bind var compose = stitch.bind(null, true); but I think this nicely separates the configurable part from the actual usage part.
  7. This is JS library that provides functional style functions such as map, reduce etc… just a shame about the argument ordering as it means partial application cannot be applied out of the box.
  8. For more info partial application check out this post
  9. Tail Recursion

The difference between partial application & currying

As I’m working my way through Functional Javascript[1] (currently on chapter 5) I only now feel as though I understand what separates partial application from currying, I believe the main reason is that coming from an OOP background I think it takes quite some time to get used to functions playing such a critical role and the idea of functions taking other functions as parameters and returning functions from functions[2] is a large step away from objects being the focus point they are in OOP.

Once you get passed this first hurdle it then becomes easier to breakdown a Higher-Order Function and to be able to start see the signature[3] and how you can use either partial application or currying to created configured re-usable functions.

To illustrate how we can use currying to create re-usable functions I will need a volunteer function[4]:

var changeCasing = function (items, prop, upper) {
  var results = [];
  var len = items.length;
  for (var i = 0; i < len; i++) {
    var item = items[i];
    var val = item[prop];
    item[prop] = (upper) ? val.toUpperCase() : val.toLowerCase();
    results.push(item);
  }
  return results;
};

If this is not making you scream out use map or at least forEach you may want to come back after learning more about functional programming, aside from the fact this function is not utilizing any functional programming we can still wrap it up to use currying then with some tweaks partial application.

First we need some data to supply to it:

var data = [
    {id: 1, name: 'joe'},
    {id: 2, name: 'bob'},
    {id: 3, name: 'jill'},
];

So if I want to transform the name of the user into upper case as is the case here, it’s simply:

var upperCaseNames = changeCasing(data, 'name', true);

However we have a lot of repeating parameters that are always the same for this data array (the upper bool flag and the property name) we would like to pre-populate these parameters and then just be able to call the function providing the data array.

Currying

We can achieve this by modifying the changeCasing function to this[5]:

var changeCasingCurryable = function (upper) {
  return function (prop) {
    return function (items) {
      var results = [];
      var len = items.length;
      for (var i = 0; i < len; i++) {
        var item = items[i];
        var val = item[prop];
        item[prop] = (upper) ? val.toUpperCase() : val.toLowerCase();
        results.push(item);
      }
      return results;
    }
  }
};

Don’t worry I remember how I felt first time I saw code like this and goes back to my first point, notice however that once we get passed the nested function code the main body has not changed at all and we can call it via:

var upperCaseNames = changeCasingCurryable(true)('name')(data);

There are a few things to note here:

  1. We now have a separate function call for each argument expected
  2. I modified the changeCasing function so that we accept the parameters from right-to-left[6] which is why the upper bool flag is in the first call

Now we can start to reap the benefits of the modified function, let’s got through supplying values for each function call:

var upper = changeCasingCurryable(true);
// now I have a version that will always change the property provided to uppercase
var upperCaseNames = upper('name')(data);

var upperNames = changeCasingCurryable(true)('name');
// now I have a version that will always change the name property to uppercase
var upperCaseNames = upperNames(data);

Each time I’am providing just enough of the values I want to be applied and being handed back a function ready for the next value, once the values are all supplied the body is executed and I get the result back.

If we had to change each function like changeCasing to take advantage of currying this would be a major headache and require a lot of work when working with other libraries luckily we can take advantage of JS’s dynamic nature to build a generic functions that we can apply to a functions we want to curry:

var curry3 = function (fun) {
  return function (third) {
    return function (second) {
      return function (first) {
        return fun(first, second, third);      
      }
    }
  }
};

We can then change the changeCasingCurryable to be:

var changeCasingCurryable = curry3(changeCasing);

This function is explicit about the number of arguments to curry on, you can use the arguments variable to dynamically curry on the remaining arguments however this can cause you issues given the leniency JS has regarding function arguments.[7]

Partial Application

To demonstrate effective use of partial application, I will need to make some adjustments to changeCasing’s signature:

var changeCasing = function (upper, prop, items) {
//...
}

I have swapped the arguments so that the array of items are the last argument, this is the only change we need to make next we need a function we can use to apply partial application:

var partial = function (fun) {
  var partialArgs = Array.prototype.slice.call(arguments, 1);
  return function () {
    var remainingArgs = Array.prototype.slice.call(arguments);
    return fun.apply(fun, partialArgs.concat(remainingArgs));      
  }
};

This function takes the function we want to partially apply and also the parameters we want to fix the returned function then executes the function using the fixed parameters plus the passed into values, this can then be applied to changeCasing:

var changeCasingPartial = partial(changeCasing, true, 'name');
var upperCaseNames = changeCasingPartial(data);

There are a few things worth noting:

  1. Unlike the currying usage we don’t have a function returned for each argument, instead we can fix any number of parameters and the returned function is ready to be executed with the remaining parameters.
  2. Also unlike the currying usage we are dynamically applying the arguments rather than being explicit, this approach works well with JS’s lenient argument approach to function calls as we typically want to provide values for the first few parameters and then be able to provide the remaining at a later point.

Instead of using the custom partial function above you can also make use of the bind function[8] if it is supported in the target runtime, then the changeCasing above would change to:

var changeCasingPartial = changeCasing.bind(null, true, 'name');
var upperCaseNames = changeCasingPartial(data);

Note the context is supplied as the first parameter.

Conclusion

Both currying and partial application are very similar, they both return you back a function that has had one or more values bound to its parameters, however there are some notable differences:

  1. Currying can be performed on each argument in either direction, partial application is only performed left-to-right.
  2. Currying only deals with single argument functions, so we end up with a function for every argument we want to curry on. Partial application can be applied for any number of arguments, so you could provide every expected parameter and then the returned function could be executed with no parameters (although this would be of little use!)

  1. A great book for those wanting to learn about the functional programming paradigm but in a language which you already know
  2. Higher-Order Functions
  3. One way I have found it easier is by using a comment signature notation I have seen being used:
    //+ methodname :: input_type -> return_type
    //+ map :: fun(a -> b) -> [a] -> [b]
    

    I think it is based off other FP languages, to breakdown the map example:

    1. takes a function that expects any type as input and returns any type
    2. next it takes an array of the a type (passed to first function)
    3. finally returns an array of the b type (returned from first function)
  4. I really wanted to try and find a real world example however it always ended up with lots of other techniques detracting away from the main point of the post (I will try and get another post showing real world examples of functional programming in JS)
  5. The reason this is possible in JS is because of closures otherwise the nested function would not be able to capture the argument passed to it’s containing function
  6. Currying can be applied in either direction, I have chosen this direction because partial application always goes left-to-right so it emphasizes one of the key differences
  7. This is a demonstration to illustrate how you can get into trouble when not being explicit with function arguments.
  8. Bind Function

Handling retries part 4 – using both OOP & functional programming

Handling retries part 1 – using inheritance

Handling retries part 2 – using composition

Handling retries part 3 – using functional

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Having the best of both worlds

In the last part we had a couple of issues with using just a functional programming (FP) approach (certainly in C#, full FP languages have dealt with these issues)

Lets deal with the client duplication issue first, ideally what we would like is that we can have the FP retry behaviour but to have it wrapped up in an object that can be re-used:

public class RetryInstance
{    
    public TimeSpan Interval { get; set; }    
    public int Retries { get; set; }    
    public Action&lt;Exception&gt; OnException { get; set;}    
    public Action OnFailure = { get; set; }    
    public void Execute(Action action)    
    {        
        var retryCount = this.Retries;        
        while (retryCount &gt; 0)        
        {            
            try            
            {                
                this.action();                
                break;            
            }            
            catch (Exception ex)            
            {                
                retryCount--;                
                this.OnException(ex);            
            }            
            Thread.Sleep(this.Interval);        
            }        
        this.OnFailure();    
    }
}

This would be used like this:

var retryInstance = new retryInstance()
{    
    Interval = TimeSpan.FromSeconds(30),    
    Retries = 5,    
    OnException = ex =&gt; Log.Error(ex),    
    OnFailure = () =&gt; Log.Fatal(&quot;fail!&quot;)
};
    
retryInstance.Execute(networkFilCopier.DoCopy());
// later in the code
retryInstance.Interval = TimeSpan.FromMinutes(1);
retryInstance.Execute(networkFilCopier.DoCopy());

Here we have moved the FP code into an RetryInstance object we can the use this object to maintain the state of how we want the retries, intervals, callbacks to behave and in the example we can adjust this when we need to without having to duplicate anything. The readability of the code has improved quite a lot as well, but I think we can go one step better by introducing a Builder object to help us build a RetryInstance and also to provide it with suitable defaults:

public class RetryConfiguration
{    
    private TimeSpan _interval;    
    private int _retries;    
    private Action&lt;Exception&gt; _onException;    
    private Action _onFailure;    
    
    public void SetInterval(TimeSpan duration)    
    {        
        _interval = duration;    
    }    
    
    public void SetRetries(int count)    
    {        
        _retries = count;    
    }    
    
    public void WhenExceptionRaised(Action&lt;Exception&gt; handler)    
    {        
        _onException = handler;    
    }    
    
    public void WhenFailed(Action handler)    
    {        
        _onFailure = handler;    
    }    
    
    public RetryInstance Build()    
    {        
        return new RetryInstance()        
        {            
            Retries = _retries,            
            Interval = _interval,            
            OnException = _onException,            
            OnFailure = _onFailure        
        };    
    }
}
    
public static class RetryFactory
{    
    public static RetryInstance New(Action&lt;RetryConfiguration&gt; configuration)    
    {        
        config = new RetryConfiguration();        
        config.SetInterval(TimeSpan.FromSeconds(30));        
        config.SetRetries(5);        
        config.WhenExceptionRaised(_ =&gt;; {}); // no-op        
        config.WhenFailed(() =&gt;; {}); // no-op        
        configuration(config);        
        return config.Build();    
    }
}

This would then be used by client code like this:

RetryFactory.New(cfg =&gt;                
    {                    
        cfg.SetInterval(TimeSpan.FromMinutes(1));                    
        cfg.SetRetries(3);                    
        cfg.WhenExceptionRaised(ex =&gt; Log.Error(ex));                    
        cfg.WhenFailed(() =&gt; Log.Fatal(&quot;fail!&quot;));                
    }).Execute(networkFileCopier.DoCopy());

We have introduced a couple more objects one is a configuration object that exposes a really nice API to setup our RetryInstance object and the other is our builder object/fluent DSL that exposes a static method to create a new RetryInstance and also provides our suitable defaults that we can choose to override.

Summary

We have certainly covered a lot of ground in these series of posts and have ended up with quite a lot of options for how we could solve the issue in the introduction and there are no doubt way more other ways that I couldn’t think of! Some of these may be overkill especially for the simplistic case of retry behaviour we have been looking at as with most design & architecture its always a balance and this goes hand in hand with how the behaviour is going to be used. Here is a breakdown of how I would personally go about deciding how to implement the retry behaviour:

  • If I have one object that has one single use for using retry logic I would probably stick to a simple method call like the pseudo-code in the introduction
  • Once I have another use in a separate object I may decide to use inheritance (if no inheritance hierarchy is present) or move to using composition (decorator pattern if I don’t want the object to manage the retries)
  • If I start to have lots of objects wanting retry behaviour and especially across different projects that’s when I would probably move to using OOP & FP together to provide the callers with a nice API to use and to also reduce the amount of code they need to write in order to use it

I hope that this series has helped to put forward the case that we should be open to various different ways of designing software to solve problems and just because in the past you have always done it one way it doesn’t mean that you should then apply that across the board as each problem tends to unique.

Handling retries part 3 – using functional

Handling retries part 1 – using inheritance

Handling retries part 2 – using composition

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Functional

Sometimes it helps to take a look at different languages and programming styles when facing a problem to see how you would solve the problem and whether you can take any of the techniques and utilise them, this is especially true of functional programming (FP) now that C# has a lot more support of FP constructs (lambdas, generics, tuples etc…).

If we take a look at a javascript example of how we can achieve the retry behavour:

var Retrier = {
    execute: function (action, exception, failure, interval, retries) {
        try
        {
            action();
            return;
        }
        catch (ex)
        {
            retries--;
            exception(ex);
            if (retries > 0) {
              var vals = {
                retries: retries,
                interval: interval,
                action: action,
                exception: exception,
                failure: failure
              };
              window.setTimeout(function() {
                  Retrier.execute(vals.action, vals.exception, vals.failure, vals.interval, vals.retries);
                }, vals.interval);
            } else {
                failure();
            }
        }
    }
};

This would then be used like this:

Retrier.execute(function () { // action
                  networkFileCopier.DoCopy();
                },
                function (ex) { // exception
                  console.log('exception raised! ' + ex);
                },
                function () { // failure
                  console.log('fail!');
                }, 1000, 5);

I’ll be the first to admit that my javascript is not the best as I don’t tend to use it (I have omitted the anonymous function to close the scope), there are a number of major differences we have had to take into account:

  • Javascript does not have a concept of a class and instead just uses objects , therefore we have a simple object to hang the execute method off you can think of it as a static method in C#
  • All of the state is maintained inside the call I could have had the Retrier object have properties and this would work better if we wanted to have a standard number of retries, interval and way of handling errors, instead I have stuck to more of a FP style
  • You generally don’t want to do any sort of blocking in javascript as this would either block the UI thread in the browser or block the processing thread in NodeJS therefore instead we have to use the setTimeout function to tell javascript to call a specific function sometime in the future based on the interval
  • Due to the fact that we have to use setTimeout instead of sleeping the thread for the interval we use a recursive call with the retries value decremented each time, before we can do so we have to setup a closure vals otherwise the variables would be lost as javascript uses function scoping

Whenever using recursion we need to be careful not to end up overflowing the stack but in this case unless your going to retry a task several thousand times this should not be an issue.

So let’s take the above and create a C# equivelant:

public static class Retrier
{
    public static void Execute(Action action, Action exception, Action failure, TimeSpan interval, int retries)
    {
        var retryCount = retries;
        while (retryCount > 0)
        {
            try
            {
                action();
                break;
            }
            catch (Exception ex)
            {
                retryCount--;
                exception(ex);
            }
            Thread.Sleep(interval);
        }
        failure();
    }
}

This would then be used like this:

Retrier.Execute(() => networkFileCopier.DoCopy(),
                ex => Log.Error(ex),
                () => Log.Fatal("fail!"),
                TimeSpan.FromSeconds(30),
                5);

Well we have completely eliminated OOP from the retry behaviour here and instead are left with a single class to hold our Execute method, from the client side they are no longer required to create new objects to hook into the retry behaviour however there are a couple of issues:

  • There is going to be quite a bit of duplication from the client code as each time they need to setup all the callback methods and also assign the interval and retry amount
  • The API for the caller is very obtuse, once you start to have lamdas being passed in to method calls it can start to get difficult to understand (named arguments can help but is generally an indication your API could do with being changed)

In the next part I want to leverage OOP and FP together to see if we can fix the issues above.

Handling retries part 2 – using composition

Handling retries part 1 – using inheritance

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Composition

In chapter 1 of the Design Patterns GoF book there is a section titled Inheritance versus Composition I highly recommend anyone with the book who has not read this to go and take a look as it really distills the problems with relying too heavily on inheritance and even includes there own principle:

Favor object composition over class inheritance

The principle holds up when you see how many of the design patterns use composition as opposed to inheritance, so lets give composition a go:

public class Retrier
{
    protected readonly IRunner _runner;

    public int RetryCount { protected get; set; }
    public TimeSpan Interval { protected get; set; }
    public event EventHandler OnException = {};
    public event EventHandler OnFailure = {};

    public Retrier(IRunner runner)
    {
        _runner = runner;
    }

    public void Execute()
    {
        var retries = RetryCount;
        while (retries > 0)
        {
            try
            {
                _runner.Run();
                break;
            }
            catch (Exception ex)
            {
                retries--;
                OnException(this, ex);
            }
            Thread.Sleep(Interval);
        }
        OnFailure(this, EventArgs.Empty);
    }
}

public interface IRunner
{
    void Run();
}

This would then be used as follows:

public class NetworkFileCopier : IRunner
{
    protected Retrier _retrier;

    public NetworkFileCopier()
    {
        _retrier = new Retrier(this);
        _retrier.Interval = TimeSpan.FromSeconds(30);
        _retrier.OnException += ex => Log.Error(ex);
    }

    public void DoCopy()
    {
        _retrier.Execute();
    }

    public void Run()
    {
        // do file copy here
    }
}

Now we have wrapped up the behaviour inside the Retrier object and we reference it inside NetworkFileCopier, unlike the inheritance version we no longer need to be in an inheritance hierarchy so NetworkFileCopier can inherit from some other base class if it needed to. It does need to implement an interface so that the Retrier object knows what to call when it gets executed however this could be changed so that you pass the Retrier a delegate to call instead.

We still have the issue though that NetworkFileCopier is still having to manage the retry object the next section will remove this in case this is an issue.

Decorator pattern

One way we could split this responsibility out of NetworkFileCopier is to use the Decorator Pattern:

public interface IFileCopier
{
    void DoCopy();
}

public class NetworkFileCopier : IFileCopier
{
    public void DoCopy()
    {
        // do file copy here
    }
}

public class RetryFileCopier : IFileCopier, IRunner
{
    protected readonly IFileCopier _decoratedFileCopier;
    protected Retrier _retrier;

    public RetryFileCopier(IFileCopier decoratedFileCopier)
    {
        _decoratedFileCopier = decoratedFileCopier;
        _retrier = new Retrier(this);
        _retrier.Interval = TimeSpan.FromSeconds(30);
        _retrier.OnException += ex => Log.Error(ex);
    }

    public void DoCopy()
    {
        _retrier.Run();
    }

    public void Run()
    {
        _decoratedFileCopier.DoCopy();
    }
}

This can then be used by client code like this:

var fileCopier = new RetryFileCopier(
                    new NetworkFileCopier());
fileCopier.DoCopy();

The first thing to note is how slimmed down NetworkFileCopier is now its only concern is copying files this means that if we needed to change the retry behaviour we do not need to make any changes to this object a good example of orthogonal code also the client gets to decide if we want the behaviour or not.

I feel that these versions are nicer than the inheritance version we looked at in part 1 however it still feels like we need to perform quite a few tasks (or ceremony) to get this to work:

  • Introduce new interface IRunner so that Retrier can communicate with the method to execute (this can be alleviated by a delegate)
  • Introduce a new interface IFileCopier for the decorator pattern to be utilised
  • Introduce new object RetryFileCopier to wrap up the retry behaviour

In the next part I’m going to be throwing OOP out of the window and looking at how functional programming in C# could potentially save us from some of this overhead.

Handling retries part 1 – using inheritance

Introduction

In our line of work there are usually many ways to accomplish a particular task (for better or worse), in these series of posts I want to try and demonstrate various different techniques that we can use and also what benefits we can gain from each.

So without further ado here is the scenario I want to be able to support:

I need a way of performing a particular action that can also handle an exception being raised by re-trying the action after a specified amount of time for a specified number of retries.

here is the pseudo-code to get an idea:

set retries = 5
    while retries > 0
        begin
            call task
            exit while
        exception
            decrement retries
            call exception
        end
        call sleep 3
    end while
call failure

The most basic way to accomplish this would be to simply have the C# replicate exactly what we have above and this would do the trick but means that if we had other tasks that needed to behave the same way we would end up duplicating the code for every instance ideally we want to re-use this behaviour.

Inheritance

In true OOP fashion many will reach for the tried and tested inheritance model to wrap up the behaviour above inside a base class à la Template Method Pattern:

public abstract class RetryBase
{
    public RetryBase()
    {
        Interval = TimeSpan.FromSeconds(10);
        RetryCount = 5;
    }

    protected TimeSpan Interval
    {
        get; set;
    }
    protected int RetryCount
    {
        get; set;
    }

    public void Execute()
    {
        var retries = retryCount;
        while (retries > 0)
        {
            try
            {
                ExecuteImpl();
                break;
            }
            catch (Exception ex)
            {
                retries--;
                Exception(ex);
            }
            Thread.Sleep(Interval);
        }
        Failure();
    }

    protected abstract void ExecuteImpl();

    protected virtual void Exception(Exception ex)
    {
    }

    protected virtual void Failure()
    {
    }
}

public class NetworkFileCopier : RetryBase
{
    public NetworkFileCopier()
    {
        // override to 30 secs
        Interval = TimeSpan.FromSeconds(30);
    }

    protected override void ExecuteImpl()
    {
        // do file copy here
    }

    // override to provide logging
    protected override void Exception(Exception ex)
    {
        Log.Error(ex);
    }
}

Client usage:

var networkFileCopier = new NetworkFileCopier();
networkFileCopier.Execute();

We now have reusable behaviour for our retry logic and we also have the ability to override the interval & retries and also get hooked into calls when an exception occurs or in case of failure. There are some issues with this approach though:

  • Firstly it requires quite a bit of work to be able to get this behaviour because we need to inherit from a specific class if we had many actions that needed this behaviour this could get tedious
  • Inheritance is static, once the class is compiled into the hierarchy it cannot change its behaviour dynamically (removing retry logic on demand) without extra code hooks this breaks OCP.
  • The NetworkFileCopier is now intrinsically tied to this inheritance hierachy if it already inherited from another base class we would then need that class to inherit from RetryBase or change RetryBase to inherit from the existing base class (yuk!)
  • Before NetworkFileCopier was happily getting on with it’s responsibility of copying a file over the network now it has to worry about retry logic (intervals, retry count, exception handling etc…) this breaks SRP.