Applying Model-Based Testing for Fault-Injection in cloud services – Part III

2-services model

Interestingly enough, when modeling more than one service, the model will also generate tests that verify services continue functioning even when others go down. Essentially what we are checking here is that we don’t have any unnecessary dependencies between the services, and that a service won’t hang because another one is down.

When designing models including more than one service the state-space can rapidly explode. Consider adding just another service: Payment Service into the model which we depend on upon check-out.

We extend our model by adding:

    public class PaymentService : ModelService

    {

        public override string Name { get { return "Payment"; } }

        public void ProcessPayment()

        {

            ConditionRunning();

        }

    }

And changing the model to:

    static class WebshopModelProgram

    {

        static ShoppingCartService cartService = new ShoppingCartService();

        static PaymentService paymentService = new PaymentService();

        [Rule(Action = "AddToCart()")]

        static void AddToCart()

        {

            cartService.AddItem();

        }

        [Rule(Action = "ProceedToCheckOut()/result")]

        static bool ProceedToCheckOut()

        {

            if(cartService.Count() == 0)

                return false;

            cartService.Empty();

            paymentService.ProcessPayment();

            return true;

        }

Applying Model-Based Testing for Fault-Injection in cloud services – Part II

Okay, so I have to admit I’m mixing things up a bit here. This week I wanted to go to a 2-service model, but I realized that would be jumping ahead of things. Instead I want to take the next steps in actually implementing the shopping cart service and the model interface to this service, as this has some interesting challenges.

First the service…

Shopping cart service

The shopping cart service is a Windows Azure WCF Service Web Role project. It implements a very crude interface. You can add an item to the cart, reset it (empty it), retrieve the number of stored items and proceed to checkout (which includes emptying the cart). The interface is:

    public interface IShoppingCart

    {

        /// <summary>

        /// Empties the shopping cart

        /// </summary>

        [OperationContract]

        void Reset();

        /// <summary>

        /// Adds the item with given name to the shopping cart

        /// </summary>

        /// <param name="item">Name of item to add</param>

        [OperationContract]

        void AddItem(string item);

        /// <summary>

        /// Returns the number of items currently in the cart

        /// </summary>

        /// <returns>Number of items in cart</returns>

        [OperationContract]

        int GetCount();

        /// <summary>

        /// Continues to payment input, and empties the shopping cart

        /// </summary>

        /// <returns>False if no items are in the cart</returns>

        [OperationContract]

        bool ProceedToCheckout();

    }

The service can be started from within Visual Studio where you can debug it, or you can generate an Azure ASP.NET Web Role that consumes it.

Writing tests against a service is piece-of-cake. You simply add a service reference to the test project and create a static instance of the service client, which you can then call. The adapter layer of our model based tests will simply call the appropriate service method for each action, with the exception of the “KillService” and “RestoreService” rules that are special.

The implementation of a shopping cart service can be found in the appendix [1]. 

These actions are a bit more tricky to implement...

Applying Model-Based Testing for Fault-Injection in cloud services – Part I

I’ve been playing around with the thought on how to leverage model based testing for cloud based services. Of course, you can create models that interface with your services the usual way like testing any application, but can we get more out of it?

Well – what are some of the fundamental design differences when designing services in the cloud? Scale – it is all about scale, and with scale comes fragmentation and connectivity issues. Instead of having a giant monolithic application maintaining state, the state must instead be tracked in requests or some means of distributed state must exist. The points become interesting to attack from a testing perspective. You start asking yourself, what would happen if a service is suddenly unavailable, in a buggy implementation I could lose part of my state, can the services recover from that error condition? Are there services that are more vulnerable than others? What if the service dies just after I sent it a message, will my time-out recovery mechanism handle this correctly?

To make things more understandable I’ll give an example of a cloud based application. Imagine we are testing an online web shop composed of a website and three supporting services one for authentication, payments and shopping carts talking together to provide a fully functioning application. The underlying implementation could be fragmented like this.

Immediately we start asking questions like: What if the shopping cart service goes down, will my user loose the selection? What if the payment service does not respond to a payment request, is the error propagated to the website or will it be swallowed by the shopping cart service?

Model-Generated tests as part of regression

Any automated test case can be included in your regression suite – but does it make sense to include Model-Based tests in a regression suite? I’ve asked myself that question at work today, and I’d like to discuss some of the pros and cons here.

To put things into perspective, our regression suite has thousands of highly stable automated tests, and it is run after any minute change to the product has been made. If any single test from the regression suite fails the product change is reverted. Given this setting our four main priorities are reliability, efficiency, maintainability and ease of debugging.

Reliability

Reliability of the automated test cases is of course top priority, we cannot afford to reject developer changes due to test instabilities.

However, often reliability is governed by the underlying framework and not the tests themselves. From my experience model generated tests has the same reliability as any other function test.

Efficiency

Execution speed of the test is the main concern. We are time constrained in how long we can test a developers change, because we cannot slow the productivity of the organization down to a grinding halt because we want to run an unbounded number of tests.

In terms of execution speed model generated tests suffers because they often repeat a lot of steps between tests, effectively re-running the same piece of the test over and over again, where a manually written test case could leverage database backup/restore to avoid redoing steps or simply design smarter cases with less redundancy.

But MBT also suffers from generating too many tests. The model exploration will generate all possible combinations, with no means of determining a priority on individual tests. Effectively we cannot make a meaningful choice between model generated tests, so we are either forced to take all, none or a random selection. Because we want to minimize the risk of bugs slipping into the product, making arbitrary decisions is unadvisable (I have some ideas how we can do better at this, but that is for another blog post).

Maintainability

Any regression suite will sooner or later need to change, because requirements are changing for the product. Thus some amount of work will be put into refactoring existing regressions tests once in a while.

Model generated tests are actually easier to maintain (given you have staff with the required competencies) than regular functional tests. This boils down to what I blogged about earlier – the essence being that we can easier accommodate design changes because changes to the model automatically propagate into the tests. The additional abstraction actually helps us in this case. Conceptually it is also easier to understand what needs to be done to accommodate a change in requirements when you have a picture of the model state space to look at.

Ease of debugging

Congratulations your test is failing! But is this a bug or a defective test case? This must be the ultimate question we need to answer – and we want an answer fast!

A good starting point is to understand the conditions that apply before the failing step is taken. Reading the test code at first is not very helpful in trying to establish this, because it is auto generated gibberish with no comments to help you understand it. So at first we may conclude that this is a problem.

However, from my experience, even if it is harder to debug failing model based tests, it is not impossible, instead it changes the game. It is now a matter of understanding how the generated test case relates to the state space of the model. The state at which the test were before the offending step was taken is easily read, and for nice models you can trace the steps easily and in that way build up the understanding of the conditions.

Once you get good at it you start to read the model picture and realize important facts. For example, if you have one failing test making a state transition that another test is taking without failing, then something is amass. You need to start investigate why your model does not mimic the system under test, and this could be either a limitation of your model you need to handle or an actual bug in the product.

Conclusion

Model generated tests for regressions suites make sense on some points and are even better on the maintainability aspect, but unfortunately it falls flat on its face when we start considering efficiency. The lack of optimization in execution combined with a complete lack of prioritization of tests makes for a poor test pool when trying to establish what is relevant to run for regression.