T-wise combinatorial Model-Based testing – Part II

In the previous post we saw how Model-Based Testing can be used to generate combinatorial input to the SUT. This is very nice, because we can capture this behavior in a generic way, and easily extend it, and the model will automatically generate the necessary combinations.

One of the oddities we observed, however, was that the model generated equivalent test cases where the parameter order was swapped. For pair-wise testing this is an annoyance because the model generates double the number of necessary tests, but for higher orders of t this leads to big problems as the duplications scale as n x t! (that’s t-factorial!), where n is the number on unique tests and t is the dimensionality of the combinatorial test generation.

 

The way to solve this is to fix the order in which parameters are set. Any single parameter need not be set, but it has to be evaluated in the right order. However, when defining models we must cast the solution in terms of what is not allowed and specify this as conditions. The following implementation is a generic solution that fixes the order in which parameters are set:

    public class TWiseOrderedParameter<T>

    {

        class SettableParameter<T>

        {

            public T Value;

            public bool isSet = false;

        }

        public const int TWISELIMIT = 2;

        static int ParameterCount = 0;

        static SequenceContainer<SettableParameter<T>> values = new SequenceContainer<SettableParameter<T>>();

        private SettableParameter<T> settableParameter = new SettableParameter<T>();

        public bool IsSet { get { return settableParameter.isSet; } }

        public static int ParametersSet()

        {

            return ParameterCount;

        }

        public T Value

        {

            get

            {

                return settableParameter.Value;

            }

            set

            {

                if (!IsSet)

                {

                    ParameterCount++;

                    Condition.IsTrue(ParameterCount <= TWISELIMIT);

                    foreach (SettableParameter<T> v in values)

                    {

                        if(Object.ReferenceEquals(v, settableParameter))

                            break;

                        Condition.IsFalse(v.isSet);

                    }

                    settableParameter.isSet = true;

                }

                settableParameter.Value = value;

            }

        }

        public TWiseOrderedParameter() {

            values.Add(settableParameter);

        }

    }

Now to implement this in our model we simply change the state variables to be:

        static TWiseOrderedParameter<int> ParamX = new TWiseOrderedParameter<int>();

        static TWiseOrderedParameter<int> ParamY = new TWiseOrderedParameter<int>();

        static TWiseOrderedParameter<int> ParamZ = new TWiseOrderedParameter<int>();

And viola, the generated test cases are:

No more duplicates, and a closer inspection shows that the parameters are set in the following order: z, y, x.

Conclusion

The power of Model-Based Testing becomes apparent in this scenario. Using relatively simple means we can limit the set of generated test cases. However, I would like to add that one has to carefully scrutinize both the model and the generated test cases to avoid any undesired effects. Model-Based Testing is quite powerful, and with great power comes great responsibilities J!

[1] T-wise testing solution