James Craig

Building Your Own Unit Testing Framework in C#

For when you're bored on a Sunday with nothing else to do.
Feb 05 2011 by James Craig

About two weeks ago I became bored and when I am bored, I write code. In this case I was curious how difficult it would be to write my own unit testing framework. I mean I use xUnit.net for most of my day to day testing, but I never gave much thought into what goes into creating the framework itself. It turns out that it's fairly simple. Truth be told there are only a couple of parts that the framework needs:

 /\*
 Copyright (c) 2011 <a href="http://www.gutgames.com">James Craig</a>
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.\*/
  
 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using Utilities.DataTypes.Patterns.BaseClasses;
 using System.Reflection;
 using MoonUnit.Attributes;
 using MoonUnit.Exceptions;
 using MoonUnit.InternalClasses;
 using Utilities.Profiler;
 #endregion
  
 namespace MoonUnit
 {
     /// <summary>
     /// Manager class for unit testing framework
     /// </summary>
     public class Manager : Singleton<Manager>
     {
         #region Constructor
  
         /// <summary>
         /// Constructor
         /// </summary>
         protected Manager()
             : base()
         {
         }
  
         #endregion
  
         #region Functions
  
         /// <summary>
         /// Tests an assembly
         /// </summary>
         /// <param name="AssemblyToTest">Assembly to test</param>
         /// <returns>The XML string of the results</returns>
         public string Test(Assembly AssemblyToTest)
         {
             Type\[\] Types = AssemblyToTest.GetTypes();
             return Test(Types);
         }
  
         /// <summary>
         /// Test a list of types
         /// </summary>
         /// <param name="Types">Types to test</param>
         /// <returns>The XML string of the results</returns>
         public string Test(Type\[\] Types)
         {
             Output TempOutput = new Output();
             foreach (Type Type in Types)
                 TestClass(Type, TempOutput);
             return TempOutput.ToString();
         }
  
         /// <summary>
         /// Tests a type
         /// </summary>
         /// <param name="Type">Type to test</param>
         /// <returns>The XML string of the results</returns>
         public string Test(Type Type)
         {
             Output TempOutput = new Output();
             TestClass(Type, TempOutput);
             return TempOutput.ToString();
         }
  
         private static void TestClass(Type Type, Output TempOutput)
         {
             MethodInfo\[\] Methods = Type.GetMethods();
             foreach (MethodInfo Method in Methods)
             {
                 object\[\] Attributes = Method.GetCustomAttributes(false);
                 foreach (Attribute TempAttribute in Attributes)
                 {
                     if (TempAttribute is TestAttribute)
                     {
                         TestAttribute Attribute = (TestAttribute)TempAttribute;
                         if (!Attribute.Skip)
                         {
                             StopWatch Watch = new StopWatch();
                             object TestClass = Type.Assembly.CreateInstance(Type.FullName);
                             try
                             {
                                 Watch.Start();
                                 Method.Invoke(TestClass, Type.EmptyTypes);
                                 Watch.Stop();
                                 if (Attribute.TimeOut > 0 && Watch.ElapsedTime > Attribute.TimeOut)
                                     throw new TimeOut("Method took longer than expected");
                                 TempOutput.MethodCalled(Method);
                             }
                             catch (Exception e)
                             {
                                 if(e.InnerException!=null)
                                     TempOutput.MethodCalled(Method, e.InnerException);
                                 else
                                     TempOutput.MethodCalled(Method, e);
                             }
                         }
                         else
                         {
                             TempOutput.MethodCalled(Method, new Skipped(Attribute.ReasonForSkipping));
                         }
                     }
                 }
             }
         }
  
         #endregion
     }
 }

I'm using code from my utility library and will skip over those parts but you should be able to figure out what they're doing. Anyway this class only has a couple of functions of note. The first couple are simply Test. These functions are called to actually run tests (either sending in a Type, array of Types, or an assembly). These then call TestClass which goes through and sees if any methods on the class are marked as a test. We do this by using the following attribute:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 #endregion
  
 namespace MoonUnit.Attributes
 {
     /// <summary>
     /// Attribute used to denote a test function
     /// </summary>
     \[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)\]
     public class TestAttribute : Attribute
     {
         #region Constructor
  
         /// <summary>
         /// Constructor
         /// </summary>
         public TestAttribute(long TimeOut=0)
             : base()
         {
             Skip = false;
             this.TimeOut = TimeOut;
         }
  
         /// <summary>
         /// Constructor
         /// </summary>
         /// <param name="ReasonForSkipping">Reason for skipping this test</param>
         public TestAttribute(string ReasonForSkipping,long TimeOut=0)
             : base()
         {
             this.Skip = true;
             this.ReasonForSkipping = ReasonForSkipping;
             this.TimeOut = TimeOut;
         }
  
         #endregion
  
         #region Properties
  
         /// <summary>
         /// Determines if the test should be skipped
         /// </summary>
         public bool Skip { get; private set; }
  
         /// <summary>
         /// The reason for skipping this test
         /// </summary>
         public string ReasonForSkipping { get; private set; }
  
         /// <summary>
         /// If it takes longer than this, consider it a timeout/failed test
         /// </summary>
         public long TimeOut { get; private set; }
  
         #endregion
     }
 }

With this attribute, we can mark a method to be run by the system. For instance:

 class Example
 {
     public void DoesNotRun(){}
     \[Test\]
     public void Runs(){}
 }

In this case Runs would be called by the system and DoesNotRun would not be. You can also tell the system to skip a test by giving an explanation. You can also give it an amount of time for it to wait. If it takes longer than a specified time, the test is considered a failure. The last bit of code that is of interest is when an error is detected (and thus the individual test is considered a failure), we need some way to record this. We do this with the aid of Output:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using MoonUnit.Exceptions;
 using System.Reflection;
 #endregion
  
 namespace MoonUnit.InternalClasses
 {
     /// <summary>
     /// Output class
     /// </summary>
     internal class Output
     {
         #region Constructor
  
         public Output()
         {
             MethodsCalled = new List<TestMethod>();
         }
  
         #endregion
  
         #region Functions
  
         public void MethodCalled(MethodInfo MethodInformation, Exception Exception=null)
         {
             MethodsCalled.Add(new TestMethod(MethodInformation, Exception));
         }
  
         public override string ToString()
         {
             string AssemblyLocation = Assembly.GetAssembly(this.GetType()).Location;
             AssemblyName Name = AssemblyName.GetAssemblyName(AssemblyLocation);
             StringBuilder Builder = new StringBuilder();
             Builder.Append("<?xml version=\\"1.0\\" encoding=\\"UTF-8\\" ?>");
             Builder.Append(string.Format("<MoonUnit><Header><FileLocation>{0}</FileLocation><Version>{1}</Version></Header><Tests>", AssemblyLocation, Name.Version));
             foreach (TestMethod Method in MethodsCalled)
             {
                 Builder.Append(Method.ToString());
             }
             Builder.Append("</Tests><Footer></Footer></MoonUnit>");
             return Builder.ToString();
         }
  
         #endregion
  
         #region Properties
  
         public virtual List<TestMethod> MethodsCalled { get; private set; }
  
         #endregion
     }
 }

This class simply gets called when a test method is called and the result, it then stores the results in a list and can later export the results into an XML string. The individual test records are stored in the following class:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using MoonUnit.Exceptions;
 using System.Reflection;
 #endregion
  
 namespace MoonUnit.InternalClasses
 {
     /// <summary>
     /// Holds information about test methods
     /// </summary>
     internal class TestMethod
     {
         #region Constructor
  
  
         public TestMethod(MethodInfo MethodInformation,Exception Exception)
         {
             this.MethodInformation = MethodInformation;
             this.Exception = Exception;
         }
  
         #endregion
  
         #region Functions
  
         public override string ToString()
         {
             StringBuilder Builder = new StringBuilder();
             Builder.Append(string.Format("<Test><Class name=\\"{0}\\" /><Method name=\\"{1}\\" />",
                                             MethodInformation.DeclaringType.Name,
                                             MethodInformation.Name));
             if (Exception == null)
                 Builder.Append("<Passed />");
             else
                 Builder.Append(string.Format("<Failed>{0}</Failed>", Exception.ToString()));
             Builder.Append("</Test>");
             return Builder.ToString();
         }
  
         #endregion
  
         #region Properties
  
         public virtual MethodInfo MethodInformation { get; private set; }
  
         public virtual Exception Exception { get; set; }
  
         #endregion
     }
 }

This class simply stores the method information and the result for an individual test and can export that data it to a string later. That's it. With these four classes, we have our set of classes for finding tests to run, running them, and storing the results.  That just leaves us with assertions and exceptions. Assertions are simply predicates that should be true at a specific position in our code. Which really just means they're a set of functions that determine if something is truly in the state that we're expecting. In some systems like nUnit, there's like 60 of these functions (most are overloaded versions of themselves). In xUnit, there's much fewer (mostly because they use generics, etc. to achieve the same things). In our case we're going to use the xUnit style (in fact many of them are the same name and look about the same, because... Well... There's not much to them really). Anyway, let's take a look at a couple:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using MoonUnit.Exceptions;
 using System.Collections;
 using System.Text.RegularExpressions;
 #endregion
  
 namespace MoonUnit
 {
     /// <summary>
     /// Used to make assertions
     /// </summary>
     public static class Assert
     {
         #region Functions
  
         #region DoesNotThrow
  
         /// <summary>
         /// Used when a specific type of exception is not expected
         /// </summary>
         /// <typeparam name="T">Type of the exception</typeparam>
         /// <param name="Delegate">Delegate called to test</param>
         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
         public static void DoesNotThrow<T>(VoidDelegate Delegate, string UserFailedMessage = "Assert.DoesNotThrow<T>() Failed")
         {
             try
             {
                 Delegate();
             }
             catch (Exception e)
             {
                 if (e is T)
                     throw new DoesNotThrow(typeof(T), e, UserFailedMessage);
             }
         }
  
         /// <summary>
         /// Used when a specific type of exception is not expected and a return value is expected when it does not occur.
         /// </summary>
         /// <typeparam name="T">Exception type</typeparam>
         /// <typeparam name="R">Return type</typeparam>
         /// <param name="Delegate">Delegate that returns a value</param>
         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
         /// <returns>Returns the value returned by the delegate or thw appropriate exception</returns>
         public static R DoesNotThrow<T, R>(ReturnObjectDelegate<R> Delegate, string UserFailedMessage = "Assert.DoesNotThrow<T,R>() Failed")
         {
             try
             {
                 return Delegate();
             }
             catch (Exception e)
             {
                 if (e is T)
                     throw new DoesNotThrow(typeof(T), e, UserFailedMessage);
             }
             return default(R);
         }
  
         #endregion
  
         #region Empty
  
         /// <summary>
         /// Determines if an IEnumerable is empty or not
         /// </summary>
         /// <param name="Collection">Collection to check</param>
         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
         public static void Empty(IEnumerable Collection, string UserFailedMessage = "Assert.Empty() Failed")
         {
             if (Collection == null)
                 throw new ArgumentNullException("Collection");
             foreach (object Object in Collection)
                 throw new NotEmpty(Collection,UserFailedMessage);
         }
  
         #endregion
  
         #region Equal
  
         /// <summary>
         /// Determines if two objects are equal
         /// </summary>
         /// <typeparam name="T">Object type</typeparam>
         /// <param name="Expected">Expected result</param>
         /// <param name="Result">Actual result</param>
         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
         public static void Equal<T>(T Expected, T Result,string UserFailedMessage="Assert.Equal() Failed")
         {
             Equal(Expected, Result, new InternalClasses.EqualityComparer<T>(), UserFailedMessage);
         }
  
         /// <summary>
         /// Determines if two objects are equal
         /// </summary>
         /// <typeparam name="T">Object type</typeparam>
         /// <param name="Expected">Expected result</param>
         /// <param name="Result">Actual result</param>
         /// <param name="Comparer">Comparer used to compare the objects</param>
         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
         public static void Equal<T>(T Expected, T Result, IEqualityComparer<T> Comparer, string UserFailedMessage = "Assert.Equal() Failed")
         {
             if (!Comparer.Equals(Expected, Result))
                 throw new NotEqual(Expected, Result, UserFailedMessage);
         }
  
         #endregion
  
         #region False
  
         /// <summary>
         /// Determins if something is false
         /// </summary>
         /// <param name="Value">Value</param>
         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
         public static void False(bool Value, string UserFailedMessage = "Assert.False() Failed")
         {
             if (Value)
                 throw new NotFalse(UserFailedMessage);
         }
  
         #endregion
  
         #endregion
     }
 }

The code above only contains four of the assertions that I've added to the system. Anyway, the basic Assert class is static with a number of static functions. In this case we're looking at DoesNotThrow, Empty, Equal, and False. DoesNotThrow takes in a delegate, which it calls, and sees if it throws a specific error, the Empty function just checks if an IEnumerable is empty (not null, but empty), the equal function checks if two items are equal using an IEqualityComparer, and False just checks if a boolean value is false. Every single assertion is like this, very simple, maybe 5 lines of code, and that's it. The Equals function (and others like it) are a bit more complex as you need to write an IEqualityComparer but I've written one that the system will use by default:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using MoonUnit.Exceptions;
 using System.Reflection;
 using System.Collections;
 #endregion
  
 namespace MoonUnit.InternalClasses
 {
     /// <summary>
     /// Internal equality comparer
     /// </summary>
     /// <typeparam name="T">Data type</typeparam>
     internal class EqualityComparer<T>:IEqualityComparer<T>
     {
         #region Functions
  
         public bool Equals(T x, T y)
         {
             if (!typeof(T).IsValueType
                 || (typeof(T).IsGenericType
                 && typeof(T).GetGenericTypeDefinition().IsAssignableFrom(typeof(Nullable<>))))
             {
                 if (Object.Equals(x, default(T)))
                     return Object.Equals(y, default(T));
                 if (Object.Equals(y, default(T)))
                     return false;
             }
             if (x.GetType() != y.GetType())
                 return false;
             if (x is IEnumerable && y is IEnumerable)
             {
                 EqualityComparer<object\> Comparer = new EqualityComparer<object\>();
                 IEnumerator XEnumerator = ((IEnumerable)x).GetEnumerator();
                 IEnumerator YEnumerator = ((IEnumerable)y).GetEnumerator();
                 while (true)
                 {
                     bool XFinished = !XEnumerator.MoveNext();
                     bool YFinished = !YEnumerator.MoveNext();
                     if (XFinished || YFinished)
                         return XFinished & YFinished;
                     if (!Comparer.Equals(XEnumerator.Current, YEnumerator.Current))
                         return false;
                 }
             }
             if (x is IEquatable<T>)
                 return ((IEquatable<T>)x).Equals(y);
             if (x is IComparable<T>)
                 return ((IComparable<T>)x).CompareTo(y) == 0;
             if (x is IComparable)
                 return ((IComparable)x).CompareTo(y) == 0;
             return x.Equals(y);
         }
  
         public int GetHashCode(T obj)
         {
             throw new NotImplementedException();
         }
  
         #endregion
     }
 }

Oh and if you're at all experienced with xUnit's code base, it's pretty much the same as the one they use. Slightly different but there's only so many ways to write these things. Anyway, if you look at the code, if the assertion fails you will notice that it throws an exception. These exceptions use the following base class:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 #endregion
  
 namespace MoonUnit.BaseClasses
 {
     /// <summary>
     /// Base exception class
     /// </summary>
     public class BaseException:Exception
     {
         #region Constructor
  
         /// <summary>
         /// Constructor
         /// </summary>
         /// <param name="ExceptionText">Exception Text</param>
         /// <param name="Expected">Expected value</param>
         /// <param name="Result">Actual result</param>
         public BaseException(object Expected, object Result,string ExceptionText)
             : base(ExceptionText)
         {
             this.Expected = Expected;
             this.Result = Result;
         }
  
         #endregion
  
         #region Properties
  
         /// <summary>
         /// Expected result
         /// </summary>
         public virtual object Expected { get; private set; }
  
         /// <summary>
         /// Actual result
         /// </summary>
         public virtual object Result { get; private set; }
  
         #endregion
  
         #region Functions
  
         public override string ToString()
         {
             StringBuilder Builder = new StringBuilder();
             Builder.Append(string.Format("<Expected>{0}</Expected><Result>{1}</Result><ErrorText>{2}</ErrorText><Trace>{3}</Trace><ErrorType>{4}</ErrorType>",
                                         Expected, 
                                         Result, 
                                         this.Message, 
                                         this.StackTrace,
                                         this.GetType().Name));
             return Builder.ToString();
         }
  
         #endregion
     }
 }

The exception base class just takes in the expected value and the result and spits out the result to text when ToString is called. Fairly simple. An an example of an exception class would look like this:

 #region Usings
 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using MoonUnit.BaseClasses;
 #endregion
  
 namespace MoonUnit.Exceptions
 {
     /// <summary>
     /// Exception thrown if two items are equal
     /// </summary>
     public class Equal : BaseException
     {
         #region Constructor
  
         /// <summary>
         /// Constructor
         /// </summary>
         /// <param name="ExceptionText">Exception Text</param>
         /// <param name="Expected">Expected value</param>
         /// <param name="Result">Actual result</param>
         public Equal(object Expected, object Result,string ExceptionText)
             : base(Expected, Result, ExceptionText)
         {
         }
  
         #endregion
     }
 }

The vast majority of the exception classes just look like that (note that the one above is actually thrown from the NotEqual function that I didn't show you and not the Equal function). So create about 20 assertions, have 20 exceptions that match up, and we're done. With this we can create a class library, have the system load it, look through for our tests and give us results in an XML doc. The only thing left is to write a front end of some description, but I'll leave that for part 2. Oh and if you've paid attention to the namespaces, yes, I have indeed named this thing MoonUnit, and yes I will be releasing this on Codeplex. Anyway, take a look, leave feedback, and happy coding.

Unit Testing xUnit.net