Making Nunit tests execute in order

Have you ever wanted to run a given suite of Nunit tests in order? Normally your tests should be created in such a way that execution order is no concern. I stand by that rule, however there is that one edge case were you need to break that rule. The one scenario I came across was with R.NET whereby in a given process you can only initialize one R Engine and once disposed you cannot reinitialize. One of my test happened to deal with disposing the engine and if this test were to run before any other test(s) they would fail.

You will find below a code snippet of how I managed to get this to run in order using Nunit.


using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using NUnit.Framework;
using RSamples;

    public class OrderedTestAttribute : Attribute
    {
        public int Order { get; set; }

        public OrderedTestAttribute(int order)
        {
            this.Order = order;
        }
    }

    public class TestStructure
    {
        public Action Test;
    }

    public class SampleTests
    {
        [TearDown]
        public void CleanUpAfterTest()
        {
            REngineExecutionContext.ClearLog();
        }

        [OrderedTest(0)]
        public void Test1(){}

        [OrderedTest(1)]
        public void Test2(){}

        [OrderedTest(2)]
        public void Test3(){}

        [TestCaseSource(sourceName: "TestSource")]
        public void MainTest(TestStructure test)
        {
            test.Test();
        }

        public static IEnumerable<TestCaseData> TestSource
        {
            get
            {
                var assembly = Assembly.GetExecutingAssembly();
                Dictionary<int, List<MethodInfo>> methods = assembly
                    .GetTypes()
                    .SelectMany(x => x.GetMethods())
                    .Where(y => y.GetCustomAttributes().OfType<OrderedTestAttribute>().Any())
                    .GroupBy(z => z.GetCustomAttribute<OrderedTestAttribute>().Order)
                    .ToDictionary(gdc => gdc.Key, gdc => gdc.ToList());

                foreach (var order in methods.Keys.OrderBy(x => x))
                {
                    foreach (var methodInfo in methods[order])
                    {
                        MethodInfo info = methodInfo;
                        yield return new TestCaseData(
                            new TestStructure
                            {
                                Test = () =>
                                {
                                    object classInstance = Activator.CreateInstance(info.DeclaringType, null);
                                    info.Invoke(classInstance, null);
                                }
                            }).SetName(methodInfo.Name);
                    }
                }
            }
        }
    }

DeploymentItemAttribute for NUnit

In a previous post I talked about the proper use of the DeploymentItemAttribute when running MSTests. If you do move to NUnit and have gotten used to depending on this attribute then worry not, I have you covered. Below I have code samples showing the custom attribute I have created to imitate the MSTest behaviour and usage samples. A good scenario of when you might want to use this is when you have a resource used by multiple tests and can be modified by any of the tests, in which case you want each test to have its own fresh copy.


    public class SampleTest : TestBase
    {
        [DeploymentItemAttribute(@"resources\file.txt", "TestName")]
        [Test]
        public void TestName()
        {
        }
    }

    public class TestBase
    {
        [SetUp]
        public void Setup()
        {
            this.DeployItems();
        }

        private void DeployItems()
        {
            var currentType = this.GetType();
            var publicMethodsWithTestAttr = currentType.GetMethods(BindingFlags.Instance | BindingFlags.Public)
                .Where(x => x.CustomAttributes.Any(y=>y.AttributeType == typeof(TestAttribute)));

            //Iterate through each test method and deploy files as necessary
            foreach (MethodInfo testMethodInfo in publicMethodsWithTestAttr)
            {
                var deploymentItemAttrs = testMethodInfo.GetCustomAttributes(typeof(DeploymentItemAttribute));
                foreach (DeploymentItemAttribute deploymentItemAttr in deploymentItemAttrs)
                {
                    this.DeployFile(deploymentItemAttr.Path,deploymentItemAttr.OutputDirectory);
                }
            }
        }

        private void DeployFile(string path, string outputDirectory = null)
        {
            string filePath = path.Replace("/", "\\");

            string originalItemPath = new Uri(Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), filePath)).LocalPath;
            string originalItemName = Path.GetFileName(originalItemPath);

            string runFolderPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);

            Debug.WriteLine("DeploymentItem: Copying " + originalItemPath + " to " + runFolderPath);

            string itemPathInBin;
            if (string.IsNullOrEmpty(outputDirectory))
            {
                itemPathInBin = new Uri(Path.Combine(runFolderPath, originalItemName)).LocalPath;
            }
            else if (!string.IsNullOrEmpty(Path.GetPathRoot(outputDirectory)))
            {
                itemPathInBin = new Uri(Path.Combine(outputDirectory, originalItemName)).LocalPath;
            }
            else
            {
                itemPathInBin = new Uri(Path.Combine(runFolderPath, outputDirectory, originalItemName)).LocalPath;
            }

            if (File.Exists(originalItemPath)) // It's a file
            {
                string parentFolderPathInBin = new DirectoryInfo(itemPathInBin).Parent.FullName;

                if (!Directory.Exists(parentFolderPathInBin))
                {
                    Directory.CreateDirectory(parentFolderPathInBin);
                }

                File.Copy(originalItemPath, itemPathInBin, true);

                FileAttributes fileAttributes = File.GetAttributes(itemPathInBin);
                if ((fileAttributes & FileAttributes.ReadOnly) != 0)
                {
                    File.SetAttributes(itemPathInBin, fileAttributes & ~FileAttributes.ReadOnly);
                }
            }
            else if (Directory.Exists(originalItemPath)) // It's a folder
            {
                if (Directory.Exists(itemPathInBin))
                {
                    Directory.Delete(itemPathInBin, true);
                }

                Directory.CreateDirectory(itemPathInBin);

                foreach (string dirPath in Directory.GetDirectories(originalItemPath, "*", SearchOption.AllDirectories))
                {
                    Directory.CreateDirectory(dirPath.Replace(originalItemPath, itemPathInBin));
                }

                foreach (string sourcePath in Directory.GetFiles(originalItemPath, "*.*", SearchOption.AllDirectories))
                {
                    string destinationPath = sourcePath.Replace(originalItemPath, itemPathInBin);
                    File.Copy(sourcePath, destinationPath, true);

                    FileAttributes fileAttributes = File.GetAttributes(destinationPath);
                    if ((fileAttributes & FileAttributes.ReadOnly) != 0)
                    {
                        File.SetAttributes(destinationPath, fileAttributes & ~FileAttributes.ReadOnly);
                    }
                }
            }
            else
            {
                Debug.WriteLine("Warning: Deployment item does not exist - \"" + originalItemPath + "\"");
            }
        }
    }

    [AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Struct, AllowMultiple = true, Inherited = false)]
    public class DeploymentItemAttribute : Attribute
    {
        public DeploymentItemAttribute(string path, string outputDirectory = null)
        {
            this.Path = path;
            this.OutputDirectory = outputDirectory;
        }

        public string Path { get; set; }

        public string OutputDirectory { get; set; }
    }

Just don’t do it session #2 – Unit test mistakes

Having worked on a few large projects I have learnt to appreciate disciplined unit testing more and more. As such there are a few things that I have seen that mess with maintaining this discipline.

  1. Misleading unit test names

I have found a lot of unit tests that end up rotting because they communicate falsely what they do. When maintaining and cleaning up unit tests especially during a bug fix it can be quite frustrating to have to go through a unit test line by line to understand what it does. Hand in hand with this it is important that each test does something very specific avoiding very general unit test names that are not very clear.

2. Allowing failing tests to go unattended

Quite simply, never do this EVER! Much like a large residential building allowing one broken window to go unaddressed will make it easy for continued abuse to creep one until a point is reached when addressing these problem is difficult. I have seen this so many times were tests that were ignored for a month require months of extra effort to get them green again.

3. Creating tests that are affected by the environment they are run in and running sequence

A tricky scenario to pinpoint and address is when you have tests that fail intermittently and developers have to argue often that tests run well on their machine. Lets ignore things that are out of our control such as differences in .NET frameworks installed. There are a number of things that can be neglected that lead to this. A common one is failure to deploy files appropriately and this I have addressed in a recent post. You end up having tests that fail because one runner cannot find the files while others can or tests that share a file and expect it to be clean but one test updates it before the other. Similarly to this you can end up having test that pass/fail based on the run order which really is not in your control. You might wonder how, a common one I have seen is a required dependency that in some cases is there because a certain test ran first and had that resolved and fails when that particular test does not run first. Make it a priority that you tests are isolated and run in all the environments they are required to.

4. Viewing unit tests as tool to find bugs or regression issues

There are many more effective practices such as manual testing and automated integration testing that can address this. Unit tests are more valuable as a development process that verifies the expected behavior of components/unit.

5. Testing performance in unit tests

6. Creating long running unit tests

7. Writing unit tests when the code being tested has become large and complex

8. Expending energy on testing the language/frameworks/trivial code/boiler plate code

9. Testing configuration settings

10. Testing Garbage Collection!

11. Not mocking out externals

 

A few good things to try and do

  • Make use of the red-green-refactor approach
  • Make use of the arrange-act-assert approach
  • Write code that is easy to unit test
  • Do code reviews on unit tests
  • Make a use of test categories
  • Test as part of continuous integration/build process
  • Create new tests for every defect
  • Run code coverage metrics
  • Make use of descriptive messages in asserts
  • Write tests that cleanup after themselves
  • Ensure that the entire team is clued up on proper unit testing. Consider having a small bootcamp to address this. You entire suite of tests is as good as the worst test in there.

 

Proper user of DeploymentItemAttribute

One of the necessary evils of working with MS Test is knowing how to make use of DeployItemAttribute to ensure proper deployment of files when unit tests are run.

One thing that makes it challenging for programmers to appreciate this attribute is the inconsistency of how unit tests are executed, making it possible for some file accesses to work depending on how the test is run then suddenly when you deploy to team city you have to argue that it worked on your machine.

The DeploymentItemAttribute allows you to indicate which files you want to be available when running the unit tests with an option to have files copied to a unique folder for each test to ensure a clean copy for each tests. The DeploymentItemAttribute applies to test methods and can be applied multiple times.

    [TestClass]
    public class UnitTest1:UnitTest
    {

        [TestMethod]
        [DeploymentItem(@&quot;Resources\sample.txt&quot;, &quot;TestMethod1&quot;)]
        //Second parameters is optional and if omitted files will be deployed to the same TestMethod folder
        public void TestMethod1()
        {
            var path = this.GetDeploymentItemPath(&quot;sample.txt&quot;);
            Assert.IsTrue(true);
        }
    }

The code sample above makes use of an extension method to resolve the path to the deployed file during test run and a base class for all unit tests.

   public class UnitTest
    {
        public TestContext TestContext { get; set; }
    }

    public static class UnitTestExtensions
    {
        [MethodImpl(MethodImplOptions.NoInlining)]
        public static string GetDeploymentItemPath(this UnitTest unitTest, string filename, int skipFrames = 1)
        {
            var stackTrace = new StackTrace(1, false);
            var method = stackTrace.GetFrame(0).GetMethod();
            var attributes = method.GetCustomAttributes(typeof(DeploymentItemAttribute), false);

            var deploymentItemAttributes = attributes.Cast&lt;DeploymentItemAttribute&gt;().ToList();

            var targetDeploymentItem = deploymentItemAttributes.FirstOrDefault(x =&gt; x.Path.IndexOf(filename, StringComparison.Ordinal) != -1);
            string path = Path.Combine(unitTest.TestContext.TestDeploymentDir, targetDeploymentItem.OutputDirectory, Path.GetFileName(targetDeploymentItem.Path));
            return path;
        }
    }