Getting up to speed with TestBox

Over the past several months, I've been trying to work more habits into my workflow that make the coding product I'm creating better, more solid and done with a higher degree of confidence and less back and forth between myself and QA. One way that I've been trying to do that is by implementing some Unit and Integration Tests into my work flow rather than just simply "code and reload" and see if it worked.

The idea is that we can run pieces of our code in place to see if the individual functions in our CFCs are working as expected and going from the bottom up to see if the entire process is working as we expect.

Last fall at CFUnited, I took a training course with Ortus Solutions and discovered CommandBox and Testbox which have changed the way that I approach projects.  There are a number of ways to set this up but here is what seems to be working for me

Prepping the environment

1. Open up CommandBox and navigate to the root of your site. CommandBox isn't essential for this process but it is going to make everything much much easier. 
2. Install TestBox by typing "install testbox --saveDev". This will download TestBox from ForgeBox (the online repo) and put into a folder called "testbox" in the root of your site. This is important because TestBox needs to be accesible via both a browser and via createobject(). It need to be accessible via createObject because the tests we write are going to extend "testbox.system.basespec" to have access to the testing framework. The browser needs access because we can create "runners" which will make the calls to run the tests. One type of runner is a browser based .cfm page which can run your tests and present the results to you. It's also possible to set up configurations for build systems and more complicated setups but we're going to keep it simple. 
3. Set up the location where you are going to house your tests. For me I just made a "tests" folder in the root of the website next to the testbox folder. More on that later. 
4. In the /testbox/test-browser/ folder, copy the index.cfm file to your /tests/ folder. This is going to serve as our default "test runner" for now. 
5. Open the index.cfm file you just copied and change the root mapping to
   <cfset rootMapping = "/tests">
6. In CommandBox, go tot the root of the website ("cd .." from tests) and type "server start". This will spin up an incidence of Lucee and a webserver on your computer and open up your site in your default browser. Whether your site is able to spin up without any additional configuration is beyond the scope of this document but assuming you can open the site, continue. 

So far our folder structure is 

root

        testbox

        tests

        other site files

The plan of attack

I was working on an PDF renderer which would accept a drag and drop jQuery ajax submission of a file, convert it to a PDF file, save the info to the database and then return the results of the process to the user. There was the added complication of the user submitting .msg files from Outlook which would include attachments which would need to be included but we don't need to get that complicated at this point.

The methods and CFCs for the project fell into four levels:

1. The gatekeeper which accepted the file submission from and the returned the response to the user. It also handled some simple immediate tasks such see if the file was uploaded, supported etc
2. The decision maker - This looked at the type of file which was submitted and sent the file to the correct function to be processed
3. The processing files which were similar in their job but since the process of converting an image to a PDF in CF is different from how a document is processed which is different from how an email was processed, there are different functions. 
4. The micro-services layer which includes some generic functions which already existed on on the site. Many of these were either DAOs, user processing functions etc which didn't need to be duplicated. 

Why is this so important?

Just to be clear, my project wasn't this organized when I started. It morphed into this but I'm laying it out so that you can see the end result and how it impacted the testing set up. Each level is dependent on the one below it which reveals the need for two types of tests.

Types of Tests

There are several types of testing. Two of these are:

Unit Testing - this tests one and only one method to make sure that it is working as expected. 

Integration Testing - This tests how two or more methods interact. I might have two methods which pass their unit tests with flying colors but they aren't talking to each other. The unit tests would pass. The Integration test would fail. This also might mean going back to re-writing some of your unit tests so that they would catch what you know know is a flaw. 

In the above "DML*" chart, it's pretty clear that the microservices functions on the bottom are the most specialized. They do one thing and one thing only. These are ripe for some self contained unit tests so let's start there. 

Planning your tests organization

A couple quick points about organizing your tests

1. To a point, it doesn't matter where they go as long as then can "see" the testbox folder and access it. They don't necessarily access each other so their relative location to other tests wasn't functionally important. I was constantly moving my tests around as I tried to figure out the best way to organize them. 
2. There are pros and cons to every organization structure so there will be some trial and error. This includes what tests are run together and also the order that the tests are run. It is very easy to get bogged down in the "best" way to do something so to avoid that temptation, we're just going to jump in and start. 

Creating The First Unit Tests

Let's say that there is a FolderFunctions.cfc which contains multiple methods, one of which is called createUserFolders. This ensures that a series of folders which the users' account needs to exist, actually do exist. What we are testing can be summed up with "If I submit this base folder path, do the resulting folders then exist?". To keep things as non-complicated as possible, we're going to make a folder for each of the CFCs in our codebase and then make one testing suite for each method. If that isn't clear, it will be and we'll revisit that decision later. 

Steps to creating our first test:

1. Open Command Box and navigate to our tests folder
2. Create a folder called MicroServices by typing "mkdir MicroServices"
3. Navigate into the MicroServices folder ("cd MircoServices") and then create  folder called FolderFunctions by typing "mkdir FolderFunctions".
4. Navigate into FolderFunctions ("cd FolderFunctions").
5. Have TestBox create the first test suite by typing "testbox create bdd name=createUserFolders". This creates a CFC called createUserFolders.cfc which looks like this:

/*** My BDD Test*/
component extends="testbox.system.BaseSpec"{
   
/*********************************** LIFE CYCLE Methods ***********************************/
   // executes before all suites+specs in the run() method   
   function beforeAll(){
   }

   // executes after all suites+specs in the run() method   
   function afterAll(){
   }

/*********************************** BDD SUITES ***********************************/
   function run(){
   
      describe( "My test Suite", function(){
         
         it( "should do something", function(){
                expect( false ).toBeTrue();
         });
         
         it( "should do something else", function(){
                expect( false ).toBeTrue();
         });
         
      });
      
   }
   
}

A couple of quick thoughts on this

 - It's in the (relatively) new script format for CF. Don't let that intimidate you if you haven't made the change yet. It will work perfectly well on the same server as your existing tag based CFCS and scripts (as long as they're running on a recent enough version of CF).

- If it's not intuitively clear what's happening, stay tuned and we'll parse the functions. 

Files, Suites, Tests and Specs

A quick vocab lesson about TestBox and testing in general from the very specific to the very general:

• Spec: a spec is a very specific requirement for which we are testing. Examples could include;
  • "This result should be a struct".
  • "This struct should have the key 'numberFilesRendered'".
  • "The key numberFileRendered should be numeric".
  • "The key 'numberFilesRendered' should be greater than 0"
  Specs correspond to the "expect" statements.
• Tests: tests are groups of specs that are grouped under the same title. "The result should be a struct" could be a test which would have one spec. "The result should have the following 5 keys" could be a test with 5 specs within it. Tests correspond with the "it" functions.
• Suites: A suite is a collection of tests all grouped under the same title. Suites correspond with the "describe" functions.
• Files: Files are just that files. The createUserFolders.cfc is a file. No major confusion there but I mention it here because Suites can actually span mulitple files which we'll look at in the next post. 

Writing the Test Suite for our createUserFolders method

We are going to instantiate the FolderFunctions object, submit a fake folder structure to it, and then check and make sure that all of the folders were created. Here is what the createUserFolders method looks like:

<cffunction name="createUsersFolders">
    <cfargument name="userpath" default="">

    <cftry>
        <cfif userpath eq ''>
           <cfset userPath=createObject("component","folders").returnFilePath(whichFilePath='checkinpath',whichSubFolder='')>
        </cfif>

        <cfset folderArray=[APPLICATION.fileServer_CheckIn,userpath,'#userpath#input','#userpath#pdf','#userpath#originals']>

        <cfloop array="#folderArray#" index="path">
            <cfif not directoryExists(path)>
                <cfset directoryCreate(path)>
            </cfif>
        </cfloop>

        <cfset returnme={success=true,message=""}>

        <cfreturn returnme>

        <cfcatch type="any">

            <cfset returnme={success=true,message=cfcatch.message}>

            <cfreturn returnme>  

        </cfcatch>

    </cftry>
</cffunction>

This is pretty straightforward, it accepts the folder we give it, if there is no path given, it gets one according to another function with existing rules. It then loops over an array of file paths, creating the ones which need to be created and then returning a structure with two keys - success and message with true/false and a message as needed. 

Questions based on this example

1. Why did I do it in tag format rather than script - absolutely no reason except for that there is alot of existing code in tag format and I wanted to demonstate that it still worked.
2. Why do you make allowances for passing in userpath AND then as a backup get the system assigned path? -There are two main reasons for this:
   
   1. Good tests are designed to test a very specific method completely apart from the outside world. If we relied on the other method to provide the path for us, we would actually be testing both the createUserFolders method AND the returnFilePath method at the same time. If something breaks, in what method was the problem? Smaller functions with definite tasks allow for easier testing which leads to more solid diagnosing later. 
   2. By allowing a path to be passed in, it means that we can pass in a file path that is completely random and will not impact our actual working environment and therefore minimize the chance that something will be overwritten or deleted by our test.
3. Why did you have a separate line for setting the "returnme" variable rather than simply setting the implicit variable in the <cfreturn> tag. This code was developed recently on a CF10 and we found a weird bug concerning implicit variables and <cfreturn> which has long been cleared up in CF itself, but I kept the formatting for this example. This had absolute nothing to do with TestBox.  

Let's get On with it!

The first thing we need to do for each of our tests is to instantiate the FolderFunction cfc and pass a path into it. This is easy enough -
pathWeAreTesting="c:\my\path\to\folder";
myResult = createobject("component","folderFunctions").createUserFolders(pathWeAreTesting);

Remember we're doing this in script format but you can how this works, even if you're only used to tag format. This gives us a variable of myResult which is the structure with the success and message in it and also creates the folders on the harddrive. But where do we put that? Since we're going to be starting simple and getting more complicated, let's put it right into a test.

In the tests/MmicroServices/FolderFunctions/createUserFolders.cfc file, put the two lines in the first "it" function you see. Your describe function will look like this:

describe( "My test Suite", function(){
   
   it( "should do something", function(){
      pathWeAreTesting="c:\my\path\to\folder";
      myResult = createobject("component","includesYacht.CFCSYacht.PDFFileFunctions");
              expect( false ).toBeTrue();
   });
   
   it( "should do something else", function(){
              expect( false ).toBeTrue();
   });
   
});

Let's actually run that test which means we need to set up a simple way to run tests.

Running Your First Test Runner

In your browser, navigate to http://127.0.0.1:####/tests/. If you don't remember the port that CommandBox used simply type "server open" from the root of your site then go to /tests. You'll see something like this:

The buttons are .cfm and cfc files. Interpersed are links to folders and other files. Click on the one we made for FolderFunctions and then on the button for our createUsersFolder.cfc and, in a new tab, you'll see:

This says we ran 1 suite consisting of two tests which had two failures. This makes sense since there were two tests made automatically when the page was made and we haven't done anything to make them pass yet so, so far, all is good. Since we also didn't get an error, that means that it was able to instantiate the FolderFunctions object.

Fleshing Out Our First test

We're going to do this step by step.

Step 1
Add the call to the end of the createObject to make it run the createUsersFoldersMethod and pass in the path we're testing. Also, let's have it dump out the result it's getting

it( "should do something", function(){
   pathWeAreTesting="c:\my\path\to\folder";
   myResult = createobject("component","includesYacht.CFCSYacht.PDFFileFunctions").createUsersFolders(pathWeAreTesting);
   writeDump(myresult);
             expect( false ).toBeTrue();
});

As you might expect, we get this:

Notice two thing. First, the dump is above the results display because TestBox is running the entire page and then displaying the results afterwards. Second, the same tests we had before are still there. Well, yes. We haven't actually tested our results yet. Let's go on to step 2.

Step 2

It may not need to be said but in upacking the describe, it, and expect statements, we see that they are regular functions. Regular CFscript functions. Looking at them one at a time 

describe( "My test Suite", function(){

});

If we were going to use named parameters instead of positional parameters it would look like this:

describe( label="My test Suite", body=function(){

})

We're passing the anonymous function as the parameter, just like we would pass in a string or  struct. This is saying "Run this suite and call it 'My Test Suite. The list of tests to run are in the body parameter". We can rewrite the title of our suite to something else like "The createUserFolder method should...".

When we look at an it() function we see the same basic format where we pass in a label and a function which contains all of the specs for that test. 

it( "should do something", function(){
   pathWeAreTesting="c:\my\path\to\folder";
   myResult = createobject("component","includesYacht.CFCSYacht.PDFFileFunctions").createUsersFolders(pathWeAreTesting);
   writeDump(myresult);
   expect( false ).toBeTrue();
});

The "should do something" is the label and the function as the list of specs. This test is going to specifically test is the result we get is a Struct so rename the test "be a struct". If you read the describe label and the it label together you get "The createUserFolder method should...be a struct' which presents a very clear description about what it is intended to test. 

Which brings us to the actual spec:

expect( false ).toBeTrue();

There are three parts to this: the value, the matcher and the expectation. The value in our case is the myResult variable. The matcher in the example above is "toBeTrue()". Well, as the name indictes, this expects the value passed in to be true. If you notice, the default function is passing in false, so it is no surprise that it is failing since false does not equal true. We're going to change this spec to what we actually want to test which looks like this:

expect( myResult ).toBeTypeOf("struct"); 

When we refresh our runner page, we get this:

The green text indicates a successful test. Best Practice would dictate that you write the test first, have it fail, then change the code as needed to pass the test. Sometimes in refactoring that might be more difficult but it does lead to fewer false positives. 

Step 3

Finishing up our initial test we might get these specs:

describe( "The createUserFolder method should...", function(){
   it( "Return a struct", function(){
      pathWeAreTesting="c:\my\path\to\folder";
      myResult = createobject("component","includesYacht.CFCSYacht.PDFFileFunctions").createUsersFolders(pathWeAreTesting);
      writeDump(myresult);
              expect( myResult  ).toBeTypeOf("struct");
   });
   
   it( "return a struct with the two keys: succcess and message", function(){
              expect( myResult ).tohavekey("success");
      expect( myResult ).tohavekey("message");
   });
   it("should have the folder c:\my exist",function(){
      expect(directoryExists("c:\my")).tobetrue();
   });
   it("should have the folder c:\my\path exist",function(){
         expect(directoryExists("c:\my\path")).tobetrue();
   });
   it("should have the folder c:\my\path\to exist",function(){
         expect(directoryExists("c:\my\path\to")).tobetrue();
   });
   it("should have the folder c:\my\path\to\folder to exist",function(){
         expect(directoryExists("c:\my\path\to\folder")).tobetrue();
   });
});

If we look at the value we're passing in they include a struct and a function that returns a boolean (directoryexists). For our expectations we used the built in matchers of "toBeTypeOf" to test for datatype and "toBeTrue" to test the integrity of the statement. There are dozens of matchers built into TestBox and you can aso create your own custom ones as well. For the remaining tests, we are passing in the result of directoryExists which is a boolean so we can use the toBeTrue() matcher. 

Step 4 (The last)

These sets of specs have changed the "state" of the hard drive meaning that they haven't left everything as they found it. Any subsequent test is not testing in the same environment. As a result, tests sometimes need to clean up after themselves. The two functions at the top of the CFC beforeAll() and afterAll() can be use to set up your tests and also to tear down (or clean up). 

If we do a small refactor of our testing cfc we can move the instantiation of the object and passing in of our path to the beforeAll() function. This is run....well, before all the suites so variables declared there can be used in the it() (test) functions. Because of how the life cycle events are set up, anything in the beforeAll() method is not available in the describe() functions but are available in the it() functions. 

In the afterAll() method, we can add a directoryDelete call to remove the c:\my\path\to\folder folders so to reduce clutter and to restore the original state and keep the tests. 

Here is the whole page:

/*** My BDD Test*/component extends="testbox.system.BaseSpec"{
   
/*********************************** LIFE CYCLE Methods ***********************************/
   // executes before all suites+specs in the run() method   function beforeAll(){
      pathWeAreTesting="c:\my\path\to\folder";
      myResult = createobject("component","includesYacht.CFCSYacht.PDFFileFunctions").createUsersFolders(pathWeAreTesting);
   }

   // executes after all suites+specs in the run() method   function afterAll(){
      directoryDelete("C:\my",true);
   }

/*********************************** BDD SUITES ***********************************/
   function run(){
   
      describe( "The createUserFolder method should...", function(){
         it( "Return a struct", function(){
                expect( myResult  ).toBeTypeOf("struct");
         });
         
         it( "return a struct with the two keys: succcess and message", function(){
                expect( myResult ).tohavekey("success");
            expect( myResult ).tohavekey("message");
         });
         it("should have the folder c:\my exist",function(){
            expect(directoryExists("c:\my")).tobetrue();
         });
         it("should have the folder c:\my\path exist",function(){
               expect(directoryExists("c:\my\path")).tobetrue();
         });
         it("should have the folder c:\my\path\to exist",function(){
               expect(directoryExists("c:\my\path\to")).tobetrue();
         });
         it("should have the folder c:\my\path\to\folder to exist",function(){
               expect(directoryExists("c:\my\path\to\folder")).tobetrue();
         });
      });
      
   }
}

Conclusion

As much as I knew it was a good idea, getting started with any sort of testing was a bit daunting until I started just doing it. Things started progressing faster as I did it and soon I was using the tests effectively to diagnose problems early including a situation where I was going in development circle where a tweak in area 1 was breaking area 2 which I would then fix and break area 1 which I would fix and break area 2 and so on. 

To find out more about TestBox and how to proceed beyond this, the documentation is located at https://testbox.ortusbooks.com/content/.

Have fun!

*Dan Modelling Language