Practical Common Lisp

arithmetic can group related test functions into test suites. Now suppose some low-level test functions are called from multiple test suites. It's not unheard of for a test case to pass in one context but fail in another. If that happens, you'll probably want to know more than just what low-level test function contains the test case.

If you define the test suite functions such as test-arithmetic with deftest and make a small change to the *test-name* bookkeeping, you can have results reported with a 'fully qualified' path to the test case, something like this:

pass ... (TEST-ARITHMETIC TEST-+): (= (+ 1 2) 3)

Because you've already abstracted the process of defining a test function, you can change the bookkeeping details without modifying the code of the test functions.[105] To make *test-name* hold a list of test function names instead of just the name of the most recently entered test function, you just need to change this binding form:

(let ((*test-name* ',name))

to the following:

(let ((*test-name* (append *test-name* (list ',name))))

Since APPEND returns a new list made up of the elements of its arguments, this version will bind *test-name* to a list containing the old contents of *test-name* with the new name tacked onto the end.[106] When each test function returns, the old value of *test-name* will be restored.

Now you can redefine test-arithmetic with deftest instead of DEFUN.

(deftest test-arithmetic ()
  (combine-results
   (test-+)
   (test-*)))

The results now show exactly how you got to each test expression.

CL-USER> (test-arithmetic)
pass ... (TEST-ARITHMETIC TEST-+): (= (+ 1 2) 3)
pass ... (TEST-ARITHMETIC TEST-+): (= (+ 1 2 3) 6)
pass ... (TEST-ARITHMETIC TEST-+): (= (+ -1 -3) -4)
pass ... (TEST-ARITHMETIC TEST-*): (= (* 2 2) 4)
pass ... (TEST-ARITHMETIC TEST-*): (= (* 3 5) 15)
T

As your test suite grows, you can add new layers of test functions; as long as they're defined with deftest, the results will be reported correctly. For instance, the following:

(deftest test-math ()
  (test-arithmetic))

would generate these results:

CL-USER> (test-math)
pass ... (TEST-MATH TEST-ARITHMETIC TEST-+): (= (+ 1 2) 3)
pass ... (TEST-MATH TEST-ARITHMETIC TEST-+): (= (+ 1 2 3) 6)
pass ... (TEST-MATH TEST-ARITHMETIC TEST-+): (= (+ -1 -3) -4)
pass ... (TEST-MATH TEST-ARITHMETIC TEST-*): (= (* 2 2) 4)
pass ... (TEST-MATH TEST-ARITHMETIC TEST-*): (= (* 3 5) 15)
T

Wrapping Up

You could keep going, adding more features to this test framework. But as a framework for writing tests with a minimum of busywork and easily running them from the REPL, this is a reasonable start. Here's the complete code, all 26 lines of it:

(defvar *test-name* nil)
(defmacro deftest (name parameters &body body)
  'Define a test function. Within a test function we can call
   other test functions or use 'check' to run individual test
   cases.'
  `(defun ,name ,parameters
    (let ((*test-name* (append *test-name* (list ',name))))
      ,@body)))
(defmacro check (&body forms)
  'Run each expression in 'forms' as a test case.'
  `(combine-results
    ,@(loop for f in forms collect `(report-result ,f ',f))))
(defmacro combine-results (&body forms)
  'Combine the results (as booleans) of evaluating 'forms' in order.'
  (with-gensyms (result)
    `(let ((,result t))
      ,@(loop for f in forms collect `(unless ,f (setf ,result nil)))
      ,result)))
(defun report-result (result form)
  'Report the results of a single test case. Called by 'check'.'
  (format t '~:[FAIL~;pass~] ... ~a: ~a~%' result *test-name* form)
  result)

It's worth reviewing how you got here because it's illustrative of how programming in Lisp often goes.

You started by defining a simple version of your problem—how to evaluate a bunch of boolean expressions and find out if they all returned true. Just ANDing them together worked and was syntactically clean but revealed the need for better result reporting. So you wrote some really simpleminded code, chock-full of duplication and error-prone idioms that reported the results the way you wanted.

The next step was to see if you could refactor the second version into something as clean as the former. You started with a standard refactoring technique of extracting some code into a function, report- result. Unfortunately, you could see that using report-result was going to be tedious and error-prone since you had to pass the test expression twice, once for the value and once as quoted data. So you wrote the check macro to automate the details of calling report-result correctly.

While writing check, you realized as long as you were generating code, you could make a single call to check to generate multiple calls to report-result, getting you back to a version of test-+ about as concise as the original AND version.

At that point you had the check API nailed down, which allowed you to start mucking with how it worked on the inside. The next task was to fix check so the code it generated would return a boolean indicating whether all the test cases had passed. Rather than immediately hacking away at check, you paused to indulge in a little language design by fantasy. What if—you fantasized—there was already a non-short-circuiting AND construct. Then fixing check would be trivial. Returning from fantasyland you realized there was no such construct but