[mini-HOWTO] Writing Regression Tests For SBCL

Most of the complex software systems have some kind of regression tests, and SBCL is not an exception. In this short post, I want to answer the next two questions:

1. Why it is preferable to provide a regression test with the bug report?
2. How to write such a test?

Frankly speaking, regression tests are not necessary for bug reports. One may just say “this code is compiled wrongly”, and then attach the code snippet. However, such reports can be hard to understand, especially when we are trying to figure out the desired behavior. In contrary, regression tests always show the desired behavior, so they make the report more informative.

Moreover, your report with a regression test always saves the time for SBCL developer who will fix the corresponding bug. A regression test is mandatory for the most of patches, so someone should write it anyway. I do not know about the others, but I prefer the next steps while fixing a bug in SBCL:

- write a regression test. Make sure that it fails before the fix;
- fix the bug;
- make sure that all regression tests pass.

The issue originator knows the problem very well, so he or she can provide the best regression test.

So, how to write them? It is surprisingly easy in most cases. First of all, you need to find out the right test file to put your test into. These files are located in /sbcl/tests directory. They have the extension “lisp”, and the two important parts before it: the suffix “pure” or “impure”, and the main part which classifies the subject of the tests – in other words, the functionality which is tested. The main part is something like “compiler”, “loop”, “print” etc. The suffix “pure” means that the file contains no tests with global side effects; “impure” files contain tests which define variables, conditions, classes, functions and so on. For example, “loop.pure.lisp” file contains LOOP macro tests without side effects.

After you have selected a file, it's a time to write the test itself. The only constant element of all tests is a WITH-TEST wrapper, which binds a human-readable name to your test (it can do other things as well, but I use only :name keyword. See the macro definition in test-util.lisp to find out more). The rest of the test body may vary (you can use different assertion clauses, bind handlers for compilation errors or warnings etc.).

Here is a very simple (and pure) test from arith.pure.lisp:




The test passes when ASSERT receives non-NIL value. Obviously, you may use any CL code inside ASSERT. For example:




RAISES-ERROR? is an utility macro (defined in assertoid.lisp). It is useful when you want to test that a form evaluation signals an error. The optional second argument can be used to specify the condition type, as in the above example.

Here is another example – the impure test from mop.impure.lisp:




The test is impure, because it creates a class definition. Some old tests (like this) miss WITH-TEST, but it is strongly recommended for new ones.

It is often required to test that SBCL signals (or does not signals) errors and/or warnings during the particular form compilation. This pure LOOP test can be a good example:




The test idea is to check the return values of COMPILE. Another way to do something similar is to use HANDLER-CASE (the example from compiler.pure.lisp):




This test ensures that a compiler note is printed for the non-optimal operation.

In general, SBCL regression tests are very expressive and powerful - we can test quite complex functionality in the several lines of code.

To run the added tests, execute a command (on Linux):

sh run-tests.sh filename

from sbcl/tests directory. Without a file argument, it runs all tests in the regression suite.

I hope that this short post is helpful for those who have never written any SBCL regression tests. Just try to do it when submitting a new bug report – and you will help SBCL developers a lot.