Unit Tests

Table of Contents

1. Introduction

Recently, a set of regression unit tests has been added to Cyrus. This document explains the purpose implementation of those unit tests, and gives an example of how to add more unit tests (because there are never enough unit tests!).

2. What Is A Unit Test?

The definition on Wikipedia sheds some light:

...unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application.
In other words, unit testing is about verifying that small pieces of code, like individual functions, modules, or classes, work in isolation. It is not about testing the system as a whole.

The tests implemenented here are also regression tests, which in Wikipedia's words means:

Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.

In other words, the tests are designed to be easy to run and to work out fully automatically whether they have passed or failed, so that they can be run usefully by people who didn't write them.

3. Running The Tests

This section takes you through the process of running Cyrus' unit tests.

3.1. Setting Up The Machine

Cyrus' unit tests are all located in a new directory, cyrus-imapd/cunit/. They're written in C, like the remainder of Cyrus, and use the CUnit library from SourceForge, with some home grown wrappers and other improvements to make our lives easier.

Your first step is step is to ensure that the CUnit library (including the headers) is installed. Some modern operating systems already have CUnit, for example on Ubuntu you can just do:

me@ubuntu> sudo apt-get install libcunit1-dev

Alternately, you can download the CUnit source, build it and install it. It's not a complicated or difficult library, this shouldn't take long. When you've done, install it in /usr/include and /usr/lib.

3.2 Configure Script

Because of the dependency on the CUnit library, the tests are disabled by default; this means you need enable them with an option to the configure script:

me@mybox> ./configure --enable-unit-tests ... checking for CU_initialize_registry in -lcunit... yes checking CUnit/CUnit.h usability... yes checking CUnit/CUnit.h presence... yes checking for CUnit/CUnit.h... yes ...

3.3 Make

First you need to build Cyrus itself, using the traditional all: target.

me@mybox> make all ...

Then, use the new check: target to build and run the unit tests.

me@mybox> make check ... ### Making check in /home/gnb/software/cyrus/imapd/lib (a) make[1]: Entering directory `/home/gnb/software/cyrus/imapd/lib' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/home/gnb/software/cyrus/imapd/lib' ### Done with /home/gnb/software/cyrus/imapd/lib ... make[1]: Entering directory `/home/gnb/software/cyrus/imapd/cunit' (b) ... ../cunit/cunit.pl [...] --generate-wrapper mboxname.c (c) gcc -c [...] -g -O2 .cunit-mboxname.c ... gcc [...] -o unit unit.o [...] .cunit-mboxname.o [...] \ (d) ../imap/libimap.a ../lib/libcyrus.a [...] ./unit -v (e) CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: mboxname (f) Test: to_parts ... passed Test: to_userid ... passed Test: same_userid ... passed Test: same_userid_domain ... passed ... --Run Summary: Type Total Ran Passed Failed (g) suites 9 9 n/a 0 tests 51 51 50 1 asserts 474 474 473 1 make[1]: Leaving directory `/home/gnb/software/cyrus/imapd/cunit'

Let's take a closer look at what's happening here.

(a)
The check: target recurses into each subdirectory, which is normal behaviour for most targets in a top-level Makefile but actually rather useless in this case.
(b)
Finally, the check: target recurses into the cunit/ directory, which is where all the interesting stuff happens.
(c)
The cunit/ directory contains a number of C source files with test code in them. For each of those, a small wrapper C source file is generated and then compiled into an object file.
(d)
Finally, all the compiled object files are linked into an executable, with a main() routine from unit.c, and a number of libraries and object files from other parts of the Cyrus tree.
(e)
The resulting executable is then run. The -v flag tells it to be verbose.
(f)
With no other arguments, the test executable runs all the built tests one by one, telling us which ones passed and which ones failed.
(g)
At the end, the text executable prints a summary of how many tests it ran and how many passed and failed.

3.4 Using Valgrind

Some failure modes are subtle, and cannot be detected in the C code itself; this is where the Valgrind program comes in very handy. It detects buffer overruns and memory leaks and various other kinds of subtle errors.

To run the unit tests with Valgrind, use the new valgrind: target.

me@mybox> make valgrind ... valgrind --tool=memcheck --leak-check=full ./unit -v (a) ==2999== Memcheck, a memory error detector ==2999== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==2999== Using Valgrind-3.6.0.SVN-Debian and LibVEX; [...] ==2999== Command: ./unit -v ==2999== ... --Run Summary: Type Total Ran Passed Failed (b) suites 9 9 n/a 0 tests 51 51 50 1 asserts 474 474 473 1 ... ==2999== HEAP SUMMARY: (c) ==2999== in use at exit: 4,489 bytes in 134 blocks ==2999== total heap usage: 715 allocs, 581 frees, 352,763 bytes allocated ==2999== ==2999== 4 bytes in 1 blocks are definitely lost in loss record 3 of 50 ==2999== at 0x4C2815C: malloc (vg_replace_malloc.c:236) ==2999== by 0x44A0CA: xmalloc (xmalloc.c:57) ==2999== by 0x4399D8: strconcat (util.c:631) ==2999== by 0x40C059: test_uncast_null (strconcat.c:51) ==2999== by 0x61B32A9: ??? (in /usr/lib/libcunit.so.1.0.1) ==2999== by 0x61B36ED: ??? (in /usr/lib/libcunit.so.1.0.1) ==2999== by 0x61B3827: CU_run_all_tests (in /usr/lib/libcunit.so.1.0.1) ==2999== by 0x4066CC: run_tests (unit.c:144) ==2999== by 0x406806: main (unit.c:283) ==2999== ...

Here's an explanation of what's happening in the example.

(a)
The test executable is run as before, but using the valgrind program. The first thing we see is Valgrind's banner message.
(b)
The test executable proceeds as normal and eventually emits it's run summary, then exits.
(c)
After the test executable exits, Valgrind checks for memory leaks and prints both a summary of all leaks and a stack trace showing where each block of leaked memory was allocated.

I'd just like to say that I love Valgrind and I think it's immensely useful. I would have made running the tests under Valgrind the only option for the check: target, except that Valgrind is not available on all of Cyrus' supported platforms.

4. Adding Your Own Tests

Adding your own tests is quite simple. Here's how.

4.1 Where To Put Your Tests

The unit test code in Cyrus is contained in a set of C source files in the cunit directory. If you look closely, you will see that each of those C source files maps to a "Suite" in CUnit parlance. For example, cunit/glob.c is listed as the Suite "glob" in CUnit's runtime output.

Typically, each Suite tests a single module or a related set of functions; for example, cunit/glob.c contains tests for the glob module in lib/glob.c.

So, if you want to add a new test for a module which already has some existing tests, the sensible thing to do is to add a new test to the existing suite. Otherwise, you'll need to add a new Suite.

4.1 Adding A New Suite

Each Suite is a single C source file in the cunit/ directory. Your first step is to create a new C source file. For this example, you'll create a new Suite to test the CRC32 routines which live in lib/crc32.c.

me@mybox> vi cunit/crc32.c ...

The file should contain something like this.

/* Unit test for lib/crc32.c */ #include "cunit/cunit.h" (a) #include "crc32.h" (b) static void test_map(void) (c) { static const char TEXT[] = "lorem ipsum"; (d) static uint32_t CRC32 = 0x0; uint32_t c; (e) c = crc32_map(TEXT, sizeof(TEXT)-1); (f) CU_ASSERT_EQUAL(c, CRC32); (g) }

Here's an explanation of what all these bits are for.

(a)
You need to include the header "cunit/cunit.h", which is a thin Cyrus wrapper around the CUnit's library's header, <CUnit/CUnit.h> with some extra conveniences.
(b)
You should also include any headers you need for declarations of the functions which you'll be testing. Note that the Cyrus lib/ and imap/ directories are already in the include path, so any header in there can be included without the directory prefix, e.g. "crc32.h" for lib/crc32.h.
(c)
You need to have at least one function which looks like this: it takes no arguments, returns void, and is named test_whatever. It may be static or extern, but I recommend static. Functions with this signature are automatically discovered in the source code by the Cyrus unit test infrastructure, so all you have to do is write the function. Later, a CUnit test named "whatever" will be created automatically for your test_whatever function.
(d)
Here's a good place to define the test inputs and expected outputs. Note that for this example you have no idea of the actual correct output. The right thing to do there is to manually calculate the expected result from first principles, or to use a different piece of software which you believe to be working. For this example, let's just use a known incorrect value and see what happens.
(e)
Here's a good place for local variables you need during the test.
(f)
Call the function under test (crc32_map() in this example) with known inputs, and capture the results in a local variable c.
(g)
Compare the actual result in c with the expected result in CRC32. The CU_ASSERT_EQUAL() macro checks that it's two arguments are equal (using an integer comparison), and if they're different it prints a message and records a failure. Note that unlike the libc assert() macro, control will continue even if the assert fails. The CUnit library provides a whole family of similar macros, see the online CUnit documentation for more details.

Now you need to tell the Cyrus makefiles about your new Suite.

me@mybox> vi cunit/Makefile.in ...

You need to add the filename of your new test to the definition of the TESTSOURCES variable.

PROGS = unit TESTSOURCES = times.c glob.c md5.c parseaddr.c message.c \ strconcat.c conversations.c msgid.c mboxname.c \ crc32.c TESTLIBS = @top_srcdir@/imap/mutex_fake.o @top_srcdir@/imap/libimap.a \ @top_srcdir@/lib/libcyrus.a @top_srcdir@/lib/libcyrus_min.a

Now you need to use autoconf to build cunit/Makefile from cunit/Makefile.in. The easiest way to do this is to run the config.status shell script which is one of the files that autoconf generates. Alternately, you could re-run the configure script (but that would be a lot slower).

me@mybox> ./config.status ... config.status: creating cunit/Makefile ...

Now, you finally get to build and run your new test code. Run make check and you should see your new code being built and run.

me@mybox> make check ... ../cunit/cunit.pl [...] --add-sources [...] crc32.c ... ../cunit/cunit.pl [...] --generate-wrapper crc32.c gcc -c [...] -g -O2 .cunit-crc32.c gcc [...] -o unit [...] .cunit-crc32.o ... ./unit -v CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: crc32 Test: map ... FAILED 1. crc32.c:12 - CU_ASSERT_EQUAL(c=1926722702,CRC32=0)

Note how the test failure told us which in source file and at what line number the failure occurred, and what the actual and expected values were. Let's go and fix that up now.

static const char TEXT[] = "lorem ipsum"; static uint32_t CRC32 = 0x72d7748e;

Re-run make check and you'll see your test being rebuilt and rerun, and this time passing.

me@mybox> make check ... ../cunit/cunit.pl [...] --generate-wrapper crc32.c gcc -c [...] -g -O2 .cunit-crc32.c gcc [...] -o unit [...] .cunit-crc32.o ... ./unit -v CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: crc32 Test: map ... passed

4.2 Adding A Test To A Suite

Adding a new test to an existing test is easy: all you have to do is add a new function to an existing C source file in the cunit/ directory. As an example, let's add a test for the crc_iovec() function.

me@mybox> vi cunit/crc32.c ...
static void test_iovec(void) (a) { static const char TEXT1[] = "lorem"; (b) static const char TEXT2[] = " ipsum"; static uint32_t CRC32 = 0x72d7748e; uint32_t c; (c) struct iovec iov[2]; memset(iov, 0, sizeof(iov)); (d) iov[0].iov_base = TEXT1; iov[0].iov_len = sizeof(TEXT1)-1; iov[1].iov_base = TEXT2; iov[1].iov_len = sizeof(TEXT2)-1; c = crc32_iovec(iov, 2); (e) CU_ASSERT_EQUAL(c, CRC32); (f) }

Here's an explanation of what all these bits are for.

(a)
Your new test function should look like this: it takes no arguments, returns void, and is named test_whatever. It may be static or extern, but I recommend static. Functions with this signature are automatically discovered in the source code by the Cyrus unit test infrastructure, so all you have to do is write the function. Later, a CUnit test named "whatever" will be created automatically for your test_whatever function.
(b)
Here's a good place to define the test inputs and expected outputs.
(c)
Here's a good place for local variables you need during the test.
(d)
Here you set up the input conditions for the function under test.
(e)
Call the function under test with your known inputs, and capture the results in a local variable, here c.
(f)
Compare the actual result in c with the expected result in CRC32. The CU_ASSERT_EQUAL() macro checks that it's two arguments are equal (using an integer comparison), and if they're different it prints a message and records a failure. Note that unlike the libc assert() macro, control will continue even if the assert fails. The CUnit library provides a whole family of similar macros, see the online CUnit documentation for more details.

Now run make check and you'll see your test being built and run.

me@mybox> make check ... ../cunit/cunit.pl [...] --generate-wrapper crc32.c gcc -c [...] -g -O2 .cunit-crc32.c gcc [...] -o unit [...] .cunit-crc32.o ... ./unit -v CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: crc32 Test: map ... passed Test: iovec ... passed

4.3 Suite Init And Cleanup

Sometimes the behaviour of the functions under test depend on external influences such as environment variables, global variables, or the presence of certain files.

These kinds of functions need special treatment to ensure that their behaviour is locked down during the running of your tests. Otherwise, all sorts of strange behaviour may confuse the results of the tests. For example, a test might succeed the first time it's run in a given directory and fail the next time. Or a test might succeed when run by the author of the test but fail when run by another user.

CUnit provides a special arrangement which helps you in such cases: the suite initialisation and cleanup functions. These are two functions that you write and which live in the suite source. They are called from CUnit respectively before any of the tests in the suite is run, and after all tests from that suite are run.

Here's how to use them. The suite initialisation function should set up any global state that the functions under test rely on, in such a way that their state is predictable and always the same no matter who runs the test or when or how many times. Similarly the suite cleanup function should clean up any state which might possibly interfere with other test suites. Note that some suites will need an initialisation function but not necessarily a cleanup function.

Adding these functions is very easy: you just write functions of the appropriate signature (names, arguments and return type) and the Cyrus unit test infrastructure will automatically discover them and arrange for them to be called. The functions should look like (actual example taken from cunit/mboxname.c) this:

static enum enum_value old_config_virtdomains; static int init(void) { old_config_virtdomains = config_virtdomains; config_virtdomains = IMAP_ENUM_VIRTDOMAINS_ON; return 0; } static int cleanup(void) { config_virtdomains = old_config_virtdomains; return 0; }

The functions should return 0 on success, and non-zero on error. They must not call and CU_* functions or macros.