Recently, a set of regression unit tests has been added to Cyrus. This document explains the purpose implementation of those unit tests, and gives an example of how to add more unit tests (because there are never enough unit tests!).
The definition on Wikipedia sheds some light:
...unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application.In other words, unit testing is about verifying that small pieces of code, like individual functions, modules, or classes, work in isolation. It is not about testing the system as a whole.
The tests implemenented here are also regression tests, which in Wikipedia's words means:
Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.
In other words, the tests are designed to be easy to run and to work out fully automatically whether they have passed or failed, so that they can be run usefully by people who didn't write them.
This section takes you through the process of running Cyrus' unit tests.
Cyrus' unit tests are all located in a new directory, cyrus-imapd/cunit/. They're written in C, like the remainder of Cyrus, and use the CUnit library from SourceForge, with some home grown wrappers and other improvements to make our lives easier.
Your first step is step is to ensure that the CUnit library (including the headers) is installed. Some modern operating systems already have CUnit, for example on Ubuntu you can just do:
me@ubuntu> sudo apt-get install libcunit1-dev
Alternately, you can download the CUnit source, build it and install it. It's not a complicated or difficult library, this shouldn't take long. When you've done, install it in /usr/include and /usr/lib.
Because of the dependency on the CUnit library, the tests are disabled by default; this means you need enable them with an option to the configure script:
me@mybox> ./configure --enable-unit-tests ... checking for CU_initialize_registry in -lcunit... yes checking CUnit/CUnit.h usability... yes checking CUnit/CUnit.h presence... yes checking for CUnit/CUnit.h... yes ...
First you need to build Cyrus itself, using the traditional all: target.
me@mybox> make all ...
Then, use the new check: target to build and run the unit tests.
me@mybox> make check ... ### Making check in /home/gnb/software/cyrus/imapd/lib (a) make[1]: Entering directory `/home/gnb/software/cyrus/imapd/lib' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/home/gnb/software/cyrus/imapd/lib' ### Done with /home/gnb/software/cyrus/imapd/lib ... make[1]: Entering directory `/home/gnb/software/cyrus/imapd/cunit' (b) ... ../cunit/cunit.pl [...] --generate-wrapper mboxname.c (c) gcc -c [...] -g -O2 .cunit-mboxname.c ... gcc [...] -o unit unit.o [...] .cunit-mboxname.o [...] \ (d) ../imap/libimap.a ../lib/libcyrus.a [...] ./unit -v (e) CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: mboxname (f) Test: to_parts ... passed Test: to_userid ... passed Test: same_userid ... passed Test: same_userid_domain ... passed ... --Run Summary: Type Total Ran Passed Failed (g) suites 9 9 n/a 0 tests 51 51 50 1 asserts 474 474 473 1 make[1]: Leaving directory `/home/gnb/software/cyrus/imapd/cunit'
Let's take a closer look at what's happening here.
Some failure modes are subtle, and cannot be detected in the C code itself; this is where the Valgrind program comes in very handy. It detects buffer overruns and memory leaks and various other kinds of subtle errors.
To run the unit tests with Valgrind, use the new valgrind: target.
me@mybox> make valgrind ... valgrind --tool=memcheck --leak-check=full ./unit -v (a) ==2999== Memcheck, a memory error detector ==2999== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==2999== Using Valgrind-3.6.0.SVN-Debian and LibVEX; [...] ==2999== Command: ./unit -v ==2999== ... --Run Summary: Type Total Ran Passed Failed (b) suites 9 9 n/a 0 tests 51 51 50 1 asserts 474 474 473 1 ... ==2999== HEAP SUMMARY: (c) ==2999== in use at exit: 4,489 bytes in 134 blocks ==2999== total heap usage: 715 allocs, 581 frees, 352,763 bytes allocated ==2999== ==2999== 4 bytes in 1 blocks are definitely lost in loss record 3 of 50 ==2999== at 0x4C2815C: malloc (vg_replace_malloc.c:236) ==2999== by 0x44A0CA: xmalloc (xmalloc.c:57) ==2999== by 0x4399D8: strconcat (util.c:631) ==2999== by 0x40C059: test_uncast_null (strconcat.c:51) ==2999== by 0x61B32A9: ??? (in /usr/lib/libcunit.so.1.0.1) ==2999== by 0x61B36ED: ??? (in /usr/lib/libcunit.so.1.0.1) ==2999== by 0x61B3827: CU_run_all_tests (in /usr/lib/libcunit.so.1.0.1) ==2999== by 0x4066CC: run_tests (unit.c:144) ==2999== by 0x406806: main (unit.c:283) ==2999== ...
Here's an explanation of what's happening in the example.
I'd just like to say that I love Valgrind and I think it's immensely useful. I would have made running the tests under Valgrind the only option for the check: target, except that Valgrind is not available on all of Cyrus' supported platforms.
Adding your own tests is quite simple. Here's how.
The unit test code in Cyrus is contained in a set of C source files in the cunit directory. If you look closely, you will see that each of those C source files maps to a "Suite" in CUnit parlance. For example, cunit/glob.c is listed as the Suite "glob" in CUnit's runtime output.
Typically, each Suite tests a single module or a related set of functions; for example, cunit/glob.c contains tests for the glob module in lib/glob.c.
So, if you want to add a new test for a module which already has some existing tests, the sensible thing to do is to add a new test to the existing suite. Otherwise, you'll need to add a new Suite.
Each Suite is a single C source file in the cunit/ directory. Your first step is to create a new C source file. For this example, you'll create a new Suite to test the CRC32 routines which live in lib/crc32.c.
me@mybox> vi cunit/crc32.c ...
The file should contain something like this.
/* Unit test for lib/crc32.c */ #include "cunit/cunit.h" (a) #include "crc32.h" (b) static void test_map(void) (c) { static const char TEXT[] = "lorem ipsum"; (d) static uint32_t CRC32 = 0x0; uint32_t c; (e) c = crc32_map(TEXT, sizeof(TEXT)-1); (f) CU_ASSERT_EQUAL(c, CRC32); (g) }
Here's an explanation of what all these bits are for.
Now you need to tell the Cyrus makefiles about your new Suite.
me@mybox> vi cunit/Makefile.in ...
You need to add the filename of your new test to the definition of the TESTSOURCES variable.
PROGS = unit TESTSOURCES = times.c glob.c md5.c parseaddr.c message.c \ strconcat.c conversations.c msgid.c mboxname.c \ crc32.c TESTLIBS = @top_srcdir@/imap/mutex_fake.o @top_srcdir@/imap/libimap.a \ @top_srcdir@/lib/libcyrus.a @top_srcdir@/lib/libcyrus_min.a
Now you need to use autoconf to build cunit/Makefile from cunit/Makefile.in. The easiest way to do this is to run the config.status shell script which is one of the files that autoconf generates. Alternately, you could re-run the configure script (but that would be a lot slower).
me@mybox> ./config.status ... config.status: creating cunit/Makefile ...
Now, you finally get to build and run your new test code. Run make check and you should see your new code being built and run.
me@mybox> make check ... ../cunit/cunit.pl [...] --add-sources [...] crc32.c ... ../cunit/cunit.pl [...] --generate-wrapper crc32.c gcc -c [...] -g -O2 .cunit-crc32.c gcc [...] -o unit [...] .cunit-crc32.o ... ./unit -v CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: crc32 Test: map ... FAILED 1. crc32.c:12 - CU_ASSERT_EQUAL(c=1926722702,CRC32=0)
Note how the test failure told us which in source file and at what line number the failure occurred, and what the actual and expected values were. Let's go and fix that up now.
static const char TEXT[] = "lorem ipsum"; static uint32_t CRC32 = 0x72d7748e;
Re-run make check and you'll see your test being rebuilt and rerun, and this time passing.
me@mybox> make check ... ../cunit/cunit.pl [...] --generate-wrapper crc32.c gcc -c [...] -g -O2 .cunit-crc32.c gcc [...] -o unit [...] .cunit-crc32.o ... ./unit -v CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: crc32 Test: map ... passed
Adding a new test to an existing test is easy: all you have to do is add a new function to an existing C source file in the cunit/ directory. As an example, let's add a test for the crc_iovec() function.
me@mybox> vi cunit/crc32.c ...
static void test_iovec(void) (a) { static const char TEXT1[] = "lorem"; (b) static const char TEXT2[] = " ipsum"; static uint32_t CRC32 = 0x72d7748e; uint32_t c; (c) struct iovec iov[2]; memset(iov, 0, sizeof(iov)); (d) iov[0].iov_base = TEXT1; iov[0].iov_len = sizeof(TEXT1)-1; iov[1].iov_base = TEXT2; iov[1].iov_len = sizeof(TEXT2)-1; c = crc32_iovec(iov, 2); (e) CU_ASSERT_EQUAL(c, CRC32); (f) }
Here's an explanation of what all these bits are for.
Now run make check and you'll see your test being built and run.
me@mybox> make check ... ../cunit/cunit.pl [...] --generate-wrapper crc32.c gcc -c [...] -g -O2 .cunit-crc32.c gcc [...] -o unit [...] .cunit-crc32.o ... ./unit -v CUnit - A Unit testing framework for C - Version 2.1-0 http://cunit.sourceforge.net/ ... Suite: crc32 Test: map ... passed Test: iovec ... passed
Sometimes the behaviour of the functions under test depend on external influences such as environment variables, global variables, or the presence of certain files.
These kinds of functions need special treatment to ensure that their behaviour is locked down during the running of your tests. Otherwise, all sorts of strange behaviour may confuse the results of the tests. For example, a test might succeed the first time it's run in a given directory and fail the next time. Or a test might succeed when run by the author of the test but fail when run by another user.
CUnit provides a special arrangement which helps you in such cases: the suite initialisation and cleanup functions. These are two functions that you write and which live in the suite source. They are called from CUnit respectively before any of the tests in the suite is run, and after all tests from that suite are run.
Here's how to use them. The suite initialisation function should set up any global state that the functions under test rely on, in such a way that their state is predictable and always the same no matter who runs the test or when or how many times. Similarly the suite cleanup function should clean up any state which might possibly interfere with other test suites. Note that some suites will need an initialisation function but not necessarily a cleanup function.
Adding these functions is very easy: you just write functions of the appropriate signature (names, arguments and return type) and the Cyrus unit test infrastructure will automatically discover them and arrange for them to be called. The functions should look like (actual example taken from cunit/mboxname.c) this:
static enum enum_value old_config_virtdomains; static int init(void) { old_config_virtdomains = config_virtdomains; config_virtdomains = IMAP_ENUM_VIRTDOMAINS_ON; return 0; } static int cleanup(void) { config_virtdomains = old_config_virtdomains; return 0; }
The functions should return 0 on success, and non-zero on error. They must not call and CU_* functions or macros.