Snapit: Snapshot Testing for C

Snapshot testing is a fantastic way to make tests easier to read and maintain, but it's more commonly used with high level languages. I've made snapit to help me do it with C, and maybe you can use it too.

If you haven't heard of snapshot testing then it's a method of testing where, rather than hand-writing a set of assertions on some output value, you compare your result with a ground truth: the snapshot. Usually a test framework or CLI tool handles saving the snapshot somewhere for you, and you accept or reject it when things change.

The particular kind I like is where the snapshot is stored in the code itself. This lends itself to an exploratory workflow that Ian Henry documented excellently on his blog - you write your code and then use the snapshot test to see what it produces, fixing bugs as you go. The inputs that highlighted bugs in your program make for good tests.

Let's look at an example. I drop snapit.h into an empty directory and then have this in test.c:

#include <stdio.h> #include <stdint.h> #define SNAPIT_IMPLEMENTATION #include "snapit.h" static bool to_hex(char *dst, int length, uint32_t v) { if (length < 9) return false; static const char *hex_chars = "0123456789abcdef"; for (int i = 7; i >= 0; --i) { const int half_byte = (v >> i*4) & 0xf; dst[i] = hex_chars[half_byte]; } dst[8] = 0; return true; } void run_test(void) { char tmp[16]; to_hex(tmp, sizeof(tmp), 0x0); snapit(tmp, ""); to_hex(tmp, sizeof(tmp), 0x1234); snapit(tmp, ""); to_hex(tmp, sizeof(tmp), 0xffff); snapit(tmp, ""); to_hex(tmp, sizeof(tmp), 0x1a2b3c); snapit(tmp, ""); } int main(void) { snapit_init(); run_test(); return 0; }

By default the first failing test will print a message and abort the program:

> ./test test.c:28: FAIL Expected: Got: 00000000 fish: Job 2, './test' terminated by signal SIGABRT (Abort)

The magic is done through a program shipped with snapit and an environment variable. With these we can automatically update the failing tests to pass:

> SNAPIT_MODE_EDIT=1 ./test | ./snapit_ed found line 28 found line 31 found line 34 found line 37 > git diff diff --git a/test.c b/test.c index 75b3184..ac9b3f8 100644 --- a/test.c +++ b/test.c @@ -25,16 +25,16 @@ void run_test(void) char tmp[16]; to_hex(tmp, sizeof(tmp), 0x0); - snapit(tmp, ""); + snapit(tmp, "00000000"); to_hex(tmp, sizeof(tmp), 0x1234); - snapit(tmp, ""); + snapit(tmp, "43210000"); to_hex(tmp, sizeof(tmp), 0xffff); - snapit(tmp, ""); + snapit(tmp, "ffff0000"); to_hex(tmp, sizeof(tmp), 0x1a2b3c); - snapit(tmp, ""); + snapit(tmp, "c3b2a100"); } int main(void)

Even though the first test aborted when we ran it normally you can see all the failing tests were updated here.

The first test looks fine here, but the others look to be in reverse. The position we are writing to in dest is wrong - let's fix that:

- dst[i] = hex_chars[half_byte]; + dst[7 - i] = hex_chars[half_byte];

and if we rerun snapit_ed we see the correct expected values:

> SNAPIT_MODE_EDIT=1 ./test | ./snapit_ed found line 31 found line 34 found line 37 mjh@jester ~/D/s/snapit-test (master)> git diff diff --git a/test.c b/test.c index ac9b3f8..15f63f4 100644 --- a/test.c +++ b/test.c @@ -13,7 +13,7 @@ static bool to_hex(char *dst, int length, uint32_t v) for (int i = 7; i >= 0; --i) { const int half_byte = (v >> i*4) & 0xf; - dst[i] = hex_chars[half_byte]; + dst[7 - i] = hex_chars[half_byte]; } dst[8] = 0; @@ -28,13 +28,13 @@ void run_test(void) snapit(tmp, "00000000"); to_hex(tmp, sizeof(tmp), 0x1234); - snapit(tmp, "43210000"); + snapit(tmp, "00001234"); to_hex(tmp, sizeof(tmp), 0xffff); - snapit(tmp, "ffff0000"); + snapit(tmp, "0000ffff"); to_hex(tmp, sizeof(tmp), 0x1a2b3c); - snapit(tmp, "c3b2a100"); + snapit(tmp, "001a2b3c"); } int main(void)

The actual testing part is pretty simple - we do a strcmp and run a handler (see the Customisation section below) based on whether the strings were equal or not.

The interesting bit is when we are in edit mode. Let's run the tests when we were expecting empty strings, but not pipe it into snapit_ed:

> SNAPIT_MODE_EDIT=1 ./test 6:test.c,2:28,8:00000000,6:test.c,2:31,8:43210000,6:test.c,2:34,8:ffff0000,6:test.c,2:37,8:c3b2a100,

If you squint you can see the filename, a line number of the test and the value we actually got. The other characters are the encoding - netstrings. This is a simple format that encodes the length of the string as a decimal number, then has a colon, then the string and finally a comma (i.e. <len>:<'len' bytes>,). Starting with the number of bytes to expect allows parsing libraries to allocate the number of bytes needed up front - neat.

snapit_ed parses three of these netstrings into a struct for each test:

typedef struct { char *filename; int line; char *got; } failure; static netstring_error parse_failure(netstring_parser *p, failure *f) { int err = netstring_parser_parse(p, &f->filename); if (err) return err; char *line = 0; err = netstring_parser_parse(p, &line); if (err) goto cleanup_filename; f->line = atoi(line); free(line); err = netstring_parser_parse(p, &f->got); if (err) goto cleanup_filename; return 0; cleanup_filename: free(f->filename); return err; } int main() { failure failures[MAX_FAILURES] = {0}; netstring_parser p = (netstring_parser){.fd = STDIN_FILENO, 0}; int i = 0; for (; i < MAX_FAILURES; i++) { netstring_error err = parse_failure(&p, &failures[i]); if (err == NS_ERR_EOF) break; if (err) return 1; } if (i == 0) return 0; }

We then need to edit the file. We have the line number the test was on, so we know where we need to start replacing. We start by reading the file into memory and creating a new one to write into. We then call rewrite_failures to fix things up:

static bool rewrite_failures( failure *failures, int count, const char *contents, FILE *f) { if (count == 0) return true; stb_lexer lexer; char *string_store = malloc(1024); const char *s = contents; int line = 1; for (int i = 0; i < count; i++) { while (line < failures[i].line) { if (*s == 0) return false; fputc(*s, f); if (*s++ != '\n') continue; line++; } fprintf(stderr, "found line %d\n", failures[i].line); while (isspace(*s)) fputc(*s++, f); stb_c_lexer_init(&lexer, s, 0, string_store, 1024); if (!fixup_failure(&lexer, failures[i].got, f)) return false; s = lexer.parse_point; } fprintf(f, s); free(string_store); return true; }

Writing fixup_failure is a little tricky as we want to keep the expression we are testing against and only update the expected value. One way to do this would be to have snapit be a macro that outputs its test expression as a netstring.

I wanted something more flexible so I decided to parse the line the test is on. To do this I have used the stb_c_lexer.h header-only library from Sean Barrett which provides a C lexer. I then have a very simple parser:

#define TOR if (!x) return false static bool lexer_expect_token(stb_lexer *lexer, int token) { if (!stb_c_lexer_get_token(lexer)) { fprintf(stderr, "could not get token\n"); return false; } if (lexer->token == token) return true; fprintf( stderr, "expected %s, got %s\n", token_string(token), token_string(lexer->token)); return false; } static bool fixup_failure(stb_lexer *lexer, const char *prefix, const char *got, FILE *f) { TOR(lexer_expect_token(lexer, CLEX_id)); fprintf(f, lexer->string); TOR(lexer_expect_token(lexer, '(')); fprintf(f, "("); TOR(stb_c_lexer_get_token(lexer)); if (lexer->token != CLEX_id && lexer->token != CLEX_dqstring) return false; if (lexer->token == CLEX_id) { fprintf(f, "%s", lexer->string); } else { fprintf(f, "\"%s\"", lexer->string); } TOR(lexer_expect_token(lexer, ',')); TOR(stb_c_lexer_get_token(lexer)); fprintf(f, ", \"%s\"", got); while (true) { TOR(stb_c_lexer_get_token(lexer)); if (lexer->token == ')') break; } TOR(lexer_expect_token(lexer, ';')); fprintf(f, ");"); return true; }

This has only been tested on Linux and is reliant on POSIX APIs. Maybe it would work on Windows.

The parser is restrictive currently, e.g. it wouldn't handle snapit("FOO", ("FOO")) because of the extra parens. Still, it is robust enough for my uses and the problems can slowly be patched as I run into them in real usage.

The code isn't super robust or even performant. Lots of error checks are missing and using fputc for almost every byte of the resultant file is Not Good. I don't really care though - I wrote this library to unblock a side project and it works well enough for that. When spending my working days making code maintainable and robust sometimes it's nice to do something that is good enough!

The full code is on sourcehut. The current version has support for a dry-run mode that won't overwrite your current file, and it also allows you to specify a macro to wrap your string literals in. This is useful if you've defined your own string type.