I've tried multiple sources for solutions to this problem. They all either require modifying the source code, and architecture specific exploit such as writing in a jmp instruction to detour the function, or using a macro and including the c file. The first one is extremely annoying to deal with, the second is usually not possible due to page protections, and the third introduces a lot of problems with linking multiple files containing different mocks and unit test for the same source file. Is there any better method of doing this?
You can user function pointer in your nominal code. You assign them at init with nominal implemetation in your application. In your unit test you can then assign the function pointer to the mock implmentation. Function pointer is a common practice used to implement interface in C.
Here is a gist of how that could be done:
typedef struct {
void (*method) ();
} interface;
void run(itf *interface)
{
itf->method();
}
void methodImpl()
{
printf("nominal code");
}
void methodMock()
{
printf("mock code");
}
void do_run()
{
interface itf;
itf.method = methodImpl;
run(&itf);
}
void test_run()
{
interface itf;
itf.method = methodMock;
run(&itf);
}
I'm currently testing out a few approaches on how to test and fuzz a C API. In the process thereof I found KLEE which runs the code symbolically, meaning that it tries to cover all branches that depend on some symbolic input and checks for all sorts of errors. I managed to get it to work and am now asking if my approach is good or if it has some major drawbacks or problems.
Let's suppose we have following simple but buggy API:
#include <assert.h>
static int g_state;
void setState(int state) {
g_state = state;
}
void run(void) {
if (g_state == 123) {
assert(0);
}
}
If the state is set to 123 and then run() is invoked the placed assertion fails.
For this I have written following symbolic test with KLEE:
#include "klee/klee.h"
#include "buggy_api.h"
int main(void) {
for (int i = 0; i < 2; ++i) { // sequentially call 2 APIs
int f_select = klee_choose(2); // what API to call
if (f_select == 0) {
int state = 0;
klee_make_symbolic(&state, sizeof(state), "state");
setState(state);
} else if (f_select == 1) {
run();
}
}
return 0;
}
When running with KLEE, the sequence of calls necessary to trigger the assertion is found almost immediately. I then tried to extend the API with a few other functions and hid the assertion behind a combination of states. KLEE again found the planted bug but naturally took a bit longer.
Is this how I can efficiently use KLEE for checking an API? Or is there documentation about a better approach?
In order to test an API with KLEE, you indeed need to write a driver that calls it. Yours works well, I'm not sure why you're using a for loop though? This smaller example should work:
#include "klee/klee.h"
#include "buggy_api.h"
int main(void) {
int state;
klee_make_symbolic(&state, sizeof(state), "state");
setState(state);
run();
return 0;
}
I have read that C does not support dynamic function calls. My program has an ever growing number of test cases implemented as separate functions like -
int testcase1(void);
int testcase2(void);
int testcase3(void);
Each time I add a new test case, I also have have to add the call to my main function like -
int main(int argc, char **argv){
assert(!testcase1());
assert(!testcase2());
assert(!testcase3());
}
I would prefer to call something like assert(!testcase*()) where * matches any string which resolves to a valid function name in my program.
Can you think of a more convenient solution?
If you all your testcases have same signature then you can use an array of function pointers:
void (*func[])() = { testcase1, testcase2 };
for (size_t i = 0; i < sizeof(func)/sizeof(func[0]); i++) {
assert(!func[i]());
}
The best solution is likely to write a few extra lines of code when you add new test cases - it really isn't a big issue. I would recommend something along the lines of the function pointer array, as suggested in another answer.
However, just to show that everything is possible in C if you throw ugly macros at the problem, here is a not recommended alternative:
#include <assert.h>
#include <stdbool.h>
#include <stdio.h>
#define TEST_CASES \ // list of "x macros"
X(testcase1) \
X(testcase2) \
X(testcase3)
#define X(func) bool func (void); // declare function prototypes
TEST_CASES
#undef X
bool (*const test_cases[])(void) = // array of read-only function pointers
{
#define X(func) &func, // point at each function
TEST_CASES
#undef X
};
int main (void)
{
for(size_t i=0; i<sizeof(test_cases)/sizeof(test_cases[0]); i++)
{
assert(test_cases[i]());
}
}
bool testcase1 (void) { puts(__func__); return true; }
bool testcase2 (void) { puts(__func__); return true; }
bool testcase3 (void) { puts(__func__); return false; }
Output:
testcase1
testcase2
testcase3
Assertion failed!
For each new test case, you would only have to write a function definition and then add it to the "x macro" list TEST_CASES. However, you need very good reasons to introduce ugly tricks like these in production code!
You can use function pointers. Read also about closures (but C99 or C11 don't have them) and callbacks.
Many operating systems provide dynamic loading. On POSIX operating systems (such as Linux or MacOSX) you can get a function pointer (actually an address) from its name in some library (or in the program executable) using dlopen & dlsym. Other operating systems may provide similar functionalities.
At last, you should consider having your testing main function be generated by some script (or some program emitting C code), using metaprogramming techniques. So you would write something which generates the C code of your testing main having a long sequence of assert, and improve your build procedure (e.g. your Makefile if using make) to run appropriately that specialized C code generator. Details are of course specific to your code. You might add some conventions (e.g. add some special comment to be parsed by your test generator, etc...).
I decided to follow #Nominal Animal and #Basile Starynkevitch's approach. In mymainprog.c, I added -
int runtests(void){
void *testh;
int (*testp)(void);
char *dlmsg;
int rc;
char funcname[8];
int testnum;
testh = dlopen("libsmtests.so", RTLD_LAZY);
if (!testh){
printf("%s\n", dlerror());
return 1;
}
dlerror();
for (testnum =1; testnum < 1000; testnum++){
sprintf(funcname,"testcase%d", testnum);
*(void **) (&testp) = dlsym(testh, funcname);
dlmsg = dlerror();
if (dlmsg == NULL) {
rc = (*testp)();
printf("%s called, rc=%d\n", funcname, rc);
}
}
dlclose(testh);
return 0;
}
I add my testcases to a separate file (testcases.c) like this -
int testcase1(void){
return [some testcase expression]
}
int testcase2(void){
return [another testcase expression]
}
and then compile it as a shared library with position-independant code (-fPIC) to libsmtests.so. The advantage is slightly less typing since I don't need to code a call to testNNNN() after adding the implementation of a new functionint testcaseNNN(void) to testcases.c
I'm writing a unit-test to check some API calls. I am using check to test. My module is build with CMake (idk if it matters).
My test calls a function (which I need to test) and this function makes a call to another binary.
Simplified version of it looks like this.
/* unitTest.c */
#include "libraryAPI.h"
void letsMakeACall(void)
{
ck_assert_eq(foo("water"), 0);
}
-- Module I am working on---
/*libraryAPI.c*/
#include "legacyLib.h"
void foo(const char *drink )
{
if (checkDrink(drink)!=0)
{
return 1;
}else
{
return 0;
}
}
----LEGACY BINARY---
/*legacyLib.c*/
static const char* expected = "water";
void checkDrink(const char *drink)
{
if(drink == expected)
{
/*There are also a dozen functions being called which depend on legacy module initialisation*/
return 0;
}else{
return 1;
}
}
I'd like to mock response from legacyLib, because otherwise it call dozens of functions and breaks. My initial idea was to add some ifdef conditions when tests are being run, but it is against guidelines.
Because it is basically a call interception I don't know what it a best(or working) solution. What can I use to solve it?
I am also unsure how to solve this generally, I have posted a similar question, but in some cases you can do the following (presuming you are testing individual functions):
Include the .c file instead of the header .h, but after you "rename" your mocked function using a define directive:
#define checkDrink checkDrink_mocked
// preprocessor will now replace all occurrences of "checkDrink"
// with "checkDrink_mocked"
int checkDrink_mocked(const char *drink);
#include "legacyLib.c"
#undef checkDrink
Implement the renamed function:
int checkDrink_mocked(const char *drink)
{
return 15;
}
I've started to dig into the GLib documentation and discovered that it also offers a unit testing framework.
But how could you do unit tests in a procedural language? Or does it require to program OO in C?
Unit testing only requires "cut-planes" or boundaries at which testing can be done. It is quite straightforward to test C functions which do not call other functions, or which call only other functions that are also tested. Some examples of this are functions which perform calculations or logic operations, and are functional in nature. Functional in the sense that the same input always results in the same output. Testing these functions can have a huge benefit, even though it is a small part of what is normally thought of as unit testing.
More sophisticated testing, such as the use of mocks or stubs is also possible, but it is not nearly as easy as it is in more dynamic languages, or even just object oriented languages such as C++. One way to approach this is to use #defines. One example of this is this article, Unit testing OpenGL applications, which shows how to mock out OpenGL calls. This allows you to test that valid sequences of OpenGL calls are made.
Another option is to take advantage of weak symbols. For example, all MPI API functions are weak symbols, so if you define the same symbol in your own application, your implementation overrides the weak implementation in the library. If the symbols in the library weren't weak, you would get duplicate symbol errors at link time. You can then implement what is effectively a mock of the entire MPI C API, which allows you to ensure that calls are matched up properly and that there aren't any extra calls that could cause deadlocks. It is also possible to load the library's weak symbols using dlopen() and dlsym(), and pass the call on if necessary. MPI actually provides the PMPI symbols, which are strong, so it is not necessary to use dlopen() and friends.
You can realize many of the benefits of unit testing for C. It is slightly harder, and it may not be possible to get the same level of coverage you might expect from something written in Ruby or Java, but it's definitely worth doing.
At the most basic level, unit tests are just bits of code that execute other bits of code and tell you if they worked as expected.
You could simply make a new console app, with a main() function, that executed a series of test functions. Each test would call a function in your app and return a 0 for success or another value for failure.
I'd give you some example code, but I'm really rusty with C. I'm sure there are some frameworks out there that would make this a little easier too.
You can use libtap which provides a number of functions which can provide diagnostics when a test fails. An example of its use:
#include <mystuff.h>
#include <tap.h>
int main () {
plan(3);
ok(foo(), "foo returns 1");
is(bar(), "bar", "bar returns the string bar");
cmp_ok(baz(), ">", foo(), "baz returns a higher number than foo");
done_testing;
}
Its similar to tap libraries in other languages.
Here's an example of how you would implement multiple tests in a single test program for a given function that might call a library function.
Suppose we want to test the following module:
#include <stdlib.h>
int my_div(int x, int y)
{
if (y==0) exit(2);
return x/y;
}
We then create the following test program:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <setjmp.h>
// redefine assert to set a boolean flag
#ifdef assert
#undef assert
#endif
#define assert(x) (rslt = rslt && (x))
// the function to test
int my_div(int x, int y);
// main result return code used by redefined assert
static int rslt;
// variables controling stub functions
static int expected_code;
static int should_exit;
static jmp_buf jump_env;
// test suite main variables
static int done;
static int num_tests;
static int tests_passed;
// utility function
void TestStart(char *name)
{
num_tests++;
rslt = 1;
printf("-- Testing %s ... ",name);
}
// utility function
void TestEnd()
{
if (rslt) tests_passed++;
printf("%s\n", rslt ? "success" : "fail");
}
// stub function
void exit(int code)
{
if (!done)
{
assert(should_exit==1);
assert(expected_code==code);
longjmp(jump_env, 1);
}
else
{
_exit(code);
}
}
// test case
void test_normal()
{
int jmp_rval;
int r;
TestStart("test_normal");
should_exit = 0;
if (!(jmp_rval=setjmp(jump_env)))
{
r = my_div(12,3);
}
assert(jmp_rval==0);
assert(r==4);
TestEnd();
}
// test case
void test_div0()
{
int jmp_rval;
int r;
TestStart("test_div0");
should_exit = 1;
expected_code = 2;
if (!(jmp_rval=setjmp(jump_env)))
{
r = my_div(2,0);
}
assert(jmp_rval==1);
TestEnd();
}
int main()
{
num_tests = 0;
tests_passed = 0;
done = 0;
test_normal();
test_div0();
printf("Total tests passed: %d\n", tests_passed);
done = 1;
return !(tests_passed == num_tests);
}
By redefining assert to update a boolean variable, you can continue on if an assertion fails and run multiple tests, keeping track of how many succeeded and how many failed.
At the start of each test, set rslt (the variables used by the assert macro) to 1, and set any variables that control your stub functions. If one of your stubs gets called more than once, you can set up arrays of control variables so that the stubs can check for different conditions on different calls.
Since many library functions are weak symbols, they can be redefined in your test program so that they get called instead. Prior to calling the function to test, you can set a number of state variables to control the behavior of the stub function and check conditions on the function parameters.
In cases where you can't redefine like that, give the stub function a different name and redefine the symbol in the code to test. For example, if you want to stub fopen but find that it isn't a weak symbol, define your stub as my_fopen and compile the file to test with -Dfopen=my_fopen.
In this particular case, the function to be tested may call exit. This is tricky, since exit can't return to the function being tested. This is one of the rare times when it makes sense to use setjmp and longjmp. You use setjmp before entering the function to test, then in the stubbed exit you call longjmp to return directly back to your test case.
Also note that the redefined exit has a special variable that it checks to see if you actually want to exit the program and calls _exit to do so. If you don't do this, your test program may not quit cleanly.
This test suite also counts the number of attempted and failed tests and returns 0 if all tests passed and 1 otherwise. That way, make can check for test failures and act accordingly.
The above test code will output the following:
-- Testing test_normal ... success
-- Testing test_div0 ... success
Total tests passed: 2
And the return code will be 0.
There is nothing intrinsically object-oriented about testing small pieces of code in isolation. In procedural languages you test functions and collections thereof.
If you are desperate, and you'd have to be desperate, I banged together a little C preprocessor and gmake based framework. It started as a toy, and never really grew up, but I have used it to develop and test a couple of medium sized (10,000+ line) projects.
Dave's Unit Test is minimally intrusive yet it can do some tests I had originally thought would not be possible for a preprocessor based framework (you can demand that a certain stretch of code throw a segmentation fault under certain conditions, and it will test it for you).
It is also an example of why making heavy use of the preprocessor is hard to do safely.
The simplest way of doing a unit test is to build a simple driver code that gets linked with the other code, and call each function in each case...and assert the values of the results of the functions and build up bit by bit...that's how I do it anyway
int main(int argc, char **argv){
// call some function
int x = foo();
assert(x > 1);
// and so on....
}
Hope this helps.
With C it must go further than simply implementing a framework on top of existing code.
One thing I've always done is make a testing module (with a main) that you can run little tests from to test your code. This allows you to do very small increments between code and test cycles.
The bigger concern is writing your code to be testable. Focus on small, independent functions that do not rely on shared variables or state. Try writing in a "Functional" manner (without state), this will be easier to test. If you have a dependency that can't always be there or is slow (like a database), you may have to write an entire "mock" layer that can be substituted for your database during tests.
The principle unit testing goals still apply: ensure the code under test always resets to a given state, test constantly, etc...
When I wrote code in C (back before Windows) I had a batch file that would bring up an editor, then when I was done editing and exited, it would compile, link, execute tests and then bring up the editor with the build results, test results and the code in different windows. After my break (a minute to several hours depending on what was being compiled) I could just review results and go straight back to editing. I'm sure this process could be improved upon these days :)
I use assert. It's not really a framework though.
You can write a simple minimalistic test framework yourself:
// test_framework.h
#define BEGIN_TESTING int main(int argc, char **argv) {
#define END_TESTING return 0;}
#define TEST(TEST_NAME) if (run_test(TEST_NAME, argc, argv))
int run_test(const char* test_name, int argc, char **argv) {
// we run every test by default
if (argc == 1) { return 1; }
// else we run only the test specified as a command line argument
for (int i = 1; i < argc; i++) {
if (!strcmp(test_name, argv[i])) { return 0; }
}
return 0;
}
Now in the actual test file do this:
#include test_framework.h
BEGIN_TESTING
TEST("MyPassingTest") {
assert(1 == 1);
}
TEST("MyFailingTest") {
assert(1 == 2);
}
END_TESTING
If you want to run all tests, execute ./binary without command line arguments, if you want to run just a particular test, execute ./binary MyFailingTest