I'm writing a unit-test to check some API calls. I am using check to test. My module is build with CMake (idk if it matters).
My test calls a function (which I need to test) and this function makes a call to another binary.
Simplified version of it looks like this.
/* unitTest.c */
#include "libraryAPI.h"
void letsMakeACall(void)
{
ck_assert_eq(foo("water"), 0);
}
-- Module I am working on---
/*libraryAPI.c*/
#include "legacyLib.h"
void foo(const char *drink )
{
if (checkDrink(drink)!=0)
{
return 1;
}else
{
return 0;
}
}
----LEGACY BINARY---
/*legacyLib.c*/
static const char* expected = "water";
void checkDrink(const char *drink)
{
if(drink == expected)
{
/*There are also a dozen functions being called which depend on legacy module initialisation*/
return 0;
}else{
return 1;
}
}
I'd like to mock response from legacyLib, because otherwise it call dozens of functions and breaks. My initial idea was to add some ifdef conditions when tests are being run, but it is against guidelines.
Because it is basically a call interception I don't know what it a best(or working) solution. What can I use to solve it?
I am also unsure how to solve this generally, I have posted a similar question, but in some cases you can do the following (presuming you are testing individual functions):
Include the .c file instead of the header .h, but after you "rename" your mocked function using a define directive:
#define checkDrink checkDrink_mocked
// preprocessor will now replace all occurrences of "checkDrink"
// with "checkDrink_mocked"
int checkDrink_mocked(const char *drink);
#include "legacyLib.c"
#undef checkDrink
Implement the renamed function:
int checkDrink_mocked(const char *drink)
{
return 15;
}
Related
I'm currently testing out a few approaches on how to test and fuzz a C API. In the process thereof I found KLEE which runs the code symbolically, meaning that it tries to cover all branches that depend on some symbolic input and checks for all sorts of errors. I managed to get it to work and am now asking if my approach is good or if it has some major drawbacks or problems.
Let's suppose we have following simple but buggy API:
#include <assert.h>
static int g_state;
void setState(int state) {
g_state = state;
}
void run(void) {
if (g_state == 123) {
assert(0);
}
}
If the state is set to 123 and then run() is invoked the placed assertion fails.
For this I have written following symbolic test with KLEE:
#include "klee/klee.h"
#include "buggy_api.h"
int main(void) {
for (int i = 0; i < 2; ++i) { // sequentially call 2 APIs
int f_select = klee_choose(2); // what API to call
if (f_select == 0) {
int state = 0;
klee_make_symbolic(&state, sizeof(state), "state");
setState(state);
} else if (f_select == 1) {
run();
}
}
return 0;
}
When running with KLEE, the sequence of calls necessary to trigger the assertion is found almost immediately. I then tried to extend the API with a few other functions and hid the assertion behind a combination of states. KLEE again found the planted bug but naturally took a bit longer.
Is this how I can efficiently use KLEE for checking an API? Or is there documentation about a better approach?
In order to test an API with KLEE, you indeed need to write a driver that calls it. Yours works well, I'm not sure why you're using a for loop though? This smaller example should work:
#include "klee/klee.h"
#include "buggy_api.h"
int main(void) {
int state;
klee_make_symbolic(&state, sizeof(state), "state");
setState(state);
run();
return 0;
}
I have read that C does not support dynamic function calls. My program has an ever growing number of test cases implemented as separate functions like -
int testcase1(void);
int testcase2(void);
int testcase3(void);
Each time I add a new test case, I also have have to add the call to my main function like -
int main(int argc, char **argv){
assert(!testcase1());
assert(!testcase2());
assert(!testcase3());
}
I would prefer to call something like assert(!testcase*()) where * matches any string which resolves to a valid function name in my program.
Can you think of a more convenient solution?
If you all your testcases have same signature then you can use an array of function pointers:
void (*func[])() = { testcase1, testcase2 };
for (size_t i = 0; i < sizeof(func)/sizeof(func[0]); i++) {
assert(!func[i]());
}
The best solution is likely to write a few extra lines of code when you add new test cases - it really isn't a big issue. I would recommend something along the lines of the function pointer array, as suggested in another answer.
However, just to show that everything is possible in C if you throw ugly macros at the problem, here is a not recommended alternative:
#include <assert.h>
#include <stdbool.h>
#include <stdio.h>
#define TEST_CASES \ // list of "x macros"
X(testcase1) \
X(testcase2) \
X(testcase3)
#define X(func) bool func (void); // declare function prototypes
TEST_CASES
#undef X
bool (*const test_cases[])(void) = // array of read-only function pointers
{
#define X(func) &func, // point at each function
TEST_CASES
#undef X
};
int main (void)
{
for(size_t i=0; i<sizeof(test_cases)/sizeof(test_cases[0]); i++)
{
assert(test_cases[i]());
}
}
bool testcase1 (void) { puts(__func__); return true; }
bool testcase2 (void) { puts(__func__); return true; }
bool testcase3 (void) { puts(__func__); return false; }
Output:
testcase1
testcase2
testcase3
Assertion failed!
For each new test case, you would only have to write a function definition and then add it to the "x macro" list TEST_CASES. However, you need very good reasons to introduce ugly tricks like these in production code!
You can use function pointers. Read also about closures (but C99 or C11 don't have them) and callbacks.
Many operating systems provide dynamic loading. On POSIX operating systems (such as Linux or MacOSX) you can get a function pointer (actually an address) from its name in some library (or in the program executable) using dlopen & dlsym. Other operating systems may provide similar functionalities.
At last, you should consider having your testing main function be generated by some script (or some program emitting C code), using metaprogramming techniques. So you would write something which generates the C code of your testing main having a long sequence of assert, and improve your build procedure (e.g. your Makefile if using make) to run appropriately that specialized C code generator. Details are of course specific to your code. You might add some conventions (e.g. add some special comment to be parsed by your test generator, etc...).
I decided to follow #Nominal Animal and #Basile Starynkevitch's approach. In mymainprog.c, I added -
int runtests(void){
void *testh;
int (*testp)(void);
char *dlmsg;
int rc;
char funcname[8];
int testnum;
testh = dlopen("libsmtests.so", RTLD_LAZY);
if (!testh){
printf("%s\n", dlerror());
return 1;
}
dlerror();
for (testnum =1; testnum < 1000; testnum++){
sprintf(funcname,"testcase%d", testnum);
*(void **) (&testp) = dlsym(testh, funcname);
dlmsg = dlerror();
if (dlmsg == NULL) {
rc = (*testp)();
printf("%s called, rc=%d\n", funcname, rc);
}
}
dlclose(testh);
return 0;
}
I add my testcases to a separate file (testcases.c) like this -
int testcase1(void){
return [some testcase expression]
}
int testcase2(void){
return [another testcase expression]
}
and then compile it as a shared library with position-independant code (-fPIC) to libsmtests.so. The advantage is slightly less typing since I don't need to code a call to testNNNN() after adding the implementation of a new functionint testcaseNNN(void) to testcases.c
I'm doing Unit Testing of embedded C code without running on target hardware.Here is one part of the code:
uint8 tempReadback = 0;
write_to_i2c( msg->addres, msg->value );
tempReadback = read_from_i2c( msg->addres);
if( tempReadback == msg->value )
{
somethingA;
}
else
{
somethingB;
}
The function write_to_i2c() writes a value to a specific register. The function read_from_i2c() reads back the value from the register. Further, I'm using the variable tempReadback to compare if the read back value is the same as the written one. So far OK, and this works on the target hardware. Now I'm doing the Uni Tests without running the code on target Hardware (Software in the Loop). This means, the expression tempReadback == msg->value will never be true (tempReadback is 0) and I will run each time in the statement somethingB. Is there any way to fake the register read back? I'm using CppUTest as framework.
Will be thankful!
CppUTest is the perfect fit for embedded C development, since it is the only testing framework that allows to mock free functions (in your case, write_to_i2c() and read_from_i2c()).
Now, you should really be reading the CppUTest documentation or the excellent book Test Driven Development for Embedded C.
Anyway, the following code shows how to do it.
Your Unit Under Test (UUT), written in a way that is compilable (next time you ask a question on SO please make the effort):
#include "temperature.h"
#include "i2c.h"
void somethingA(void) { }
void somethingB(void) { }
void temperature_do(uint8_t address, uint8_t value) {
write_to_i2c(address, value);
const uint8_t tempReadback = read_from_i2c(address);
if (tempReadback == value) {
somethingA();
} else {
somethingB();
}
}
As you wrote, we need to "fake", or more exactly we need to "mock" write_to_i2c() and read_from_i2c(). We put the mocks in a separate file, say i2c_mock.cpp, so that when building the Unit Test, we link against the mock and not against the real implementation:
extern "C" {
#include "i2c.h"
};
#include "CppUTestExt/MockSupport.h"
void write_to_i2c(uint8_t address, uint8_t value) {
mock().actualCall(__FUNCTION__)
.withParameter("address", address)
.withParameter("value", value);
}
uint8_t read_from_i2c(uint8_t address) {
mock().actualCall(__FUNCTION__)
.withParameter("address", address);
uint8_t ret = mock().returnIntValueOrDefault(0);
return ret;
}
Please refer to the CppUMock documentation for details. This is just classic CppUMock boilerplate.
Last part is the Unit Test:
extern "C" {
#include "temperature.h" // UUT
};
#include "CppUTest/TestHarness.h"
#include "CppUTest/CommandLineTestRunner.h"
#include "CppUTestExt/MockSupport.h"
TEST_GROUP(Temperature)
{
void setup() {}
void teardown() {
mock().checkExpectations();
mock().clear();
}
};
TEST(Temperature, somethingA)
{
const uint8_t value = 10;
mock().ignoreOtherCalls();
mock().expectOneCall("read_from_i2c").ignoreOtherParameters()
.andReturnValue(value);
temperature_do(10, value);
}
TEST(Temperature, somethingB)
{
const uint8_t value = 10;
mock().ignoreOtherCalls();
mock().expectOneCall("read_from_i2c").ignoreOtherParameters()
.andReturnValue(value+1);
temperature_do(10, value);
}
int main(int argc, char** argv) {
return CommandLineTestRunner::RunAllTests(argc, argv);
}
This UT will actually give 100% branch coverage. Again, I cannot explain all the details. If you observe and compare test cases somethingA and somethingB, you will see what is needed to cause the UUT to once go in the path that calls somethingA() and once to go in the path that calls somethingB().
Let's take for example
mock().expectOneCall("read_from_i2c")
.ignoreOtherParameters()
.andReturnValue(value+1);
Here we are saying to CppUmock to expect a call to function read_from_i2c(), to ignore what the parameters are and, this is of fundamental importance, to return value + 1 (or anything else you fancy that is different from value). This will cause the UUT to go in the path that calls somethingB().
Happy embedded C development and happy unit testing!
I take a look into your proposed book about TDD and the mock objects. OK, so as I understand, for example in this line (mocks are already created):
mock().expectOneCall("read_from_i2c").ignoreOtherParameters()
.andReturnValue(value);
temperature_do(10, value);
the program "jumps" to the mocked "read_from_i2c" function (from i2c_mock.cpp) with parameters defined by myself and not the real one from the Unit Under Test? Or we really call our function from Unit Under Test but we manipulate this function with the parameters defined in the mock?
I have implemented a facade pattern that uses C functions underneath and I would like to test it properly.
I do not really have control over these C functions. They are implemented in a header. Right now I #ifdef to use the real headers in production and my mock headers in tests. Is there a way in C to exchange the C functions at runtime by overwriting the C function address or something? I would like to get rid of the #ifdef in my code.
To expand on Bart's answer, consider the following trivial example.
#include <stdio.h>
#include <stdlib.h>
int (*functionPtr)(const char *format, ...);
int myPrintf(const char *fmt, ...)
{
char *tmpFmt = strdup(fmt);
int i;
for (i=0; i<strlen(tmpFmt); i++)
tmpFmt[i] = toupper(tmpFmt[i]);
// notice - we only print an upper case version of the format
// we totally disregard all but the first parameter to the function
printf(tmpFmt);
free(tmpFmt);
}
int main()
{
functionPtr = printf;
functionPtr("Hello world! - %d\n", 2013);
functionPtr = myPrintf;
functionPtr("Hello world! - %d\n", 2013);
return 0;
}
Output
Hello World! - 2013
HELLO WORLD! - %D
It is strange that you even need an ifdef-selected header. The code-to-test and your mocks should have the exact same function signatures in order to be a correct mock of the module-to-test. The only thing that then changes between a production-compilation and a test-compilation would be which .o files you give to the linker.
It is possible With Typemock Isolator++ without creating unnecessary new levels of indirection. It can be done inside the test without altering your production code. Consider the following example:
You have the Sum function in your code:
int Sum(int a, int b)
{
return a+b;
}
And you want to replace it with Sigma for your test:
int Sigma(int a, int b)
{
int sum = 0;
for( ; 0<a ; a--)
{
sum += b;
}
return sum;
}
In your test, mock Sum before using it:
WHEN_CALLED: call the method you want to fake.
ANY_VAL: specify the args values for which the mock will apply. in this case any 2 integers.
*DoStaticOrGlobalInstead: The alternative behavior you want for Sum.
In this example we call Sigma instead.
TEST_CLASS(C_Function_Tests)
{
public:
TEST_METHOD(Exchange_a_C_function_implementation_at_run_time_is_Possible)
{
void* context = NULL; //since Sum global it has no context
WHEN_CALLED(Sum (ANY_VAL(int), ANY_VAL(int))).DoStaticOrGlobalInstead(Sigma, context);
Assert::AreEqual(2, Sum(1,2));
}
};
*DoStaticOrGlobalInstead
It is possible to set other types of behaviors instead of calling an alternative method. You can throw an exception, return a value, ignore the method etc...
For instance:
TEST_METHOD(Alter_C_Function_Return_Value)
{
WHEN_CALLED(Sum (ANY_VAL(int), ANY_VAL(int))).Return(10);
Assert::AreEqual(10, Sum(1,2));
}
I don't think it's a good idea to overwrite functions at runtime. For one thing, the executable segment may be set as read-only and even if it wasn't you could end up stepping on another function's code if your assembly is too large.
I think you should create something like a function pointer collection for the one and the other set of implementations you want to use. Every time you want to call a function, you'll be calling from the selected function pointer collection. Having done that, you may also have proxy functions (that simply call from the selected set) to hide the function pointer syntax.
I am writing a large C program for embedded use. Every module in this program has an init() function (like a constructor) to set up its static variables.
The problem is that I have to remember to call all of these init functions from main(). I also have to remember to put them back if I have commented them out for some reason.
Is there anything clever I do to make sure that all of these functions are getting called? Something along the lines of putting a macro in each init function that, when you call a check_inited() function later, sends a warning to STDOUT if not all the functions are called.
I could increment a counter, but I'd have to maintain the correct number of init functions somewhere and that is also prone to error.
Thoughts?
The following is the solution I decided on, with input from several people in this thread
My goal is to make sure that all my init functions are actually being called. I want to do
this without maintaining lists or counts of modules across several files. I can't call
them automatically as Nick D suggested because they need to be called in a certain order.
To accomplish this, a macro included in every module uses the gcc constructor attribute to
add the init function name to a global list.
Another macro included in the body of the init function updates the global list to make a
note that the function was actually called.
Finally, a check function is called in main() after all of the inits are done.
Notes:
I chose to copy the strings into an array. This not strictly necessary because the
function names passed will always be static strings in normal usage. If memory was short
you could just store a pointer to the string that was passed in.
My reusable library of utility functions is called "nx_lib". Thus all the 'nxl' designations.
This isn't the most efficient code in the world but it's only called a boot time so that
doesn't matter for me.
There are two lines of code that need to be added to each module. If either is omitted,
the check function will let you know.
you might be able to make the constructor function static, which would avoid the need to give it a name that is unique across the project.
this code is only lightly tested and it's really late so please check carefully before trusting it.
Thank you to:
pierr who introduced me to the constructor attribute.
Nick D for demonstrating the ## preprocessor trick and giving me the framework.
tod frye for a clever linker-based approach that will work with many compilers.
Everyone else for helping out and sharing useful tidbits.
nx_lib_public.h
This is the relevant fragment of my library header file
#define NX_FUNC_RUN_CHECK_NAME_SIZE 20
typedef struct _nxl_function_element{
char func[NX_FUNC_RUN_CHECK_NAME_SIZE];
BOOL called;
} nxl_function_element;
void nxl_func_run_check_add(char *func_name);
BOOL nxl_func_run_check(void);
void nxl_func_run_check_hit(char *func_name);
#define NXL_FUNC_RUN_CHECK_ADD(function_name) \
void cons_ ## function_name() __attribute__((constructor)); \
void cons_ ## function_name() { nxl_func_run_check_add(#function_name); }
nxl_func_run_check.c
This is the libary code that is called to add function names and check them later.
#define MAX_CHECKED_FUNCTIONS 100
static nxl_function_element m_functions[MAX_CHECKED_FUNCTIONS];
static int m_func_cnt = 0;
// call automatically before main runs to register a function name.
void nxl_func_run_check_add(char *func_name)
{
// fail and complain if no more room.
if (m_func_cnt >= MAX_CHECKED_FUNCTIONS) {
print ("nxl_func_run_check_add failed, out of space\r\n");
return;
}
strncpy (m_functions[m_func_cnt].func, func_name,
NX_FUNC_RUN_CHECK_NAME_SIZE);
m_functions[m_func_cnt].func[NX_FUNC_RUN_CHECK_NAME_SIZE-1] = 0;
m_functions[m_func_cnt++].called = FALSE;
}
// call from inside the init function
void nxl_func_run_check_hit(char *func_name)
{
int i;
for (i=0; i< m_func_cnt; i++) {
if (! strncmp(m_functions[i].func, func_name,
NX_FUNC_RUN_CHECK_NAME_SIZE)) {
m_functions[i].called = TRUE;
return;
}
}
print("nxl_func_run_check_hit(): error, unregistered function was hit\r\n");
}
// checks that all registered functions were called
BOOL nxl_func_run_check(void) {
int i;
BOOL success=TRUE;
for (i=0; i< m_func_cnt; i++) {
if (m_functions[i].called == FALSE) {
success = FALSE;
xil_printf("nxl_func_run_check error: %s() not called\r\n",
m_functions[i].func);
}
}
return success;
}
solo.c
This is an example of a module that needs initialization
#include "nx_lib_public.h"
NXL_FUNC_RUN_CHECK_ADD(solo_init)
void solo_init(void)
{
nxl_func_run_check_hit((char *) __func__);
/* do module initialization here */
}
You can use gcc's extension __attribute__((constructor)) if gcc is ok for your project.
#include <stdio.h>
void func1() __attribute__((constructor));
void func2() __attribute__((constructor));
void func1()
{
printf("%s\n",__func__);
}
void func2()
{
printf("%s\n",__func__);
}
int main()
{
printf("main\n");
return 0;
}
//the output
func2
func1
main
I don't know how ugly the following looks but I post it anyway :-)
(The basic idea is to register function pointers, like what atexit function does.
Of course atexit implementation is different)
In the main module we can have something like this:
typedef int (*function_t)(void);
static function_t vfunctions[100]; // we can store max 100 function pointers
static int vcnt = 0; // count the registered function pointers
int add2init(function_t f)
{
// todo: error checks
vfunctions[vcnt++] = f;
return 0;
}
...
int main(void) {
...
// iterate vfunctions[] and call the functions
...
}
... and in some other module:
typedef int (*function_t)(void);
extern int add2init(function_t f);
#define M_add2init(function_name) static int int_ ## function_name = add2init(function_name)
int foo(void)
{
printf("foo\n");
return 0;
}
M_add2init(foo); // <--- register foo function
Why not write a post processing script to do the checking for you. Then run that script as part of your build process... Or better yet, make it one of your tests. You are writing tests, right? :)
For example, if each of your modules has a header file, modX.c. And if the signature of your init() function is "void init()"...
Have your script grep through all your .h files, and create a list of module names that need to be init()ed. Then have the script check that init() is indeed called on each module in main().
If your single module represents "class" entity and has instance constructor, you can use following construction:
static inline void init(void) { ... }
static int initialized = 0;
#define INIT if (__predict_false(!initialized)) { init(); initialized = 1; }
struct Foo *
foo_create(void)
{
INIT;
...
}
where "__predict_false" is your compiler's branch prediction hint. When first object is created, module is auto-initialized (for once).
Splint (and probably other Lint variants) can give a warning about functions that are defined but not called.
It's interesting that most compilers will warn you about unused variables, but not unused functions.
Larger running time is not a problem
You can conceivably implement a kind of "state-machine" for each module, wherein the actions of a function depend on the state the module is in. This state can be set to BEFORE_INIT or INITIALIZED.
For example, let's say we have module A with functions foo and bar.
The actual logic of the functions (i.e., what they actually do) would be declared like so:
void foo_logic();
void bar_logic();
Or whatever the signature is.
Then, the actual functions of the module (i.e., the actual function declared foo()) will perform a run-time check of the condition of the module, and decide what to do:
void foo() {
if (module_state == BEFORE_INIT) {
handle_not_initialized_error();
}
foo_logic();
}
This logic is repeated for all functions.
A few things to note:
This will obviously incur a huge penalty performance-wise, so is
probably not a good idea (I posted
anyway because you said runtime is
not a problem).
This is not a real state-machine, since there are only two states which are checked using a basic if, without some kind of smart general logic.
This kind of "design-pattern" works great when you're using separate threads/tasks, and the functions you're calling are actually called using some kind of IPC.
A state machine can be nicely implemented in C++, might be worth reading up on it. The same kind of idea can conceivably be coded in C with arrays of function pointers, but it's almost certainly not worth your time.
you can do something along these lines with a linker section. whenever you define an init function, place a pointer to it in a linker section just for init function pointers. then you can at least find out how many init functions have been compiled.
and if it does not matter what order the init functions are called, and the all have the same prototype, you can just call them all in a loop from main.
the exact details elude my memory, but it works soemthing like this::
in the module file...
//this is the syntax in GCC..(or would be if the underscores came through in this text editor)
initFuncPtr thisInit __attribute((section(.myinits)))__= &moduleInit;
void moduleInit(void)
{
// so init here
}
this places a pointer to the module init function in the .myinits section, but leaves the code in the .code section. so the .myinits section is nothing but pointers. you can think of this as a variable length array that module files can add to.
then you can access the section start and end address from the main. and go from there.
if the init functions all have the same protoytpe, you can just iterate over this section, calling them all.
this, in effect, is creating your own static constructor system in C.
if you are doing a large project and your linker is not at least this fully featured, you may have a problem...
Can I put up an answer to my question?
My idea was to have each function add it's name to a global list of functions, like Nick D's solution.
Then I would run through the symbol table produced by -gstab, and look for any functions named init_* that had not been called.
This is an embedded app so I have the elf image handy in flash memory.
However I don't like this idea because it means I always have to include debugging info in the binary.