I'm currently testing out a few approaches on how to test and fuzz a C API. In the process thereof I found KLEE which runs the code symbolically, meaning that it tries to cover all branches that depend on some symbolic input and checks for all sorts of errors. I managed to get it to work and am now asking if my approach is good or if it has some major drawbacks or problems.
Let's suppose we have following simple but buggy API:
#include <assert.h>
static int g_state;
void setState(int state) {
g_state = state;
}
void run(void) {
if (g_state == 123) {
assert(0);
}
}
If the state is set to 123 and then run() is invoked the placed assertion fails.
For this I have written following symbolic test with KLEE:
#include "klee/klee.h"
#include "buggy_api.h"
int main(void) {
for (int i = 0; i < 2; ++i) { // sequentially call 2 APIs
int f_select = klee_choose(2); // what API to call
if (f_select == 0) {
int state = 0;
klee_make_symbolic(&state, sizeof(state), "state");
setState(state);
} else if (f_select == 1) {
run();
}
}
return 0;
}
When running with KLEE, the sequence of calls necessary to trigger the assertion is found almost immediately. I then tried to extend the API with a few other functions and hid the assertion behind a combination of states. KLEE again found the planted bug but naturally took a bit longer.
Is this how I can efficiently use KLEE for checking an API? Or is there documentation about a better approach?
In order to test an API with KLEE, you indeed need to write a driver that calls it. Yours works well, I'm not sure why you're using a for loop though? This smaller example should work:
#include "klee/klee.h"
#include "buggy_api.h"
int main(void) {
int state;
klee_make_symbolic(&state, sizeof(state), "state");
setState(state);
run();
return 0;
}
Related
I have read that C does not support dynamic function calls. My program has an ever growing number of test cases implemented as separate functions like -
int testcase1(void);
int testcase2(void);
int testcase3(void);
Each time I add a new test case, I also have have to add the call to my main function like -
int main(int argc, char **argv){
assert(!testcase1());
assert(!testcase2());
assert(!testcase3());
}
I would prefer to call something like assert(!testcase*()) where * matches any string which resolves to a valid function name in my program.
Can you think of a more convenient solution?
If you all your testcases have same signature then you can use an array of function pointers:
void (*func[])() = { testcase1, testcase2 };
for (size_t i = 0; i < sizeof(func)/sizeof(func[0]); i++) {
assert(!func[i]());
}
The best solution is likely to write a few extra lines of code when you add new test cases - it really isn't a big issue. I would recommend something along the lines of the function pointer array, as suggested in another answer.
However, just to show that everything is possible in C if you throw ugly macros at the problem, here is a not recommended alternative:
#include <assert.h>
#include <stdbool.h>
#include <stdio.h>
#define TEST_CASES \ // list of "x macros"
X(testcase1) \
X(testcase2) \
X(testcase3)
#define X(func) bool func (void); // declare function prototypes
TEST_CASES
#undef X
bool (*const test_cases[])(void) = // array of read-only function pointers
{
#define X(func) &func, // point at each function
TEST_CASES
#undef X
};
int main (void)
{
for(size_t i=0; i<sizeof(test_cases)/sizeof(test_cases[0]); i++)
{
assert(test_cases[i]());
}
}
bool testcase1 (void) { puts(__func__); return true; }
bool testcase2 (void) { puts(__func__); return true; }
bool testcase3 (void) { puts(__func__); return false; }
Output:
testcase1
testcase2
testcase3
Assertion failed!
For each new test case, you would only have to write a function definition and then add it to the "x macro" list TEST_CASES. However, you need very good reasons to introduce ugly tricks like these in production code!
You can use function pointers. Read also about closures (but C99 or C11 don't have them) and callbacks.
Many operating systems provide dynamic loading. On POSIX operating systems (such as Linux or MacOSX) you can get a function pointer (actually an address) from its name in some library (or in the program executable) using dlopen & dlsym. Other operating systems may provide similar functionalities.
At last, you should consider having your testing main function be generated by some script (or some program emitting C code), using metaprogramming techniques. So you would write something which generates the C code of your testing main having a long sequence of assert, and improve your build procedure (e.g. your Makefile if using make) to run appropriately that specialized C code generator. Details are of course specific to your code. You might add some conventions (e.g. add some special comment to be parsed by your test generator, etc...).
I decided to follow #Nominal Animal and #Basile Starynkevitch's approach. In mymainprog.c, I added -
int runtests(void){
void *testh;
int (*testp)(void);
char *dlmsg;
int rc;
char funcname[8];
int testnum;
testh = dlopen("libsmtests.so", RTLD_LAZY);
if (!testh){
printf("%s\n", dlerror());
return 1;
}
dlerror();
for (testnum =1; testnum < 1000; testnum++){
sprintf(funcname,"testcase%d", testnum);
*(void **) (&testp) = dlsym(testh, funcname);
dlmsg = dlerror();
if (dlmsg == NULL) {
rc = (*testp)();
printf("%s called, rc=%d\n", funcname, rc);
}
}
dlclose(testh);
return 0;
}
I add my testcases to a separate file (testcases.c) like this -
int testcase1(void){
return [some testcase expression]
}
int testcase2(void){
return [another testcase expression]
}
and then compile it as a shared library with position-independant code (-fPIC) to libsmtests.so. The advantage is slightly less typing since I don't need to code a call to testNNNN() after adding the implementation of a new functionint testcaseNNN(void) to testcases.c
I'm writing a unit-test to check some API calls. I am using check to test. My module is build with CMake (idk if it matters).
My test calls a function (which I need to test) and this function makes a call to another binary.
Simplified version of it looks like this.
/* unitTest.c */
#include "libraryAPI.h"
void letsMakeACall(void)
{
ck_assert_eq(foo("water"), 0);
}
-- Module I am working on---
/*libraryAPI.c*/
#include "legacyLib.h"
void foo(const char *drink )
{
if (checkDrink(drink)!=0)
{
return 1;
}else
{
return 0;
}
}
----LEGACY BINARY---
/*legacyLib.c*/
static const char* expected = "water";
void checkDrink(const char *drink)
{
if(drink == expected)
{
/*There are also a dozen functions being called which depend on legacy module initialisation*/
return 0;
}else{
return 1;
}
}
I'd like to mock response from legacyLib, because otherwise it call dozens of functions and breaks. My initial idea was to add some ifdef conditions when tests are being run, but it is against guidelines.
Because it is basically a call interception I don't know what it a best(or working) solution. What can I use to solve it?
I am also unsure how to solve this generally, I have posted a similar question, but in some cases you can do the following (presuming you are testing individual functions):
Include the .c file instead of the header .h, but after you "rename" your mocked function using a define directive:
#define checkDrink checkDrink_mocked
// preprocessor will now replace all occurrences of "checkDrink"
// with "checkDrink_mocked"
int checkDrink_mocked(const char *drink);
#include "legacyLib.c"
#undef checkDrink
Implement the renamed function:
int checkDrink_mocked(const char *drink)
{
return 15;
}
How can I catch the timeout exception in a third dll function,I use c language in Windows
I want to catch a timeout Exception while call a thirdly dll function, you know the function takes a long while, and I need it return a value in limited time, if it doesn't return in the time, I will give it a default value.
I have to look for so much infomation about but it doesn't work.
I get the two point:
1.use the alarm function in ,but it only work in Linux,I can't use it in Windows even I use the MinGW standerd GCC complier.
2.use the timeSetEvent function in and the setjmp/longjmp function in ,the three function maybe so closed to take it work.but I use them caused the programe dump,windows pops a DialogMessage say something wrong.
I give the code and the picture like this :
`
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#include <setjmp.h>
jmp_buf j;
/**
* 时间中断函数
*/
void PASCAL OneMilliSecondProc(UINT wTimerID, UINT msg, DWORD dwUser, DWORD dwl, DWORD dw2) {
printf("Timout!\n");
longjmp(j,1);
}
int longTimeFunction(){
while (1) {
printf("operating...\n");
Sleep(1000);
}
return 0;
}
int main(){
HANDLE hHandle;
UINT wTimerRes_1ms;//定义时间间隔
UINT wAccuracy; //定义分辨率
UINT TimerID_1ms; //定义定时器句柄
wTimerRes_1ms = 5000;
if((TimerID_1ms = timeSetEvent(
wTimerRes_1ms,
wAccuracy,
(LPTIMECALLBACK)OneMilliSecondProc, // 回调函数
(DWORD)(1), // 用户传送到回调函数的数据；
TIME_PERIODIC//周期调用定时处理函数
)) == 0) {
printf("start!!!!!!!!!!!\n");
} else {
printf("end!!!!!!!!!!!\n");
}
int temp = 0;
if(setjmp(j) == 0){
temp = longTimeFunction();
}else{
printf("xxxxxx...\n");
temp = -1;
}
printf("%d\n", temp);
return 0;
}
`
Unlike UNIX signals, timeSetEvent doesn't interrupt a thread, the callback runs in parallel and longjmping across threads is undefined behavior.
Concerning your actual question, this is a bad idea. Such an abortion could leave the library in an inconsistent state.
Instead, try to get the library vendor to offer an API that accepts a timeout, or use another library that already supports it.
I'm doing Unit Testing of embedded C code without running on target hardware.Here is one part of the code:
uint8 tempReadback = 0;
write_to_i2c( msg->addres, msg->value );
tempReadback = read_from_i2c( msg->addres);
if( tempReadback == msg->value )
{
somethingA;
}
else
{
somethingB;
}
The function write_to_i2c() writes a value to a specific register. The function read_from_i2c() reads back the value from the register. Further, I'm using the variable tempReadback to compare if the read back value is the same as the written one. So far OK, and this works on the target hardware. Now I'm doing the Uni Tests without running the code on target Hardware (Software in the Loop). This means, the expression tempReadback == msg->value will never be true (tempReadback is 0) and I will run each time in the statement somethingB. Is there any way to fake the register read back? I'm using CppUTest as framework.
Will be thankful!
CppUTest is the perfect fit for embedded C development, since it is the only testing framework that allows to mock free functions (in your case, write_to_i2c() and read_from_i2c()).
Now, you should really be reading the CppUTest documentation or the excellent book Test Driven Development for Embedded C.
Anyway, the following code shows how to do it.
Your Unit Under Test (UUT), written in a way that is compilable (next time you ask a question on SO please make the effort):
#include "temperature.h"
#include "i2c.h"
void somethingA(void) { }
void somethingB(void) { }
void temperature_do(uint8_t address, uint8_t value) {
write_to_i2c(address, value);
const uint8_t tempReadback = read_from_i2c(address);
if (tempReadback == value) {
somethingA();
} else {
somethingB();
}
}
As you wrote, we need to "fake", or more exactly we need to "mock" write_to_i2c() and read_from_i2c(). We put the mocks in a separate file, say i2c_mock.cpp, so that when building the Unit Test, we link against the mock and not against the real implementation:
extern "C" {
#include "i2c.h"
};
#include "CppUTestExt/MockSupport.h"
void write_to_i2c(uint8_t address, uint8_t value) {
mock().actualCall(__FUNCTION__)
.withParameter("address", address)
.withParameter("value", value);
}
uint8_t read_from_i2c(uint8_t address) {
mock().actualCall(__FUNCTION__)
.withParameter("address", address);
uint8_t ret = mock().returnIntValueOrDefault(0);
return ret;
}
Please refer to the CppUMock documentation for details. This is just classic CppUMock boilerplate.
Last part is the Unit Test:
extern "C" {
#include "temperature.h" // UUT
};
#include "CppUTest/TestHarness.h"
#include "CppUTest/CommandLineTestRunner.h"
#include "CppUTestExt/MockSupport.h"
TEST_GROUP(Temperature)
{
void setup() {}
void teardown() {
mock().checkExpectations();
mock().clear();
}
};
TEST(Temperature, somethingA)
{
const uint8_t value = 10;
mock().ignoreOtherCalls();
mock().expectOneCall("read_from_i2c").ignoreOtherParameters()
.andReturnValue(value);
temperature_do(10, value);
}
TEST(Temperature, somethingB)
{
const uint8_t value = 10;
mock().ignoreOtherCalls();
mock().expectOneCall("read_from_i2c").ignoreOtherParameters()
.andReturnValue(value+1);
temperature_do(10, value);
}
int main(int argc, char** argv) {
return CommandLineTestRunner::RunAllTests(argc, argv);
}
This UT will actually give 100% branch coverage. Again, I cannot explain all the details. If you observe and compare test cases somethingA and somethingB, you will see what is needed to cause the UUT to once go in the path that calls somethingA() and once to go in the path that calls somethingB().
Let's take for example
mock().expectOneCall("read_from_i2c")
.ignoreOtherParameters()
.andReturnValue(value+1);
Here we are saying to CppUmock to expect a call to function read_from_i2c(), to ignore what the parameters are and, this is of fundamental importance, to return value + 1 (or anything else you fancy that is different from value). This will cause the UUT to go in the path that calls somethingB().
Happy embedded C development and happy unit testing!
I take a look into your proposed book about TDD and the mock objects. OK, so as I understand, for example in this line (mocks are already created):
mock().expectOneCall("read_from_i2c").ignoreOtherParameters()
.andReturnValue(value);
temperature_do(10, value);
the program "jumps" to the mocked "read_from_i2c" function (from i2c_mock.cpp) with parameters defined by myself and not the real one from the Unit Under Test? Or we really call our function from Unit Under Test but we manipulate this function with the parameters defined in the mock?
I had an earlier question about integrating Mathematica with functions written in C++.
This is a follow-up question:
If the computation takes too long I'd like to be able to abort it using Evaluation > Abort Evaluation. Which of the technologies suggested in the answers make it possible to have an interruptible C-based extension function? How can "interruptibility" be implemented on the C side?
I need to make my function interruptible in a way which will corrupt neither it, nor the Mathematica kernel (i.e. it should be possible to call the function again from Mathematica after it has been interrupted)
For MathLink - based functions, you will have to do two things (On Windows): use MLAbort to check for aborts, and call MLCallYieldFunction, to yield the processor temporarily. Both are described in the MathLink tutorial by Todd Gayley from way back, available here.
Using the bits from my previous answer, here is an example code to compute the prime numbers (in an inefficient manner, but this is what we need here for an illustration):
code =
"
#include <stdlib.h>
extern void primes(int n);
static void yield(){
MLCallYieldFunction(
MLYieldFunction(stdlink),
stdlink,
(MLYieldParameters)0 );
}
static void abort(){
MLPutFunction(stdlink,\" Abort \",0);
}
void primes(int n){
int i = 0, j=0,prime = 1, *d = (int *)malloc(n*sizeof(int)),ctr = 0;
if(!d) {
abort();
return;
}
for(i=2;!MLAbort && i<=n;i++){
j=2;
prime = 1;
while (!MLAbort && j*j <=i){
if(i % j == 0){
prime = 0;
break;
}
j++;
}
if(prime) d[ctr++] = i;
yield();
}
if(MLAbort){
abort();
goto R1;
}
MLPutFunction(stdlink,\"List\",ctr);
for(i=0; !MLAbort && i < ctr; i++ ){
MLPutInteger(stdlink,d[i]);
yield();
}
if(MLAbort) abort();
R1: free(d);
}
";
and the template:
template =
"
void primes P((int ));
:Begin:
:Function: primes
:Pattern: primes[n_Integer]
:Arguments: { n }
:ArgumentTypes: { Integer }
:ReturnType: Manual
:End:
";
Here is the code to create the program (taken from the previous answer, slightly modified):
Needs["CCompilerDriver`"];
fullCCode = makeMLinkCodeF[code];
projectDir = "C:\\Temp\\MLProject1";
If[! FileExistsQ[projectDir], CreateDirectory[projectDir]]
pname = "primes";
files = MapThread[
Export[FileNameJoin[{projectDir, pname <> #2}], #1,
"String"] &, {{fullCCode, template}, {".c", ".tm"}}];
Now, here we create it:
In[461]:= exe=CreateExecutable[files,pname];
Install[exe]
Out[462]= LinkObject["C:\Users\Archie\AppData\Roaming\Mathematica\SystemFiles\LibraryResources\
Windows-x86-64\primes.exe",161,10]
and use it:
In[464]:= primes[20]
Out[464]= {2,3,5,7,11,13,17,19}
In[465]:= primes[10000000]
Out[465]= $Aborted
In the latter case, I used Alt+"." to abort the computation. Note that this won't work correctly if you do not include a call to yield.
The general ideology is that you have to check for MLAbort and call MLCallYieldFunction for every expensive computation, such as large loops etc. Perhaps, doing that for inner loops like I did above is an overkill though. One thing you could try doing is to factor the boilerplate code away by using the C preprocessor (macros).
Without ever having tried it, it looks like the Expression Packet functionality might work in this way - if your C code goes back and asks mathematica for some more work to do periodically, then hopefully aborting execution on the mathematica side will tell the C code that there is no more work to do.
If you are using LibraryLink to link external C code to the Mathematica kernel, you can use the Library callback function AbortQ to check if an abort is in progress.