In our company we develop bare metal embedded software for microcontrollers. Until now we have been using manual unit test on targets or simulators, specially for Renesas microcontrollers (RL78 and RX families). We're planning now to go into automatic unit tests. The idea is to integrate them in our existing CI system.
At this point we've got a dilema. Until now we've been running unit test using the same compiler and target (or simulator) that later has been used to deploy the software into production. We'd like to maintain this approach, as the developers (and everybody) specially appreciate to test and deploy using the same conditions. So the idea would be to take a testing tool/library programmed in C that allows as to compile and run the tests in an embedded environment using a simulator. (Ex. http://www.throwtheswitch.org/unity)
But, on the other side, we cope with two upcoming situations that make the dilema arise:
We're more and more going to Cortex uC, where it's more difficult to get specific simulators to allow automation. (Ex. Renesas RA family)
Many of the advanced testing tools are developed in C++ and thought for PC environment using gcc/g++ compiler in a x86 architecture, that doesn't match that of the Cortex targets compiled using arm-none-eabi-gcc that we foresee to use.
So, at this point, we're wondering, and this would be my question, what kind of reliability can have unit tests run using gcc if our final target will be a Cortex uC and the binaries will finally be generated using arm-none-eabi-gcc. indirectly I'd be asking for the differences between gcc and arm-none-eabi-gcc when compiling for different targets.
I'd appreciate feedback from someone knowing about gcc internals that could have coped with the same kind of problem.
Thanks in advance,
Ignasi Villagrasa
Generally, simulators are useless, but especially so for production testing. Since it is an embedded system, you want to test software and hardware both - testing software without the intended MCU and hardware in place is just nonsense.
If you insist on using fluffware like simulators or PC "test suites" then realize:
It is an incomplete test which does not test core functionality of your product.
It cannot be used to test drivers/hardware-related code, it can only test abstract algorithms.
It can only be used for development testing, never for production testing.
As for how to correctly test your specific embedded system, it depends on the application and what the product is supposed to do. If you do your projects by the book then you have: Specification, leading to implementation, leading to tests. The sole purpose of a test is to verify that the implementation follows the specification.
So if the specification says that the product should activate 10 relays, you will need to flash the software onto the live MCU on the real PCB and a correctly performed test then verifies that all 10 relays get activated as they should.
This complete and correct product test cannot be done in any other way. So ask yourself if you actually need the incorrect and incomplete simulated test at all. Perhaps your development-related testing should focus on more meaningful things like design reviews, coding standards, static analysis, code reviews etc.
I can't figure out how a desktop environment developer test his code. Usually, a C or C++ programmer compiles his code an then run it (i'm not one of those programmers, i'm a web one).
So, you usually build your gui application over some kind of desktop environment (windows, mac os x, gnome, kde, xfce...), sow how they build and test their gui desktop?
And if this is a silly question, how does a kernel programmer test his code? for example linux kernel? how do you know that what you just wrote works?
Testing is a very broad term there are many types (partial list):
unit tests - test small pieces of code. test that the code behaves as expected.
system tests - test whole application in real world scenarios.
performance tests - test what is the performance of the application or part of it.
GUI testing - test operation of GUI elements (not so common as automated tests)
static analysis - compiler warnings on steroids
dynamic analysis - at a minimum memory checks - check mem allocations and usage
coverage tests - check that all code is executed.
formal verification tests (very advanced) - e.g. check when assertions/assumptions are broken.
Kernel code can be debugged by connecting using a 2nd computer (host). Virtual machines uses the same principal and simplify the setup but can't always work as HW might not exist in the guest VM.
The kernel (all OSes) has trace mechanism(s) for printing progress/problems. In Linux the simple trace is shown via the dmesg command (prints a cyclic buffer).
User mode code can easily be stopped and debugged via a debugger.
Desktop Environments
Testing Desktop Environments in real world scenarios can be kind of annoying, so the developer would have to watch out for every small error he makes, if he doesn't, he will have a hard time developing the DE.
As stated by #egur, there are multiple ways of testing his code, the easiest one and most important (but cannot be used in some cases, of course), he can test that code in a simplified program.
A Desktop Environment consists of many parts, however, in your case, I suppose you're talking about the session manager (or window manager) which is responsible for almost everything. So, if he were to test that, he would simply exit his current DE and use the new executable. In case of some error, he can always keep a backup of the old executable or fix the faulty code using some commandline text editor (like vim, or nano).
Kernel
It's quite hard to test, some kernel developers just write some code and make sure it's fine and compiles, then simply let his users test (by ACK'ing the code, etc.), then it can be submitted into the kernel code. Reasoning behind that is, the developer may not have the hardware needed to test the code.
Right now, you can compile and run the kernel in usermode (UML) if you have heard of it, so some developers may go for it. However, some developers may also want to test it themselves (They of course back up the current kernel incase of a screw up).
The way to test a desktop application is related to the way of control the application unassisted or remotely.
The Cross Platform GUI Test Automation tool (I don't know if this project has a web) project helps you to chose the interfaces/libraries required to solve the problem.
In Linux[1] uses the accessibility libraries to control the application, you have Cobra[2] for Windows and PyATOM[3] for MacOS, but I don't know what kind of technology uses in this platforms.
http://ldtp.freedesktop.org/wiki/
https://github.com/ldtp/cobra
https://github.com/pyatom/pyatom
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I worked on an embedded system this summer written in straight C. It was an existing project that the company I work for had taken over. I have become quite accustomed to writing unit tests in Java using JUnit but was at a loss as to the best way to write unit tests for existing code (which needed refactoring) as well as new code added to the system.
Are there any projects out there that make unit testing plain C code as easy as unit testing Java code with JUnit? Any insight that would apply specifically to embedded development (cross-compiling to arm-linux platform) would be greatly appreciated.
One unit testing framework in C is Check; a list of unit testing frameworks in C can be found here and is reproduced below. Depending on how many standard library functions your runtime has, you may or not be able to use one of those.
AceUnit
AceUnit (Advanced C and Embedded Unit) bills itself as a comfortable C code unit test framework. It tries to mimick JUnit 4.x and includes reflection-like capabilities. AceUnit can be used in resource constraint environments, e.g. embedded software development, and importantly it runs fine in environments where you cannot include a single standard header file and cannot invoke a single standard C function from the ANSI / ISO C libraries. It also has a Windows port. It does not use forks to trap signals, although the authors have expressed interest in adding such a feature. See the AceUnit homepage.
GNU Autounit
Much along the same lines as Check, including forking to run unit tests in a separate address space (in fact, the original author of Check borrowed the idea from GNU Autounit). GNU Autounit uses GLib extensively, which means that linking and such need special options, but this may not be a big problem to you, especially if you are already using GTK or GLib. See the GNU Autounit homepage.
cUnit
Also uses GLib, but does not fork to protect the address space of unit tests.
CUnit
Standard C, with plans for a Win32 GUI implementation. Does not currently fork or otherwise protect the address space of unit tests. In early development. See the CUnit homepage.
CuTest
A simple framework with just one .c and one .h file that you drop into your source tree. See the CuTest homepage.
CppUnit
The premier unit testing framework for C++; you can also use it to test C code. It is stable, actively developed, and has a GUI interface. The primary reasons not to use CppUnit for C are first that it is quite big, and second you have to write your tests in C++, which means you need a C++ compiler. If these don’t sound like concerns, it is definitely worth considering, along with other C++ unit testing frameworks. See the CppUnit homepage.
embUnit
embUnit (Embedded Unit) is another unit test framework for embedded systems. This one appears to be superseded by AceUnit. Embedded Unit homepage.
MinUnit
A minimal set of macros and that’s it! The point is to show how easy it is to unit test your code. See the MinUnit homepage.
CUnit for Mr. Ando
A CUnit implementation that is fairly new, and apparently still in early development. See the CUnit for Mr. Ando homepage.
This list was last updated in March 2008.
More frameworks:
CMocka
CMocka is a test framework for C with support for mock objects. It's easy to use and setup.
See the CMocka homepage.
Criterion
Criterion is a cross-platform C unit testing framework supporting automatic test registration, parameterized tests, theories, and that can output to multiple formats, including TAP and JUnit XML. Each test is run in its own process, so signals and crashes can be reported or tested if needed.
See the Criterion homepage for more information.
HWUT
HWUT is a general Unit Test tool with great support for C. It can help to create Makefiles, generate massive test cases coded in minimal 'iteration tables', walk along state machines, generate C-stubs and more. The general approach is pretty unique: Verdicts are based on 'good stdout/bad stdout'. The comparison function, though, is flexible. Thus, any type of script may be used for checking. It may be applied to any language that can produce standard output.
See the HWUT homepage.
CGreen
A modern, portable, cross-language unit testing and mocking framework for C and C++. It offers an optional BDD notation, a mocking library, the ability to run it in a single process (to make debugging easier). A test runner which discover automatically the test functions is available. But you can create your own programmatically.
All those features (and more) are explained in the CGreen manual.
Wikipedia gives a detailed list of C unit testing frameworks under List of unit testing frameworks: C
Personally I like the Google Test framework.
The real difficulty in testing C code is breaking the dependencies on external modules so you can isolate code in units. This can be especially problematic when you are trying to get tests around legacy code. In this case I often find myself using the linker to use stubs functions in tests.
This is what people are referring to when they talk about "seams". In C your only option really is to use the pre-processor or the linker to mock out your dependencies.
A typical test suite in one of my C projects might look like this:
#include "myimplementationfile.c"
#include <gtest/gtest.h>
// Mock out external dependency on mylogger.o
void Logger_log(...){}
TEST(FactorialTest, Zero) {
EXPECT_EQ(1, Factorial(0));
}
Note that you are actually including the C file and not the header file. This gives the advantage of access to all the static data members. Here I mock out my logger (which might be in logger.o and give an empty implementation. This means that the test file compiles and links independently from the rest of the code base and executes in isolation.
As for cross-compiling the code, for this to work you need good facilities on the target. I have done this with googletest cross compiled to Linux on a PowerPC architecture. This makes sense because there you have a full shell and os to gather your results. For less rich environments (which I classify as anything without a full OS) you should just build and run on the host. You should do this anyway so you can run the tests automatically as part of the build.
I find testing C++ code is generally much easier due to the fact that OO code is in general much less coupled than procedural (of course this depends a lot on coding style). Also in C++ you can use tricks like dependency injection and method overriding to get seams into code that is otherwise encapsulated.
Michael Feathers has an excellent book about testing legacy code. In one chapter he covers techniques for dealing with non-OO code which I highly recommend.
Edit: I've written a blog post about unit testing procedural code, with source available on GitHub.
Edit: There is a new book coming out from the Pragmatic Programmers that specifically addresses unit testing C code which I highly recommend.
Minunit is an incredibly simple unit testing framework.
I'm using it to unit test c microcontroller code for avr.
I say almost the same as ratkok but if you have a embedded twist to the unit tests then...
Unity - Highly recommended framework for unit testing C code.
#include <unity.h>
void test_true_should_be_true(void)
{
TEST_ASSERT_TRUE(true);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_true_should_be_true);
return UNITY_END();
}
The examples in the book that is mentioned in this thread TDD for embedded C are written using Unity (and CppUTest).
I'm currently using the CuTest unit test framework:
http://cutest.sourceforge.net/
It's ideal for embedded systems as it's very lightweight and simple. I had no problems getting it to work on the target platform as well as on the desktop. In addition to writing the unit tests, all that's required is:
a header file included wherever
you're calling the CuTest routines
a single additional 'C' file to be
compiled/linked into the image
some simple code added to to main to
set up and call the unit tests - I
just have this in a special main()
function that gets compiled if
UNITTEST is defined during the
build.
The system needs to support a heap and some stdio functionality (which not all embedded systems have). But the code is simple enough that you could probably work in alternatives to those requirements if your platform doesn't have them.
With some judicious use of extern "C"{} blocks it also supports testing C++ just fine.
You also might want to take a look at libtap, a C testing framework which outputs the Test Anything Protocol (TAP) and thus integrates well with a variety of tools coming out for this technology. It's mostly used in the dynamic language world, but it's easy to use and becoming very popular.
An example:
#include <tap.h>
int main () {
plan(5);
ok(3 == 3);
is("fnord", "eek", "two different strings not that way?");
ok(3 <= 8732, "%d <= %d", 3, 8732);
like("fnord", "f(yes|no)r*[a-f]$");
cmp_ok(3, ">=", 10);
done_testing();
}
There is an elegant unit testing framework for C with support for mock objects called cmocka. It only requires the standard C library, works on a range of computing platforms (including embedded) and with different compilers.
It also has support for different message output formats like Subunit, Test Anything Protocol and jUnit XML reports.
cmocka has been created to also work on embedded platforms and also has Windows support.
A simple test looks like this:
#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>
/* A test case that does nothing and succeeds. */
static void null_test_success(void **state) {
(void) state; /* unused */
}
int main(void) {
const struct CMUnitTest tests[] = {
cmocka_unit_test(null_test_success),
};
return cmocka_run_group_tests(tests, NULL, NULL);
}
The API is fully documented and several examples are part of the source code.
To get started with cmocka you should read the article on LWN.net: Unit testing with mock objects in C
cmocka 1.0 has been released February 2015.
I didn't get far testing a legacy C application before I started looking for a way to mock functions. I needed mocks badly to isolate the C file I want to test from others. I gave cmock a try and I think I will adopt it.
Cmock scans header files and generates mock functions based on prototypes it finds. Mocks will allow you to test a C file in perfect isolation. All you will have to do is to link your test file with mocks instead of your real object files.
Another advantage of cmock is that it will validate parameters passed to mocked functions, and it will let you specify what return value the mocks should provide. This is very useful to test different flows of execution in your functions.
Tests consist of the typical testA(), testB() functions in which you build expectations, call functions to test and check asserts.
The last step is to generate a runner for your tests with unity. Cmock is tied to the unity test framework. Unity is as easy to learn as any other unit test framework.
Well worth a try and quite easy to grasp:
http://sourceforge.net/apps/trac/cmock/wiki
Update 1
Another framework I am investigating is Cmockery.
http://code.google.com/p/cmockery/
It is a pure C framework supporting unit testing and mocking. It has no dependency on ruby (contrary to Cmock) and it has very little dependency on external libs.
It requires a bit more manual work to setup mocks because it does no code generation. That does not represent a lot of work for an existing project since prototypes won't change much: once you have your mocks, you won't need to change them for a while (this is my case). Extra typing provides complete control of mocks. If there is something you don't like, you simply change your mock.
No need of a special test runner. You only need need to create an array of tests and pass it to a run_tests function. A bit more manual work here too but I definitely like the idea of a self-contained autonomous framework.
Plus it contains some nifty C tricks I didn't know.
Overall Cmockery needs a bit more understanding of mocks to get started. Examples should help you overcome this. It looks like it can do the job with simpler mechanics.
We wrote CHEAT (hosted on GitHub) for easy usability and portability.
It has no dependencies and requires no installation or configuration.
Only a header file and a test case is needed.
#include <cheat.h>
CHEAT_TEST(mathematics_still_work,
cheat_assert(2 + 2 == 4);
cheat_assert_not(2 + 2 == 5);
)
Tests compile into an executable that takes care of running the tests and reporting their outcomes.
$ gcc -I . tests.c
$ ./a.out
..
---
2 successful of 2 run
SUCCESS
It has pretty colors too.
As a C newbie, I found the slides called Test driven development in C very helpful. Basically, it uses the standard assert() together with && to deliver a message, without any external dependencies. If someone is used to a full stack testing framework, this probably won't do :)
There is CUnit
And Embedded Unit is unit testing framework for Embedded C System. Its design was copied from JUnit and CUnit and more, and then adapted somewhat for Embedded C System. Embedded Unit does not require std C libs. All objects are allocated to const area.
And Tessy automates the unit testing of embedded software.
Michael Feather's book "Working Effectively with Legacy Code" presents a lot of techniques specific to unit testing during C development.
There are techniques related to dependency injection that are specific to C which I haven't seen anywhere else.
I don't use a framework, I just use autotools "check" target support. Implement a "main" and use assert(s).
My test dir Makefile.am(s) look like:
check_PROGRAMS = test_oe_amqp
test_oe_amqp_SOURCES = test_oe_amqp.c
test_oe_amqp_LDADD = -L$(top_builddir)/components/common -loecommon
test_oe_amqp_CFLAGS = -I$(top_srcdir)/components/common -static
TESTS = test_oe_amqp
CppUTest - Highly recommended framework for unit testing C code.
The examples in the book that is mentioned in this thread TDD for embedded C are written using CppUTest.
I use CxxTest for an embedded c/c++ environment (primarily C++).
I prefer CxxTest because it has a perl/python script to build the test runner. After a small slope to get it setup (smaller still since you don't have to write the test runner), it's pretty easy to use (includes samples and useful documentation). The most work was setting up the 'hardware' the code accesses so I could unit/module test effectively. After that it's easy to add new unit test cases.
As mentioned previously it is a C/C++ unit test framework. So you will need a C++ compiler.
CxxTest User Guide
CxxTest Wiki
other than my obvious bias
http://code.google.com/p/seatest/
is a nice simple way to unit test C code. mimics xUnit
After reading Minunit I thought a better way was base the test in assert macro which I use a lot like defensive program technique. So I used the same idea of Minunit mixed with standard assert. You can see my framework (a good name could be NoMinunit) in k0ga's blog
Google has excellent testing framework. https://github.com/google/googletest/blob/master/googletest/docs/primer.md
And yes, as far as I see it will work with plain C, i.e. doesn't require C++ features (may require C++ compiler, not sure).
cmockery at http://code.google.com/p/cmockery/
Cmockery is a recently launched project that consists on a very simple to use C library for writing unit tests.
First, look here: http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C
My company has a C library our customers use. We use CxxTest (a C++ unit test library) to test the code. CppUnit will also work. If you're stuck in C, I'd recommend RCUNIT (but CUnit is good too).
If you are familiar with JUnit then I recommend CppUnit.
http://cppunit.sourceforge.net/cppunit-wiki
That is assuming you have c++ compiler to do the unit tests. if not then I have to agree with Adam Rosenfield that check is what you want.
I used RCUNIT to do some unit testing for embedded code on PC before testing on the target. Good hardware interface abstraction is important else endianness and memory mapped registers are going to kill you.
try lcut! - http://code.google.com/p/lcut
API Sanity Checker — test framework for C/C++ libraries:
An automatic generator of basic unit tests for a shared C/C++ library. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files.
The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging.
Examples:
Test suite for fontconfig 2.8.0
Test suite for FreeType 2.4.8
One technique to use is to develop the unit test code with a C++ xUnit framework (and C++ compiler), while maintaining the source for the target system as C modules.
Make sure you regularly compile your C source under your cross-compiler, automatically with your unit tests if possible.
LibU (http://koanlogic.com/libu) has an unit test module that allows explicit test suite/case dependencies, test isolation, parallel execution and a customizable report formatter (default formats are xml and txt).
The library is BSD licensed and contains many other useful modules - networking, debugging, commonly used data structures, configuration, etc. - should you need them in your projects ...
I'm surprised that no one mentioned Cutter (http://cutter.sourceforge.net/)
You can test C and C++, it seamlessly integrates with autotools and has a really nice tutorial available.
In case you are targeting Win32 platforms or NT kernel mode, you should have a look at cfix.
If you're still on the hunt for test frameworks, CUnitWin32 is one for the Win32/NT platform.
This solves one fundamental problem that I faced with other testing frameworks. Namely global/static variables are in a deterministic state because each test is executed as a separate process.
What things should be kept most in mind when writing cross-platform applications in C? Targeted platforms: 32-bit Intel based PC, Mac, and Linux. I'm especially looking for the type of versatility that Jungle Disk has in their USB desktop edition ( http://www.jungledisk.com/desktop/download.aspx )
What are tips and "gotchas" for this type of development?
I maintained for a number of years an ANSI C networking library that was ported to close to 30 different OS's and compilers. The library didn't have any GUI components, which made it easier. We ended up abstracting out into dedicated source files any routine that was not consistent across platforms, and used #defines where appropriate in those source files. This kept the code that was adjusted per platform isolated away from the main business logic of the library. We also made extensive use of typedefs and our own dedicated types so that we could easily change them per platform if needed. This made the port to 64-bit platforms fairly easy.
If you are looking to have GUI components, I would suggest looking at GUI toolkits such as WxWindows or Qt (which are both C++ libraries).
Try to avoid platform-dependent #ifdefs, as they tend to grow exponentially when you add new platforms. Instead, try to organize your source files as a tree with platform-independent code at the root, and platform-dependent code on the "leaves". There is a nice book on the subject, Multi-Platform Code Management. Sample code in it may look obsolete, but ideas described in the book are still brilliantly vital.
Further to Kyle's answer, I would strongly recommend against trying to use the Posix subsystem in Windows. It's implemented to an absolute bare minimum level such that Microsoft can claim "Posix support" on a feature sheet tick box. Perhaps somebody out there actually uses it, but I've never encountered it in real life.
One can certainly write cross-platform C code, you just have to be aware of the differences between platforms, and test, test, test. Unit tests and a CI (continuous integration) solution will go a long way toward making sure your program works across all your target platforms.
A good approach is to isolate the system-dependent stuff in one or a few modules at most. Provide a system-independent interface from that module. Then build everything else on top of that module, so it doesn't depend on the system you're compiling for.
XVT have a cross platform GUI C API which is mature 15+ years and sits on top of the native windowing toollkits. See WWW.XVT.COM.
They support at least LINUX, Windows, and MAC.
Try to write as much as you can with POSIX. Mac and Linux support POSIX natively and Windows has a system that can run it (as far as I know - I've never actually used it). If your app is graphical, both Mac and Linux support X11 libraries (Linux natively, Mac through X11.app) and there are numerous ways of getting X11 apps to run on Windows.
However, if you're looking for true multi-platform deployment, you should probably switch to a language like Java or Python that's capable of running the same program on multiple systems with little or no change.
Edit: I just downloaded the application and looked at the files. It does appear to have binaries for all 3 platforms in one directory. If your concern is in how to write apps that can be moved from machine to machine without losing settings, you should probably write all your configuration to a file in the same directory as the executable and not touch the Windows registry or create any dot directories in the home folder of the user that's running the program on Linux or Mac. And as far as creating a cross-distribution Linux binary, 32-bit POSIX/X11 would probably be the safest bet. I'm not sure what JungleDisk uses as I'm currently on a Mac.
There do exist quite few portable libraries just examples I've worked within the past
1) glib and gtk+
2) libcurl
3) libapr
Those cover nearly every platform and so they are extremly useful tool.
Posix is fine on Unices but well I doubt it's that great on windows, besides we do not have any stuff for portable GUIs there.
I also second the recommendation to separate code for different platforms into different modules/trees instead of ifdefs.
Also I recommend to check beforehand what are the differences in you platforms and how you could abstract them. E.g. this is some OS related stuff (e.g. the annoying CR,CRLF,LF in text files), or hardware stuff. E.g. the previous mentioned posix compability doesnt stop you from
int c;
fread(&c, sizeof(int), 1, file);
But on different hardware platforms the internal memory layout can be complete different (endianess), forcing you to use conversion functions on some of the target platforms.
You can use NAppGUI for both console and desktop apps. The SDK uses ANSI-C and your code will work on Windows/macOS/Linux.
https://www.nappgui.com
It's free and OpenSource.