I have a library I wrote with API based on opaque structures. Using opaque structures has a lot of benefits and I am very happy with it.
Now that my API are stable in term of specifications, I'd like to write a complete battery of unit test to ensure a solid base before releasing it.
My concern is simple, how do you unit test API based on opaque structures where the main goal is to hide the internal logic?
For example, let's take a very simple object, an array with a very simple test:
WSArray a = WSArrayCreate();
int foo = 5;
WSArrayAppendValue(a, &foo);
int *bar = WSArrayGetValueAtIndex(a, 0);
if(&foo != bar)
printf("Eroneous value returned\n");
else
printf("Good value returned\n");
WSRelease(a);
Of course, this tests some facts, like the array actually acts as wanted with 1 value, but when I write unit tests, at least in C, I usualy compare the memory footprint of my datastructures with a known state.
In my example, I don't know if some internal state of the array is broken.
How would you handle that? I'd really like to avoid adding codes in the implementation files only for unit testings, I really emphasis loose coupling of modules, and injecting unit tests into the implementation would seem rather invasive to me.
My first thought was to include the implementation file into my unit test, linking my unit test statically to my library.
For example:
#include <WS/WS.h>
#include <WS/Collection/Array.c>
static void TestArray(void)
{
WSArray a = WSArrayCreate();
/* Structure members are available because we included Array.c */
printf("%d\n", a->count);
}
Is that a good idea?
Of course, the unit tests won't benefit from encapsulation, but they are here to ensure it's actually working.
I would test only the API, and focus on testing every possible corner case.
I can see the interest in checking that the memory structures hold what you expect. If you do this you will be tightly coupling the tests to the specifics of the implementation and I think creating a lot of long-term maintenance.
My thought here is that the API is the contract and if you fulfil that then yoru code is working. If you change the implementation later then presumably one of the things you need to know is that the contract is maintained. Your unit tests will verify that.
Your unit tests shouldn't depend on the internal details of the code that they're testing. Your initial example is actually a pretty good test. It does one thing, then verifies that the state of the object is as expected.
You'd want to create tests that verify the behavior of other parts of the API as well, of course. Fir example, in the array case, you'd want to have test cases that verify that the length if the array is reported correctly after adding and removing items.
Writing unit tests that depend on an exact match with a known good memory snapshot is generally a really bad idea, in that every implementation change will cause the tests to fail. If you do decide to use snapshot-based tests, make sure there's an easy to regenerate the "known good" snapshots.
I would suggest splitting the unit testing into white box and black box unit testing. The white box testing focuses on the API interface, and correctness of results, while the black box testing focuses on the internals.
To facilitate this I use a private header (e.g. example_priv.h), with a #ifdef TESTING for function prototypes that are other internal / private. Thus you can exercise internal functions for unit testing purposes, without exposing them in the general case.
The only loss with this method is losing the ability to explicitly label the internal functions as static in their source file.
I hope that is helpful.
Related
My program answers on incoming messages and do some logic based on ID`s and data included in messages.
I have a different function for each ID.
The project is pure C.
To make the code easy to work with I have adjusted all functions to the same style (same return and parameters).
I also want to evade the long switch-case constructions and make code easier to edit later, so I have created the following function:
AnswerStruct IDHandler(Request Message)
{
struct AnswerStruct ANS;
SIDHandler = IDfunctions[Message.ID];
ANS = SIDHandler(Message);
return ANS;
}
AnswerStruct is struct for answer messages.
Request is struct for incoming messages.
IDfunctions is array of pointers to functions which looks like this -
AnswerStruct func1(Request);
AnswerStruct func4(Request);
...
typedef AnswerStruct(*f)(Request);
AnswerStruct (*SIDHandler)(Request);
static f IDfunctions[IDMax] = {0, *func1, 0, 0, *func4, ...};
Function pointers placed in the array cells equal to their id`s, for example:
func1 related to message with ID=1.
func4 related to message with ID=4.
I think, that by using this array I make my life much easier.
I can call function which I need in one step (just go to the IDfunctions[ID]).
Also, adding new functions becomes a two step operation (just add function to the IDfunctions and write logic).
I doubt the efficiency of the selected solution, it seems clunky to me.
The question is - Is this a good architecture?
If no, how can I edit my solution to make it better?
Thanks.
I doubt the efficiency of the selected solution, it seems clunky to
me.
It can be less efficient to call a function via a function pointer than to call it directly by name, because the former denies the compiler any opportunity to optimize the call. But you have to consider whether that actually matters. In a system that dispatches function calls based on messages received from an external source, the I/O involved in receiving the messages is likely to be much more expensive than the indirect function calls, so the difference in call performance is unlikely to be significant.
On the other hand, your approach affords simpler logic and many fewer lines of code, which is a different and potentially more valuable kind of efficiency.
The question is - Is this a good architecture?
The general approach is perfectly good, and I don't see much to complain about in the implementation sketch provided.
Personally, I would declare array IDFunctions to be const (supposing, of course, that you don't intend to replace any of its members after their initialization), but that's a minor safety / performance detail, where again the performance dimension is probably irrelevant.
I am tasked to assist with the design of a dynamic library (exposed with a C interface) aimed to be used in embed software application on various embed platform (Android,Windows,Linux).
Main requirements are speed , and decoupling.
For the decoupling part : one of our requirement is to be able to facilitate integration and so permit backward compatibility and resilience.
My library have some entry points that should be called by the integrating software (like an initialize constructor to provide options as where to log, how to behave etc...) and could also call some callback in the application (an event to inform when task is finished).
So I have come with several propositions but as each of one not seems great I am searching advice on a better or standard ways to achieve decoupling an d backward compatibility than this 3 ways that I have come up :
First an option that I could think of is to have a generic interface call for my exposed entry points for example with a hashmap of key/values for the parameters of my functions so in pseudo code it gives something like :
myLib.Initialize(Key_Value_Option_Array_Here);
Another option is to provide a generic function to provide all the options to the library :
myLib.SetOption(Key_Of_Option, Value_OfOption);
myLib.SetCallBack(Key_Of_Callbak, FunctionPointer);
When presenting my option my collegue asked me why not use a google protobuf argument as interface between the library and the embed software : but it seems weird to me, as their will be a performance hit on each call for serialization and deserialization.
Are there any more efficient or standard way that you coud think of?
You could have a struct for optional arguments:
typedef struct {
uint8_t optArg1;
float optArg2;
} MyLib_InitOptArgs_T;
void MyLib_Init(int16_t arg1, uint32_t arg2, MyLib_InitOptArgs_T const * optionalArgs);
Then you could use compound literals on function call:
MyLib_Init(1, 2, &(MyLib_InitOptArgs_T){ .optArg2=1.2f });
All non-specified values would have zero-ish value (0, NULL, NaN), and would be considered unused. Similarly, when passing NULL for struct pointer, all optional arguments would be considered unused.
Downside with this method is that if you expect to have many new arguments in the future, structure could grow too big. But whether that is an issue, depends on what your limits are.
Another option is to simply have multiple smaller initialization functions for initializating different subsystems. This could be combined with the optional arguments system above.
I write code in C. I have been striving to write more testable code but I am a little
confused on deciding between writing pure functions that are really good for testing
but require smaller functions and hurt readability in my opinion and writing functions
that do modify some internal state.
For example (all state variables are declared static and hence are "private" to my module):
Which of this is more testable in your opinion:
int outer_API_bar()
{
// Modify internal state
internal_foo()
}
int internal_foo()
{
// Do stuff
if (internal_state_variable)
{
// Do some more stuff
internal_state_variable = false;
}
}
OR
int outer_API_bar()
{
// Modify internal state
internal_foo(internal_state_variable)
// This could be another function if repeated many
// times in the module
if (internal_state_variable)
{
internal_state_variable = false;
}
}
int internal_foo(bool arg)
{
// Do stuff
if (arg)
{
// Do some more stuff
}
}
Although second implementation is more testable wrt to internal_foo as it has no sideeffects but it makes bar uglier and requires smaller functions that make it hard for the reader to even follow small snippets as he has to constantly shift attention to different functions.
Which one do you think is better ? Compare this to writing OOPS code, the private functions most of the time use internal state and are not pure. Testing is done by setting up internal state on a mock object instance and testing the private function. I am getting a little confused on whether to use or whether to pass in internal state to private functions for the sake of "testability"
Whenever writing automated tests, ideally we want to focus on testing the specification of that unit of code, not the implementation (otherwise we create fragile tests that will break whenever we modify the implementation). Therefore, what happens internally in the object should not be of concern to the test.
For this example, I would look to build a test that:
Executes the test by calling outer_API_bar.
Asserts that the correct behavior of the call using other publicly accessible functions and/or state (there must be some way of doing this, as if the only side effect of calling outer_API_bar was internal to this unit of code, then calling this function could not impact your wider application in any way, and essentially be useless).
This way, you are able to keep the fact that you use functions like internal_foo, and variables like internal_state_variable as implementation details, which you can freely change when refactoring your code (i.e. to make it more readable) without having to change your tests.
NOTE: This suggestion is based on my own personal preference for only testing public functions, and not private ones. You will find much debate on this topic where some people pose good arguments for testing private functions being a valid thing to do.
To answer your question very specifically pure functions are waaaaay more 'testable' than any other kind of abstraction. The more pure functions you can include, the more testable your code would be. As you rightly mention, this can come at the cost of readability, and I am sure there are other trade offs to consider. My suggestion would be to aim for more pure functions and look for other techniques that would allow you to compensate on the readability side of things.
Both snippets are testable via mocks. The second one, however, has the advantage that you can also check the argument of internal_foo(bool arg) for an expected value of true or false when the mock for internal_foo() is invoked. In my opinion, that would make for a more meaningful test.
Depending on the rest of the code that we don't know, testing without mocks may be more difficult.
I work in safety critical application development. Recently as a code reviewer I complained against coding style shown below, but couldn't make a strong case against it. So what would be a good argument against such Variable redundancy/duplication, I am looking for cases where this might lead to problems or test cases which might fail, rather than just coding style.
//global data
// global data
int Block1Var;
int Block2Var;
...
//Block1
{
...
Block1Var = someCondition; // someCondition is an logical expression
...
}
//Block2
{
...
Block2Var = Block1Var; // Block2Var is an unconditional copy of Block1Var
...
}
I think a little more context would be helpful perhaps.
You could argue that the value of Block1Var is not guaranteed to stay the
same across concurrent access/modification. This is only valid if Block1Var
ever changes (ie is not only read). I don't know if you are concerned with
multi-threaded applications or not.
Readability is an important issue as well. Future code maintainers
don't want to have to trace around a bunch of trivial assignments.
Depends on what's done with those variables later, but one argument is that it's not future-proof. If, in the future, you change the code such that it changes the value of Block1Var, but Block2Var is used instead (without the additional change) later on, then this will result in erroneous behavior.
If the shown function context reaches a certain length (I'm assuming a lot of detail has been discarded to create the minimal reproducible example for this question), a good next step could be to create a new (sub-)function out of Block 2. This subfunction then should be started assigning Block1Var (-> actual parameter) to Block2Var (-> formal parameter). If there were no other coupling to the rest of the function, one could cut the rest of Block 2 and drop it as a function definition, and would only have to replace the assignment by the subfunction call.
My answer is fairly speculative, but I have seen many cases where this strategy helped me to mark useful points to split a complex function later during the development. Of course, this interpretation only applies to an intermediate stage of development and not to code that is stated to be "ready for release".
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
One way of thinking about this is: if we care about the design of the code then EasyMock is the better choice as it gives feedback to you by its concept of expectations.
If we care about the maintainability of tests (easier to read, write and having less brittle tests which are not affected much by change), then Mockito seems a better choice.
My questions are:
If you have used EasyMock in large scale projects, do you find that your tests are harder to maintain?
What are the limitations of Mockito (other than endo testing)?
I won't argue about test readability, size or testing techniques of these frameworks, I believe they are equal, but on a simple example I'll show you the difference.
Given: We have a class which is responsible for storing something somewhere:
public class Service {
public static final String PATH = "path";
public static final String NAME = "name";
public static final String CONTENT = "content";
private FileDao dao;
public void doSomething() {
dao.store(PATH, NAME, IOUtils.toInputStream(CONTENT));
}
public void setDao(FileDao dao) {
this.dao = dao;
}
}
and we want to test it:
Mockito:
public class ServiceMockitoTest {
private Service service;
#Mock
private FileDao dao;
#Before
public void setUp() {
MockitoAnnotations.initMocks(this);
service = new Service();
service.setDao(dao);
}
#Test
public void testDoSomething() throws Exception {
// given
// when
service.doSomething();
// then
ArgumentCaptor<InputStream> captor = ArgumentCaptor.forClass(InputStream.class);
Mockito.verify(dao, times(1)).store(eq(Service.PATH), eq(Service.NAME), captor.capture());
assertThat(Service.CONTENT, is(IOUtils.toString(captor.getValue())));
}
}
EasyMock:
public class ServiceEasyMockTest {
private Service service;
private FileDao dao;
#Before
public void setUp() {
dao = EasyMock.createNiceMock(FileDao.class);
service = new Service();
service.setDao(dao);
}
#Test
public void testDoSomething() throws Exception {
// given
Capture<InputStream> captured = new Capture<InputStream>();
dao.store(eq(Service.PATH), eq(Service.NAME), capture(captured));
replay(dao);
// when
service.doSomething();
// then
assertThat(Service.CONTENT, is(IOUtils.toString(captured.getValue())));
verify(dao);
}
}
As you can see both test are fairly the same and both of them are passing.
Now, let’s imagine that somebody else changed Service implementation and trying to run tests.
New Service implementation:
dao.store(PATH + separator, NAME, IOUtils.toInputStream(CONTENT));
separator was added at the end of PATH constant
How the tests results will look like right now ? First of all both tests will fail, but with different error messages:
EasyMock:
java.lang.AssertionError: Nothing captured yet
at org.easymock.Capture.getValue(Capture.java:78)
at ServiceEasyMockTest.testDoSomething(ServiceEasyMockTest.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
Mockito:
Argument(s) are different! Wanted:
dao.store(
"path",
"name",
<Capturing argument>
);
-> at ServiceMockitoTest.testDoSomething(ServiceMockitoTest.java:34)
Actual invocation has different arguments:
dao.store(
"path\",
"name",
java.io.ByteArrayInputStream#1c99159
);
-> at Service.doSomething(Service.java:13)
What happened in EasyMock test, why result wasn't captured ? Is store method wasn't executed, but wait a minute, it was, why EasyMock lies to us?
It's because EasyMock mixing two responsibilities in a single line - stubbing and verification. That's why when something is wrong it's hard to understand which part is causing failure.
Of course you can tell me - just change the test and move verify before assertion. Wow, are you serious, developers should keep in mind some magic order inforced by mocking framework?
By the way, it won’t help:
java.lang.AssertionError:
Expectation failure on verify:
store("path", "name", capture(Nothing captured yet)): expected: 1, actual: 0
at org.easymock.internal.MocksControl.verify(MocksControl.java:111)
at org.easymock.classextension.EasyMock.verify(EasyMock.java:211)
Still, it is saying to me that method was not executed, but it was, only with another parameters.
Why Mockito is better ? This framework doesn't mix two responsibilities in a single place and when your tests will fail, you will easily understand why.
if we care about the Design of the code then Easymock is the better choice as it gives feedback to you by its concept of expectations
Interesting. I found that 'concept of expectations' makes many devs put more & more expectations in the tests only to satisfy UnexpectedMethodCall problem. How does it influence the design?
The test should not break when you change code. The test should break when the feature stops working. If one likes the tests to break when any code change happens I suggest to write a test that asserts the md5 checksum of the java file :)
I'm an EasyMock developer so a bit partial but of course I've used EasyMock on large scale projects.
My opinion is that EasyMock tests will indeed breaks once in a while. EasyMock forces you to do a complete recording of what you expect. This requires some discipline. You should really record what is expected not what the tested method currently needs. For instance, if it doesn't matter how many time a method is called on a mock, don't be afraid of using andStubReturn. Also, if you don't care about a parameter, use anyObject() and so on. Thinking in TDD can help on that.
My analyze is that EasyMock tests will break more often but Mockito ones won't when you would want them to. I prefer my tests to break. At least I'm aware of what was the impacts of my development. This is of course, my personal point of view.
I don't think you should be too concerned about this. Both Easymock and Mockito can be configured to be 'strict' or 'nice' the only difference is that by default Easymock is strict wheras Mockito is nice.
As with all testing there's no hard and fast rule, you need to balance test confidence against maintainability. I typically find there are certain functional or technical areas that demand a high level of confidence for which I would use 'strict' mocks. For example we probably wouldn't want the debitAccount() method to be called more than once! However there are other cases in which the mock is really little more than a stub so we can test the real 'meat' of the code.
In the early days of Mockito's life API compatibility was a problem but more tools now support the framework. Powermock (a personal favorite) now has a mockito extension
I prefer mockito to be honest. been using EasyMock with unitils and the combination of both oftenly results in exceptions like IllegalArgumentException: not an interface as well as MissingBehaviorExceptions. In both cases though the code and test code are perfectly fine. It appeared that the MissingBehaviorException was due to the fact that mocked objects created with createMock (using classextentions!!) did produce this error. When using #Mock it did work! I do not like that kind of misleading behavior and for me that is a clear indication the developers of it do not know what they are doing. A good framework should always be easy to use and not ambiguous. The IllegalArgumentException was also due to some mingle of EasyMock internals. Also, the recording is not what I want to do. I want to test if my code throws exceptions or not and that it returns the expected results. That in combination with code coverage is the right tool for me. I do not want my tests to break whenever I put 1 line of code above or below the previous one because that improves performance or so. With mockito it is no problem. With EasyMock, it will result tests to fail even though the code is not broken. That is bad. It costs time, thus money. You want to test for expected behavior. Do you really care about the order of things? I suppose in rare occasions you might. Use Easymock then. In other case, I think you'll spend considerably less time using mockito to write your tests.
Kind regards
Lawrence