Sometimes a test is valid, but the code being tested is broken, and there is a defect logged against the test scenario. Once the defect is fixed, the test will pass.
Until then, I could mark the test as "ignore", with the defect number in the comments. Then the test won't run, it will get flagged as ignored, and the metrics will reflect the fact that it was ignored. When the defect is marked as fixed, I could find the test case and re-enable the test.
I could probably work with that, but I kinda want another option: mark the test as a defect. This causes the test to still be run, but reverses the semantics, where a failure is expected and nothing to get alarmed about, whereas a pass is a reason for celebration, and gets called out. Under this scenario, there are two more summary statistics: expected test failures and unexpected tests passing.
Does such a thing already exist in ScalaTest?
Reading the scaladoc for ScalaTest, I found the answer: pendingUntilFixed
An example (using FreeSpec, since that is my preferred syntax):
"some test that is known to be broken" in {
pendingUntilFixed {
info("Broken - see DEFECT-12345")
// original test code
...
}
}
This does EXACTLY what I was originally looking for, including failing once the code is no longer broken. Yay for ScalaTest! :)
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
When, for example, we compile the following code:
printf("hello);
we get a warning then an error about the missing " character. In my opinion, warnings inform us about a code that can be compiled but whose behaviour may be probably different from what the developper expects. Therefore my comprehension missed up two things:
Is there a complete code that can be compiled without errors while containing such a portion of code.
If such a code does not exist, why this missing character situation does not give us only an error (not a warning+error).
EDIT (I am doing my best to cope with off-topic votes recommendations):
1. Desired behavior : only one error diagnostic message, there is no need for a warning for the same thing.
Other related issues that do not let me accept the first answer:
2.1 Does printf_s() have the same issue? I tried to enable -c11 option with no success.
2.2 The historical reason to emit the warning does not seem to me to be plausible since why this double message was not used too in similar cases (old accepted constructions being forbidden in new c versions).
In my opinion, warnings inform us about a code that can be compiled
but whose behaviour may be probably different from what the developper
expects. Therefore my comprehension missed up two things:
Your opinion is irrelevant here, and neither the C standard nor the C++ standard distinguish different categories of diagnostic messages. That many compilers in fact do distinguish is an historically-based convention, albeit a widely observed one. What ultimately matters is what your compiler means by such a distinction (if indeed it makes one). On the other hand, and fortunately for you, GCC does adopt a convention similar to what you describe, as documented in its manual:
Errors report problems that make it impossible to compile your program. [...].
Warnings report other unusual conditions in your code that may indicate a problem, although compilation can (and does) proceed. [...]
(GCC 7.2 manual, section 13.9; the same or similar text appears also in earlier versions of the manual, back to at least v.4.)
Note well that the documentation frames the meaning of a warning slightly differently than you do: a GCC warning signals that compilation can proceed, but there is no assurance that it can complete successfully. If indeed it ultimately cannot then I would expect GCC, pursuant to its documentation, to also issue an error diagnostic. That is exactly what I observe with this test program, whether compiling as C or as C++:
#include <stdio.h>
int main(void) {
printf("hello);
}
I really think you are making far too much of the fact that GCC emits a warning in addition to an error in this case. That's an implementation quirk of no particular significance.
Is there a complete code that can be compiled without errors while containing such a portion of code.
It depends on exactly what you mean by that. Trivially, I could prefix the erroneous line in the above program with // to turn it into a comment, and that would make it perfectly valid C and C++ source. There are manifold other ways I could add to the given source without removing anything to make it valid -- some of them would even produce a program in which a printf() call is in fact performed.
I suppose that what you really want to know is whether there is code that would elicit the warning from GCC but not the corresponding error. To the best of my knowledge, modern GCC does not afford such code, but historically, GCC did allow it as an extension, in the form of embedded, unescaped newlines in string literals:
printf("hello);
Goodbye");
That behavior was already deprecated in GCC 3.2, and it was removed as early as GCC 4 (current is 7.2).
If such a code does not exist, why this missing character situation does not give us only an error (not a warning+error).
We can only guess, but it seems plausible that it derives from the historical existence of the language extension described above. And again, you are making far too much of this. GCC emits two diagnostics about the same problem -- so what? The ultimate purpose of the diagnostics is to help you figure out what is or may be wrong with your code, and the diagnostics GCC emits in this case do that job just fine.
Your compiler probably issues errors when it detects the program is ill-formed, and describes the immediate reason the program failed to be well formed at the location it happened.
This is often useless, because the mistake could have been many lines away.
It also issues warnings that are guesses (often educated guesses) what actually caused your problem. Maybe you forgot a ; on a previous line, failed to close a { or a (. The warning is not "this token is the point of error", but rather "this is where it all went wrong".
In reality, the C++ standard itself does not distinguish between warnings and errors; they are both diagnostics. It mandates some things cause diagnostics, and does not bar compilers from issuing additional diagnostics. Compilers are even free to compile ill formed programs with a warning.
I would expect an error for "newline in string", then a warning pointing at the open quote.
I have just dug up a bug in some code we're working with* that was failing due to the following situation:
Assert(SomeVitalFunction(foo) == OK)
This worked fine all the time the DEBUG macros were #defined:
#ifdef DEBUG
#define Assert(x) if((x) == 0){/*some error handling*/}
#else
#define Assert(x)
#endif
But when we #undef'd DEBUG it has the effect of deleting the vital function call from the code.
I can't for the life of me work out how that could ever work with DEBUG #undef'd, and it seems a bad idea generally to put any sort of function call inside an assert like this.
Have I missed something?
* = Edit to clarify following Carpetsmoker's comment: The code comes from a particularly backward cabal of Elbonian code slaves, our task has been to hack, slash, shave, polish, sanitize and apply lipstick to the thing.
You have missed nothing.
Asserts should always be written as if they could disappear at the flick of a compiler switch.
You can call functions that take a relatively long time to complete inside an assert (for example analysing the integrity of a data structure), because the function call will not be present in the release build. The flip side of this is that you cannot call functions that are necessary for correct operation.
It depends upon what SomeVitalFunction is doing. If it has no interesting side-effects, it is ok to use it inside an assert. But if calling or not calling SomeVitalFunction is essential to the program, it is a bug.
For instance, on POSIX, kill(2) with a 0 signal is only useful to test if a process is living. I would imagine that you might be sometimes tempted to use
assert(kill(sompid, 0) == 0); // process sompid exists
assuming that you always suppose that the process sompid is still running.
Likewise, you might use assert(hash_table_count(htbl)>0); to check that some hash table htbl is not empty.
BTW, notice that assert(3) is documented as being ignored if you compile with -DNDEBUG preprocessor option (not if -DDEBUG is not given).
I would like to test a little ANSI C interpreter.
My tool interprets my C program. It doesn't produce machine code; and I can't access the heap/stack after execution!
I was thinking of validating return values / outputs against GCC or something like this.
I was searching for something that fits for me, but i hardly found anything FREE or open source.
Does anybody have an idea/suggestion how to test my interpreter?
Can anybody recommend something like a test suite; or a package of test cases?
I also wrote a C interpreter and mainly relied on the gcc.c - torture/execute test cases. This test suite consists of "code fragments which have historically broken easily". The advantage of this test cases is, that they have to be executed to provide a result (and not, e.g., only compiled) and then the result is matched to an expected result. You do not need to compare it with the result of an executed program, that was compiled by GCC. You basically just need to include the files in the directory and execute them which contrasts some other test suites, where you have to parse expected results from configuration files or similar. You can download the tests from the GCC repository.
One disadvantage for you might be, that the test cases are often extensive and can also include GNU extensions such as the asm statement for executing assembler code or attributes on how to align memory content. They also include some test cases for some old K&C function style notation like in the following example (not taken from the suite):
int func(a)
int a;
{
return a + 1;
}
However, I still would recommend you to look through the test cases and see what you can execute, as they also test many "normal" corner cases for ASNI C.
I have the following C code that uses sqlite3:
if(SQLITE_OK == sqlite3_initialize()) {
self->db_open_result = sqlite3_open(self->db_uri, &(self->db));
} else {
self->db_open_result = SQLITE_ERROR;
}
Obviously I have a pretty high confidence that the code is correct and will behave as expected. However, I am measuring code coverage of my unit tests using gcov/lcov and I'm curious about how I might get my coverage number to 100% in this case. Under normal circumstances sqlite3_initialize() is never going to fail, so the else clause will never execute.
Is there a way to cause this to fail that isn't totally disruptive?
You want your unit-tests to test your code. But you also want to know that all of your test code has been properly exercised. One way to do that is to use "mocking", i.e. you replace your actual libraries (such as SQLite) with fake, or "mock" libraries and then run your programs against these fake libraries.
Whether this library replacement is done at compile time or runtime is really incidental, but in C it's easier to do it at compile time. You can do this mocking by hand, or you can use a tool such as Cmock.
In the faked library, you then provoke various errors and failures. Notably, the faked library doesn't even have to do anything or even keep track of much or any state, you can often get quite far by returning "OK" or "FAIL".
Is there a way to cause this to fail that isn't totally disruptive?
For portability reasons you should verify function success. What's happens if you not installed SQLite library? You cannot initialize the library if that happens.
"If for some reason, sqlite3_initialize() is unable to initialize the library (perhaps it is unable to allocate a needed resource such as a mutex) it returns an error code..."
So, if you want portability check the error.
How can I find where the error occurs?
In C language, the return value means what error occurs,
such as failure to open file or memory allocation.
There is no information where the error occurs.
For example, function 'foo' calls A,B,C,D.
If foo returns an error value, it might be return value of A or B or C or D.
I cannot find what function returns error.
I have to run debugger or add some codes to find what function returns error.
Yes, you'll have to use a debugger or add extra code to display some output. Also, make sure you look at the pre-conditions for the function you call (if there are any) - make sure that you obey what it demands before calling it.
I think you've answered your question, you need to use a debugger to find out.
Maybe add some breakpoint, look if the error occurs always or only sometimes, and in the same place.
C library functions like fopen() set errno to give you a bit more diagnostic information. You could use a similar strategy. But if you don't program in some way of getting extra information out like that, you are stuck with the debugger.
Design an appropriate error handling that provides the informations you need. Then write your functions around this design. Using the return values is just one of many possibilities to do this, but obviously not enough for your specific requirements.
What error occurs is a clue for where it occurs: if A can return only error #5, B only #42, C only #3 and D only #56, whenever foo returns #5, you know that it comes from A.
If error codes overlap over different functions, you can get other clues from your inputs and your outputs: if D either displays a message or issues an error #5, whenever foo returns #5 and the message gets displayed, you know that error comes from A.
If error code, inputs and output are not enough to retrieve error location, you have to increase observability of your program. Some solutions are:
add debug info and use a debugger
add logs that you can activate or inhibit at will
select unique error codes so that what error occurs identifies where it occurs as well