When undefined behavior can be considered well-known and accepted? - c

We know what undefined behavior is and we (more or less) know the reasons (performance, cross-platform compatibility) of most of them. Assuming a given platform, say Windows 32 bit, can we consider an undefined behavior as well-known and consistent across the platform? I understand there is not a general answer then I would restrict to two common UB I see pretty often in production code (in use from years).
1) Reference. Give this union:
union {
int value;
unsigned char bytes[sizeof(int)];
} test;
Initialized like this:
test.value = 0x12345678;
Then accessed with:
for (int i=0; i < sizeof(test.bytes); ++i)
printf("%d\n", test.bytes[i]);
2) Reference. Given an array of unsigned short* casting to (for example) float* and accessing it (reference, no padding between array members).
Is code relying on well-known UBs (like those) working by case (assuming compiler may change and for sure compiler version will change for sure) or even if they're UB for cross-platform code they rely on platform specific details (then it won't change if we don't change platform)? Does same reasoning apply also to unspecified behavior (when compiler documentation doesn't say anything about it)?
EDIT According to this post starting from C99 type punning is just unspecified, not undefined.

Undefined behavior means primarily a very simple thing, the behavior of the code in question is not defined so the C standard doesn't provide any clue of what can happen. Don't search more than that in it.
If the C standard doesn't define something, your platform may well do so as an extension. So if you are in such a case, you can use it on that platform. But then make sure that they document that extension and that they don't change it in the next version of your compiler.
Your examples are flawed for several reasons. As discussed in the comments, unions are made for type punning, and in particular an access to memory as any character type is always allowed. Your second example is really bad, because other than you seem to imply, this is not an acceptable cast on any platform that I know. short and float generally have different alignment properties, and using such a thing will almost certainly crash your program. Then, third, you are arguing for C on Windows, which is known for the fact that they don't follow the C standard.

First of all, any compiler implementation is free to define any behavior it likes in any situation which would, from the point of view of the standard, produce Undefined Behavior.
Secondly, code which is written for a particular compiler implementation is free to make use of any behaviors which are documented by that implementation; code which does so, however, may not be usable on other implementations.
One of the longstanding shortcomings of C is that while there are many situations where constructs which could Undefined Behavior on some implementations are handled usefully by others, only a tiny minority of such situations provide any means by which code can specify that a compiler which won't handle them a certain way should refuse compilation. Further, there are many cases in which the Standards Committee allows full-on UB even though on most implementations the "natural" consequences would be much more constrained. Consider, for example (assume int is 32 bits)
int weird(uint16_t x, int64_t y, int64_t z)
{
int r=0;
if (y > 0) return 1;
if (z < 0x80000000L) return 2;
if (x > 50000) r |= 31;
if (x*x > z) r |= 8;
if (x*x < y) r |= 16;
return r;
}
If the above code was run on a machine that simply ignores integer overflow, passing 50001,0,0x80000000L should result in the code returning 31; passing 50000,0,0x80000000L could result in it returning 0, 8, 16, or 24 depending upon how the code handles the comparison operations. The C standard, however, would allow the code to do anything whatsoever in any of those cases; because of that, some compilers might determine that none of the if statements beyond the first two could ever be true in any situation which hadn't invoked Undefined Behavior, and may thus assume that r is always zero. Note that one of the inferences would affect the behavior of a statement which precedes the Undefined Behavior.
One thing I'd really like to see would be a concept of "Implementation Constrained" behavior, which would be something of a cross between Undefined Behavior and Implementation-Defined Behavior: compilers would be required to document all possible consequences of certain constructs which under the old rules would be Undefined Behavior, but--unlike Implementation-Defined behavior--an implementation would not be required to specify one specific thing that would happen; implementations would be allowed to specify that a certain construct may have arbitrary unconstrained consequences (full UB) but would be discouraged from doing so. In the case of something like integer overflow, a reasonable compromise would be to say that the result of an expression that overflows may be a "magic" value which, if explicitly typecast, will yield an arbitrary (and "ordinary") value of the indicated type, but which may otherwise appears to have arbitrarily changing values which may or may not be representable. Compilers would be allowed to assume that the result of an operation will not be a result of overflow, but would refrain from making inferences about the operands. To use a vague analogy, the behavior would be similar to how floating-point would be if explicitly typecasting a NaN could yield any arbitrary non-NaN result.
IMHO, C would greatly benefit from combining the above concept of "implementation-constrained" behaviors with some standard predefined macros which would allow code to test whether an implementation makes any particular promises about its behavior in various situations. Additionally, it would be helpful if there were a standard means by which a section of code could request a particular "dialects" [combination of int size, implementation-constrained behaviors, etc.]. It would be possible to write a compiler for any platform which could, upon request, have promotion rules behave as though int was exactly 32 bits. For example, given code like:
uint64_t l1,l2; uint32_t w1,w2; uint16_t h1,h2;
...
l1+=(h1+h2);
l2+=(w2-w1);
A 16-bit compiler might be fastest if it performed the math on h1 and h2 using 16 bits, and a 64-bit compiler might be fastest if it added to l2 the 64-bit result of subtracting w1 from w2, but if the code was written for a 32-bit system, being able to have compilers for the other two systems generate code which would behave as it had on the 32-bit system would be more helpful than having them generate code which performed some different computation, no matter how much faster the latter code would be.
Unfortunately, there is not at present any standard means by which code can ask for such semantics [a fact which will likely limit the efficiency of 64-bit code in many cases]; the best one can do is probably to expressly document the code's environmental requirements somewhere and hope that whoever is using the code sees them.

Related

Consequences of null pointer subtraction

Technically, subtracting a null pointer is undefined behaviour in C. Clang 13 issues a warning for it.
Yet this construct is used anyway, usually to determine the alignment of a pointer. For example, BSD-derived implementations of qsort use it. See here (OpenBSD) and an explanation of what it's for:
Snippet of a code sample with null pointer subtraction from OpenBSD. Please see the link above for full context.
#define TYPE_ALIGNED(TYPE, a, es) \
(((char *)a - (char *)0) % sizeof(TYPE) == 0 && es % sizeof(TYPE) == 0)
Question: Is such code safe to use on typical modern platforms (64-bit or 32-bit) with typical modern compilers? A lot of prominent production code seemed to have used this construct for many years.
I notice that code like this was removed from FreeBSD's qsort (see revision 334928), because GCC miscompiled some of it. However, I do not understand all the details in the discussion of the issue, and I cannot tell if the problem was a direct consequence of the null pointer subtraction. However, their proposed fix essentially eliminates the null pointer subtraction. I would appreciate some clarifications on the topic.
When the C Standard was written, many hardware platforms performed pointer arithmetic in such a way that adding zero to a null pointer would yield a null pointer with no side effects, and subtracting one null pointer from another would yield zero with no side effects. These behaviors were often useful, since they could eliminate the need for corner-case code when performing tasks involving N-byte chunks of storage, where N might be zero.
Even though many platforms could support the aforementioned corner cases without having to generate any extra machine code, it was hardly clear that all platforms would be able to do so (I don't know of any particular platforms that couldn't, but wouldn't be at all surprised if some such platforms existed). The Standard thus handled such situations the same way as it handles other situations where almost all implementations would process a construct in the same useful fashion, but it might be impractical for all to do so: categorized the action as "Undefined Behavior" but allow implementations to, as a form of "conforming language extension", process it in a manner consistent with the underlying execution environment.
There was never any doubt about how such constructs should be processed on commonplace platforms. The only doubt would have been whether implementations whose target platforms would require extra machine code to yield the commonplace semantics should generate such extra machine code, and classifying such constructs as UB would allow such decisions to be made by people who were working with such platforms, and would thus be better placed than the Committee to weigh the costs and benefits of supporting the commonplace behavior.

Implementation-defined behavior in C

Please, can you report some example for implementation-defined behavior in C?
For example, I know from standard that "An implementation-defined behavior is the propagation of the high-order bit
when a signed integer is shifted right."
Can you explain to me the significant of the example and report an example?
I understand that int i; i >> 3. But why this is implementation-defined?
The definition of implementation-defined behavior in C is when something is left for the compiler to decide, and the compiler documents which choice it made.
There are hundreds of such cases in the language. The standard contains a summary of most of them in Annex J.3, which is ~15 pages long.
The specific example int i; i >> 3 is undefined behavior since the variable isn't initialized.
The specific example int i=0; i >> 3 is implementation-defined because the standard says so. C17 6.5.7/5:
The result of E1 >> E2 is E1 right-shifted E2 bit positions. /--/ If E1 has a signed type and a negative value, the resulting value is implementation-defined.
In this particular case, it depends on whether the compiler picks an arithmetic shift or a logical shift instruction from the CPU instruction set. Meaning that the standard doesn't disfavour architectures that lack an arithmetic shift. Though in practice, the vast majority of CPUs are capable of doing arithmetic shift, even RISC ones.
It's implementation defined because the C standards committee declined to define what should happen. The reason why they did that is because different CPUs do different things in such circumstances, and the ethos of C (in the interests of fast execution) is to not to put too much abstraction between the C source code and the op codes running on the CPU.
They could have chosen to define a standard behaviour, but then on some CPUs that'd require compilers to generate a load of code to make up for the fact that the CPU's own op codes don't provide that standardised behaviour. Whilst perfectly possible, it inevitably wouldn't be very efficient.
Others will no doubt be able to point to more authoritative versions of that explanation!
Any code that relies on implementation defined behaviour is only guaranteed to work under a specific platform and/or compiler. Portable programs should try to avoid such behaviour.
According to: https://clc-wiki.net/wiki/C_language:Terms:Implementation-defined_behaviour
That also gives another example:
int *o = malloc(0 * sizeof *o);
may result in o either being NULL or a unique pointer (as specified in 7.20.3 of the C99 Standard).

GCC: Should undefined behavior of overflows preserve logical consistency?

The following code produces strange things on my system:
#include <stdio.h>
void f (int x) {
int y = x + x;
int v = !y;
if (x == (1 << 31))
printf ("y: %d, !y: %d\n", y, !y);
}
int main () {
f (1 << 31);
return 0;
}
Compiled with -O1, this prints y: 0, !y: 0.
Now beyond the puzzling fact that removing the int v or the if lines produces the expected result, I'm not comfortable with undefined behavior of overflows translating to logical inconsistency.
Should this be considered a bug, or is the GCC team philosophy that one unexpected behavior can cascade into logical contradiction?
When invoking undefined behavior, anything can happen. There's a reason why it's called undefined behavior, after all.
Should this be considered a bug, or is the GCC team philosophy that one unexpected behavior can cascade into logical contradiction?
It's not a bug. I don't know much about the philosophy of the GCC team, but in general undefined behavior is "useful" to compiler developers to implement certain optimizations: assuming something will never happen makes it easier to optimize code. The reason why anything can happen after UB is exactly because of this. The compiler makes a lot of assumptions and if any of them is broken then the emitted code cannot be trusted.
As I said in another answer of mine:
Undefined behavior means that anything can happen. There is no explanation as to why anything strange happens after invoking undefined behavior, nor there needs to be. The compiler could very well emit 16-bit Real Mode x86 assembly, produce a binary that deletes your entire home folder, emit the Apollo 11 Guidance Computer assembly code, or whatever else. It is not a bug. It's perfectly conforming to the standard.
The 2018 C standard defines, in clause 3.4.3, paragraph 1, “undefined behavior” to be:
behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this document imposes no requirements
That is quite simple. There are no requirements from the standard. So, no, the standard does not require the behavior to be “consistent.” There is no requirement.
Furthermore, compilers, operating systems, and other things involved in building and running a program generally do not impose any requirement of “consistency” in the sense asked about in this question.
Addendum
Note that answers that say “anything can happen” are incorrect. The C standard only says that it imposes no requirements when there is behavior that it deems “undefined.” It does not nullify other requirements and has no authority to nullify them. Any specifications of compilers, operating systems, machine architectures, or consumer product laws; or laws of physics; laws of logic; or other constraints still apply. One situation where this matters is simply linking to software libraries not written in C: The C standard does not define what happens, but what does happen is still constrained by the other programming language(s) used and the specifications of the libraries, as well as the linker, operating system, and so on.
Marco Bonelli has given the reasons why such behaviour is allowed; I'd like to attempt an explanation of why it might be practical.
Optimising compilers, by definition, are expected to do various stuff in order to make programs run faster. They are allowed to delete unused code, unwrap loops, rearrange operations and so on.
Taking your code, can the compiler be really expected to perform the !y operation strictly before the call to printf()? I'd say if you impose such rules, there'll be no place left for any optimisations. So, a compiler should be free to rewrite the code as
void f (int x) {
int y = x + x;
int notY = !(x + x);
if (x == (1 << 31))
printf ("y: %d, !y: %d\n", y, notY);
}
Now, it should be obvious that for any inputs which don't cause overflow the behaviour would be identical. However, in the case of overflow y and notY experience the effects of UB independently, and may both become 0 because why not.
For some reason, a myth has emerged that the authors of the Standard used the phrase "Undefined Behavior" to describe actions which earlier descriptions of the language by its inventor characterized as "machine dependent" was to allow compilers to infer that various things wouldn't happen. While it is true that the Standard doesn't require that implementations process such actions meaningfully even on platforms where there would be a natural "machine-dependent" behavior, the Standard also doesn't require that any implementation be capable of processing any useful programs meaningfully; an implementation could be conforming without being able to meaningfully process anything other than a single contrived and useless program. That's not a twisting of the Standard's intention: "While a deficient implementation could probably contrive
a program that meets this requirement, yet still succeed in being useless, the C89 Committee felt that such ingenuity would probably require more work than making something useful."
In discussing the decision to make short unsigned values promote to signed int, the authors of the Standard observed that most current implementations used quiet wraparound integer-overflow semantics, and having values promote to signed int would not adversely affect behavior if the value was used in overflow scenarios where the upper bits wouldn't matter.
From a practical perspective, guaranteeing clean wraparound semantics costs a little more than would allowing integer computations to behave as though performed on larger types at unspecified times. Even in the absence of "optimization", even straightforward code generation for an expression like long1 = int1*int2+long2; would on many platforms benefit from being able to use the result of a 16x16->32 or 32x32->64 multiply instruction directly, rather than having to sign-extend the lower half of the result. Further, allowing a compiler to evaluate x+1 as a type larger than x at its convenience would allow it to replace x+1 > y with x >= y--generally a useful and safe optimization.
Compilers like gcc go further, however. Even though the authors of the Standard observed that in the evaluation of something like:
unsigned mul(unsigned short x, unsigned short y) { return x*y; }
the Standard's decision to promote x and y to signed int wouldn't adversely affect behavior compared with using unsigned ("Both schemes give the same answer in the vast majority of cases, and both give the same effective result in even more cases in implementations with two’s-complement arithmetic and quiet wraparound on signed overflow—that is, in most current implementations."), gcc will sometimes use the above function to infer within calling code that x cannot possibly exceed INT_MAX/y. I've seen no evidence that the authors of the Standard anticipated such behavior, much less intended to encourage it. While the authors of gcc claim any code that would invoke overflow in such cases is "broken", I don't think the authors of the Standard would agree, since in discussing conformance, they note: "The goal is to give the programmer a fighting chance to make powerful C programs that are also highly portable, without seeming to demean perfectly useful C programs that happen not to be portable, thus the adverb strictly."
Because the authors of the Standard failed to forbid the authors of gcc from processing code nonsensically in case of integer overflow, even on quiet-wraparound platforms, they insist that they should jump the rails in such cases. No compiler writer who was trying to win paying customers would take such an attitude, but the authors of the Standard failed to realize that compiler writers might value cleverness over customer satisfaction.

What replacements are available for formerly-widely-supported behaviors not defined by C standard

In the early days of C prior to standardization, implementations had a variety of ways of handling exceptional and semi-exceptional cases of various actions. Some of them would trigger traps which could cause random code execution if not configured first. Because the behavior of such traps was outside the scope of the C standard (and may in some cases be controlled by an operating system outside the control of the running program), and to avoid requiring that compilers not allow code which had been relying upon such traps to keep on doing so, the behavior of actions that could cause such traps was left entirely up to the discretion of the compiler/platform.
By the end of the 1990s, although not required to do so by the C standard, every mainstream compiler had adopted common behaviors for many of these situations; using such behaviors would allow improvements with respect to code speed, size, and readability.
Since the "obvious" ways of requesting the following operations are no longer supported, how should one go about replacing them in such a way as to not impede readability nor adversely affect code generation when using older compilers? For purposes of descriptions, assume int is 32-bit, ui is a unsigned int, si is signed int, and b is unsigned char.
Given ui and b, compute ui << b for b==0..31, or a value which may arbitrarily behave as ui << (b & 31) or zero for values 32..255. Note that if the left-hand operand is zero whenever the right-hand operand exceeds 31, both behaviors will be identical.
For code that only needs to run on a processor that yields zero when right-shifting or left-shifting by an amount from 32 to 255, compute ui << b for b==0..31 and 0 for b==32..255. While a compiler might be able to optimize out conditional logic designed to skip the shift for values 32..255 (so code would simply perform the shift that will yield the correct behavior), I don't know any way to formulate such conditional logic that would guarantee the compiler won't generate needless code for it.
As with 1 and 2, but for right shifts.
Given si and b such that b0..30 and si*(1<<b) would not overflow, compute si*(1<<b). Note that use of the multiplication operator would grossly impair performance on many older compilers, but if the purpose of the shift is to scale a signed value, casting to unsigned in cases where the operand would remain negative throughout shifting feels wrong.
Given various integer values, perform additions, subtractions, multiplications, and shifts, such fashion that if there are no overflows the results will be correct, and if there are overflows the code will either produce values whose upper bits behave in non-trapping and non-UB but otherwise indeterminate fashion or will trap in recognizable platform-defined fashion (and on platforms which don't support traps, would simply yield indeterminate value).
Given a pointer to an allocated region and some pointers to things within it, use realloc to change the allocation size and adjust the aforementioned pointers to match, while avoiding extra work in cases where realloc returns the original block. Not necessarily possible on all platforms, but 1990s mainstream platforms would all allow code to determine if realloc caused things to move, and determine the what the offset of a pointer into a dead object used to be by subtracting the former base address of that object (note that the adjustment would need to be done by computing the offset associated with each dead pointer, and then adding it the new pointer, rather than by trying to compute the "difference" between old and new pointers--something that would legitimately fail on many segmented architectures).
Do "hyper-modern" compilers provide any good replacements for the above which would not degrade at least one of code size, speed, or readability, while offering no improvements in any of the others? From what I can tell, not only could 99% of compilers throughout the 1990s do all of the above, but for each example one would have been able to write the code the same way on nearly all of them. A few compilers might have tried to optimize left-shifts and right-shifts with an unguarded jump table, but that's the only case I can think of where a 1990s compiler for a 1990s platform would have any problem with the "obvious" way of coding any of the above. If that hyper-modern compilers have ceased to support the classic forms, what do they offer as replacements?
Modern Standard C is specified in such a way that it can be guaranteed to be portable if and only if you write your code with no more expectations about the underlying hardware it will run on than are given by the C abstract machine the standard implicitly and explicitly describes.
You can still write for a specific compiler that has specific behaviour at a given optimization level for a given target CPU and architecture, but then do not expect any other compiler (modern or otherwise, or even a minor revision of the one you wrote for) to go out of its way to try to intuit your expectations if your code violates conditions where the Standard says that it is unreasonable to expect any well defined implementation agnostic behaviour.
Two general principles apply to standard C and standard C++:
Behavior with unsigned numbers is usually better defined than behavior with signed numbers.
Treat optimization as the quality-of-implementation issue that it is. This means that if you're worried about micro-optimization of a particular inner loop, you should read the assembly output of your compiler (such as with gcc -S), and if you discover that it fails to optimize well-defined behavior to appropriate machine instructions, file a defect report with the publisher of your compiler. (This doesn't work, however, when the publisher of the only practical compiler targeting a particular platform isn't very interested in optimization, such as cc65 targeting MOS 6502.)
From these principles, you can usually derive a well-defined way to achieve the same result, and then apply the trust-but-verify principle to the quality of generated code. For example, make shifting functions with well-defined behavior, and let the optimizer remove any unneeded checks that the architecture itself guarantees.
// Performs 2 for unsigned numbers. Also works for signed
// numbers due to rule for casting between signed and unsigned
// integer types.
inline uint32_t lsl32(uint32_t ui, unsigned int b) {
if (b >= 32) return 0;
return ui << b;
}
// Performs 3 for unsigned numbers.
inline uint32_t lsr32(uint32_t ui, unsigned int b) {
if (b >= 32) return 0;
return ui >> b;
}
// Performs 3 for signed numbers.
inline int32_t asr32(int32_t si, unsigned int b) {
if (si >= 0) return lsr32(si, b);
if (b >= 31) return -1;
return ~(~(uint32)si >> b);
}
For 4 and 5, cast to unsigned, do the math, and cast back to signed. This produces non-trapping well-defined behavior.

"Undefined behaviour" in a statement?

They say that when having UB, a program may do whatever it wants.
But if I have UB in one statement, such as
signed char a = 0x40;
a <<= 2;
or maybe even an unused(!) zero-size variable length array:
int l = 0;
char data[l];
is this in any way tolerable as only the result is undefined, or is this "bad" nevertheless?
I am especially interested in situations like these:
signed char a = 0x40;
a <<= 2;
switch (state) {
case X: return
case Y: do something with a; break;
case Z: do something else with a; break;
}
Assume that case X covers the case where the value of a is undefined, and the other cases make use of this case. It would simplify things if I was allowed to calculate this the way I want and make the distinction later.
Another situation is the one I talked about the other day:
void send_stuff()
{
char data[4 * !!flag1 + 2 * !!flag2];
uint8_t cursor = 0;
if (flag1) {
// fill 4 bytes of data into &data[cursor]
cursor += 4;
}
if (flag2) {
// fill 2 bytes of data into &data[cursor]
cursor += 2;
}
for (int i=0; i < cursor; i++) {
send_byte(data[i])
}
}
If both flags are unset, I have the "undefined" array data with length 0. But as I don't read from nor write to it, I don't see why and how it can possibly hurt...
Undefined behaviour means that it isn't defined by the C specification.
It may very well be defined (or partially defined) for a specific compiler.
Most compilers define a behavior for unsigned shift.
Most compilers define whether zero-length arrays are allowed.
Sometimes you can change the bahaviour with compiler flags, like --pedantic or flags that treat all warnings as errors.
So the answer to your question is:
That depends on the compiler. You need to check the documentation for your particular compiler.
Is it OK to rely on a specific result when you use something that is
UB according to the C standard?
That depends on what you are coding. If it is code for a specific embedded system where the likelyhood of ever porting to anywhere else is low, then by all means, rely on UB if it gives a big return. But best practice is to avoid UB when possible.
Edit:
is this in any way tolerable as only the result is undefined, or is this "bad" nevertheless?
Yes (only the result is undefined is true in practice, but in theory, the compiler manufacturer can terminate the program without breaking the C spec) and yes, it is bad nevertheless (because it requires additional tests to ensure that the behaviour remains the same after a change is made).
If the behaviour is unspecified, then you can observe what behaviour you get. Best is if you check the assembly code generated.
You need to be aware that the behaviour can change if you change anything, though. Changes that may change the behaviour include, but is not limited to, changes to the optimization level, and the application of compiler upgrades or patches.
The people who write the compilers are generally rational people which means that in most cases the program will behave in the way that was easiest for the compiler developer.
Best practice is still to avoid UB when possible.
You are confusing undefined and implementation defined behaviour.
Shifting a value by more bits than it has is implementation defined. It will have an effect and you need to read your compiler documentation. On some architectures, for instance, it it might have no effect, on others you'll be left with zero. Implementation defined behaviour isn't portable between architecture or compiler versions, requires no diagnostics, but will be consistent between runs.
However, declaring an array with a size of 0 is undefined behaviour. The compiler is free to make optimisations based on you not doing something like that, and produce code that doesn't work when you do. The compiler is free to do anything it likes if you invoke undefined behaviour, and it's possible your program will work today, and not tomorrow, or will work until you add another line somewhere else in the program or ....
Undefined means - not defined. There is no mileage in trying to work out how it's going to behave or depending on the results of said behavior.
In the context of your send_stuff function, the compiler is free to optimize your computation of cursor:
uint8_t cursor = flag1 ? 4 + 2 * !!flag2 : 2;
While this gives a different result for cursor when both flag1 and flag2 are 0, that's fine because that would result in undefined behavior anyway, so it's allowed to do whatever it wants in that case.
This is a completely machine-independent optimization, so even if you "know" that you'll always run on the same architecture with the same compiler, you can find that one day you make a seemingly unrelated change, the compiler gets tipped into a different decision about what optimal code looks like and your previously working code is suddenly behaving differently.

Resources