Integer overflow not overflowing? - c

Question

I'm doing the ctf challenge from 247CTF "impossible numbers".
The challenge is about integer overflow, and consists of the following file:
#include <stdio.h>
int main() {
int impossible_number;
FILE *flag;
char c;
if (scanf("%d", &impossible_number)) {
if (impossible_number > 0 && impossible_number > (impossible_number + 1)) {
flag = fopen("flag.txt","r");
while((c = getc(flag)) != EOF) {
printf("%c",c);
}
}
}
return 0;
}
You can try the challenge at:
$ nc 1765a1cbe1629dfc.247ctf.com 50458
It's pretty simple, you need to trigger this case:
if (impossible_number > 0 && impossible_number > (impossible_number + 1))
Which you do by inputting 2147483647, which then overflows in the line impossible_number + 1.
This works for me, but I have also tried running it locally in vs code, and here the if statement is not triggered.
After doing some debugging, I have concluded that this is the proporsition that fails:
impossible_number > (impossible_number + 1)
This is really weird to me, I have even tried adding some prints of the values:
#include <stdio.h>
int main() {
int impossible_number;
FILE *flag;
char c;
if (scanf("%d", &impossible_number)) {
printf("impossible nr: %d \n", impossible_number);
printf("plus one nr: %d \n",impossible_number + 1 );
if (impossible_number > 0 && impossible_number > (impossible_number + 1)) {
flag = fopen("flag.txt","r");
while((c = getc(flag)) != EOF) {
printf("%c",c);
}
}
}
return 0;
}
which prints this:
impossible nr: 2147483647
plus one nr: -2147483648
This makes no sense to me, why does this work on the 247CTF server, but not when I run it?

Answer 1

As has been noted in the comments, signed integer overflow is undefined behavior in C.
The game's version of the program was apparently built with a compiler that handles it naively: by actually adding 1 to impossible_number (using ordinary two's-complement addition), then comparing the result with impossible_number and executing the fopen if it's less. In that case inputting 2147483647 works, as you saw. In my tests, clang without optimizations behaves like this.
But there are other possibilities. For instance, recent versions of GCC, even with -O0, notice that the test can't be true in any case when overflow doesn't occur. And if overflow does occur, the behavior is undefined, and so the compiler is at perfect liberty to do whatever it likes in that case. So it is allowed to assume that the test can't ever be true, and that's what it does: it optimizes away the entire if block, including the test itself which is now redundant. Try on godbolt; note that the generated assembly contains no call to fopen at all. So this program compiled with GCC is not vulnerable. The same is true for clang if optimizations are enabled (-O1 or higher).
(You can force the "naive" behavior in either compiler by compiling with -fwrapv. There is also -ftrapv which forces the program to abort if signed integer overflow ever occurs; it has a substantial runtime performance cost, but might be desirable when security is critical.)
Thus for an attack like this, you have to not only read the source code of the vulnerable program, but also be able to discover or guess what is in the compiled code that the victim is actually using.