Inconsistent recv() behavior - c

Question

Running a little internal CTF to teach people some computer security basics and I've run into a strange behavior. The following is the handle function of a forking TCP server. It is just a cute little buffer overflow demonstration (taken from CSAW CTF).
When testing, I only ever bothered sending it 4097 bytes worth of data, because that will successfully overflow into the backdoor variable. However, many of the participants decided to try to send exactly 4099 bytes and this doesn't actually work. I'm not entirely sure why.
In GDB, recving 4099 bytes works just fine, but otherwise it does not. I've spent a good amount of time debugging this now, as I'd like a good explanation for everybody as to why the service behaved as it did. Is it some sort of quirk with the recv() call or am I doing something fundamentally wrong here?
void handle(int fd)
{
int backdoor = 0;
char attack[4096];
send(fd, greeting, strlen(greeting), 0);
sleep(3);
recv(fd, attack, 0x1003, 0);
if (backdoor)
{
dup2(fd, 0); dup2(fd, 1); dup2(fd, 2);
char* argv[] = {"/bin/cat", "flag", NULL};
execve(argv[0], argv, NULL);
exit(0);
}
send(fd, nope, strlen(nope), 0);
}
Edit
The executable was compiled with:
clang -o backdoor backdoor.c -O0 -fno-stack-protector
I did not use different optimization settings for debugging / the live executable. I can run the following command:
python -c "print 'A'*4099" | nc <ip> <port>
and this will not work. I then attach to the running process via GDB (setting a breakpoint directly after the recv call) and run the above command again and it does work. I have repeated this multiple times with some variations, yet the same results.
Could it be something to do with the way that the OS handles queueing excess bytes sent to the socket? When I am sending 4099 bytes with the above command, I am actually sending 5000 (Python's print appends a newline implicitly). This means that recv's newline gets truncated and is left for the next call to recv to clean up. Still can't figure out how GDB could influence this at all, but just a theory.

Answer 1

... am I doing something fundamentally wrong here?
Yes, you are expecting undefined behaviour to be predictable. It isn't.
If you compile that function with gcc, using -O3, then you'll get a warning about exceeding the size of the receive buffer (but of course you already knew that); but you'll also get a binary which does not actually bother to check backdoor. If you use clang, you don't get the warning, but you get a binary which doesn't even allocate space for backdoor.
The reason is clear: modifying backdoor through a "backdoor" is undefined behaviour, and the compiler is under no obligation to do anything you might consider logical or predictable in the face of undefined behaviour. In particular, it's allowed to assume that the undefined behaviour never happens. Since no valid program could mutate backdoor, the compiler is allowed to assume that backdoor never gets mutated, and hence it can ditch the code inside the if block as unreachable.
You don't mention how you're compiling this program, but if you're compiling without optimization to use gdb and with optimisation when you don't plan to use gdb, then you should not be surprised that undefined behaviour is handled differently. On the other hand, even if you are compiling the program with the same compiler and options in both cases, you still shouldn't be surprised, since undefined behaviour is, as it says, undefined.
Declaring backdoor as volatile might prevent the optimization. Although that's hardly the point, is it?
Note: I'm using gcc version 4.8.1 and clang version 3.4. Different versions (and even different builds) might have different results.