Program getting throttled linux c - c

Question

i have the following loop that is supposed to run at a little above 40Hz (24ms between each run). It works great for a couple of second to minutes, but then goes down to 20Hz for no apparent (at least not to me) reason.
I have tried running the code on a onion Omega 2 (that runs openwrt) and a ubuntu laptop with the same result. (the code went down to 25Hz on the omega2, and 20Hz on my laptop)
How can i force it to continue to run at the right speed? I have checked with an oscilloscope on the output, and the send_brk and send_dmx functions seem to run perfectly even when the program is throttled, so i don't think they are the problem.
double t1 = now();
double t2 = now();
while ( 1 ) {
// Attempt receive UDP data
int x = read(sock, &DMX[1], SIZE - 1);
t2 = now();
double dt = t2 - t1;
if(dt < 0.024){
usleep(500);
continue;
}
t1 = t2;
printf("Frame in %f %f/sec\n", dt, 1.0/dt);
if (x < 0) {
if(errno != EAGAIN){
fprintf(stderr, "Error reading from socket %s\n", strerror(errno));
}
}
send_brk();
send_dmx();
}
the functions called are here:
send_brk() sends a serial break that lasts 88us
void send_brk() {
ioctl(fd, TIOCSBRK);
double start = now();
while((now() - start) < 0.000088){
}
ioctl(fd, TIOCCBRK);
}
send_dmx() writes a byte buffer of lenght 512 to the serial port
void send_dmx() {
// setmode(TX);
int n = write(fd, DMX, SIZE);
if(n == -1) {
if (errno != EAGAIN) {
fprintf(stderr, "Error writing to serial %s\n", strerror(errno));
exit(1);
}
}
if (n != SIZE) {
printf("couldn't write full frame =(\n");
}
}
and now() returns the current time in second as a float.
double now() {
struct timeval tv;
gettimeofday(&tv, NULL);
double time = tv.tv_sec;
time = time + (tv.tv_usec / 1000000.0);
return time;
}