LWIP hanging after some time due to memory not freeing - c

Question

OBS: I'm using a STM32F2 microcontroller, FreeRTOS and lwIP and I'm using the raw API
I have an application in which I'm listening to one PC and connecting to another. Basically everything works fine for a while when I am trying to achieve high throughput, but after about half an hour ... about 80~90k packets received it hangs. It actually varies a little bit where it hangs, but it stopped doing it when I started closing the connection whenever tcp_write returnd err_mem.
Sometimes it hangs in this line:
/* useg should point to last segment on unacked queue */
useg = pcb->unacked;
if (useg != NULL) {
for (; useg->next != NULL; useg = useg->next); <------- here
}
Sometimes when I call tcp_write it returns ERR_MEM and it never returns anything besides after ERR_MEM. This is how I send data, basically I accept a connection, recieve data, store the PCB, do something and then reply to that same PCB:
err_t ret;
ret = tcp_write(g_response[i].pcb, data, len, 1);
if(ret == ERR_OK)
tcp_output(g_response[i].pcb);
else
tcp_close(g_response[i].pcb);
Here is how I setup the socket to listen:
pcb = tcp_new();
tcp_bind(pcb, IP_ADDR_ANY, port);
pcb = tcp_listen(pcb);
pcb->so_options |= SOF_KEEPALIVE; // enable keep-alive
pcb->keep_intvl = 1000; // sends keep-alive every second
tcp_accept(pcb, accept);
And here are my callbacks to sent and rcv
static err_t rcv(void *arg, struct tcp_pcb *pcb, struct pbuf *p, err_t err) {
if(p == NULL) {
return ERR_OK;
} else if(err != ERR_OK) {
return err;
}
tcp_recved(pcb, p->len);
// do something
pbuf_free(p);
return ERR_OK;
}
int sentcounter = 0;
static err_t sent(void *arg, struct tcp_pcb *pcb, uint16_t len) {
sentcounter++;
return ERR_OK;
}
static err_t accept(void *arg, struct tcp_pcb *pcb, err_t err) {
int i;
tcp_arg(pcb, NULL);
/* Set up the various callback functions */
tcp_recv(pcb, rcv);
tcp_err(pcb, error);
tcp_sent(pcb, sent);
tcp_accepted(pcb);
}
The way I send data where I close the pcb whenever there isn ERR_MEM may be strange, but now I have fewer lost packets and it actually got me to exchange up to 90k packets, before that it was failing randomly.
I actually need a high throughput, that's why I'm calling tcp_output instead of letting the tcpip_thread deal with sending the packet. Whenever I let this thread take car of the packet eveything just works, but it's too slow (maybe one packet every 200~300 ms, and by calling tcp_output in the function I got to the point where I'm sending the data sub 10 ms ... I'm also not transfering big amounts of data, so that helps).
Recently I've noticed that the tcpip_thread calls the input function, goes to ipv4_input, goes to memp_free and keeps going but never leaves (I'm actually running a new test right now so later I'll update this question with the callstack to the input before it freezes).
Has somebody done anything similar? Bursts of small packets with high throughput?
EDIT: As promised, here is my call stack
osMutexWait() at cmsis_os.c:681 0x800474c sys_arch_protect() at
sys_arch.c:400 0x80146a6 do_memp_free_pool() at memp.c:415 0x800dca2
memp_free() at memp.c:486 0x800dcf8 tcp_seg_free() at tcp.c:1.336
0x800fb0e tcp_receive() at tcp_in.c:1.162 0x8011712 tcp_process() at
tcp_in.c:877 0x8011048 tcp_input() at tcp_in.c:367 0x8010692
ip4_input() at ip4.c:670 0x800c688 ethernet_input() at ethernet.c:176
0x80142fe tcpip_thread() at tcpip.c:124 0x8006836
pxPortInitialiseStack() at port.c:231 0x8004cd0
Just like #Lundin said, it is a concurrency problem. I should probably try to be more careful with how the functions are called. I'll try to modify my code to work with netconn or socket instead of tcp_pcb, and then measure the speeds I get. I really need high throughput