Out of memory in a thread pool in C, Linux - c

Question

I need to create infinite loop and with a thread pool create for example 200 threads to do the job from infinite loop.
I'm using this thread pool - https://github.com/Pithikos/C-Thread-Pool
In the same time I'm monitoring the server resources (with htop) and see that memory is increasing on 3 megabytes every second until the kernel kills the application.
The code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include "thpool.h"
#define MAX_IPv4 256
/* Args for thread start function */
typedef struct {
int octet1;
int octet2;
int octet3;
int octet4;
} args_struct;
/* Thread task */
void task1(void *args) {
args_struct *actual_args = args;
printf("%d.%d.%d.%d\n", actual_args->octet1, actual_args->octet2, actual_args->octet3, actual_args->octet4);
/* Do some job */
sleep(1);
/* Free the args */
free(args);
}
/* Main function */
int main( void ) {
int i=0, j=0, n=0, m=0;
/* Making threadpool n threads */
threadpool thpool = thpool_init(200);
/* Infinite loop start from the certain ip*/
while (1) {
for (i=0; i < MAX_IPv4; ++i) {
for (j=0; j < MAX_IPv4; ++j) {
for (n=0; n < MAX_IPv4; ++n) {
for (m=0; m < MAX_IPv4; ++m) {
/* Heap memory for the args different for the every thread */
args_struct *args = malloc(sizeof *args);
args->octet1 = i;
args->octet2 = j;
args->octet3 = n;
args->octet4 = m;
/* Create thread */
thpool_add_work(thpool, (void*)task1, (void*)args);
}
}
}
}
/* Start from 0.0.0.0 */
i=0;
j=0;
n=0;
m=0;
}
/* Wait until the all threads are done */
thpool_wait(thpool);
/* Destroy the threadpool */
thpool_destroy(thpool);
return 0;
}
How to solve this issue?

Answer 1

Looking at issues for your library ( especially this one about memory consumption ).
There is a recommendation to check the job queue length threadpool.jobqueue.len;
Maybe your code could check after adding your job to the queue
Unfortunately the threadpool is an opaque pointer and you could not access the value directly.
I would recommend adding a function for the threadpool in thpool.c :
int thpool_jobqueue_length(thpool_* thpool_p) {
return thpool->jobqueue->len;
}
And don't forget the declaration in thpool.h
int thpool_jobqueue_length(threadpool);
Then modify your code
const int SOME_ARBITRARY_VALUE = 400
...
thpool_add_work(thpool, (void*)task1, (void*)args);
while( ( thpool_jobqueue_length(thpool) > SOME_ARBITRARY_VALUE ) ) {
sleep(1);
}
...

Answer 2

Looking at the code for thpool_add_work there is some memory use per call (allocating a job record to add to a queue), so as your loop runs forever, it is not surprising that it will run out of memory at some point. You are also allocating memory inside your innermost loop, so that too will help use up all your memory.
Essentially inside your inner loop you are allocating 16 bytes (assuming int is 4) for the args_struct, and thpool_add_work is also allocating 12 bytes (possibly rounded to 16 for alignment purposes).
As you can imagine, that adds up to a lot for your 4 nested loops (which are also run infinitely).