How to solve pthread_create error (11) resource temporarily unavailable? - c

Question

I'm building a project in c language (using openwrt as OS) to upload files to FTP server. and i'm using MQTT for the incoming data. So for each topic i subscribe to, i save this data and then upload it to the FTP server and to keep things going smoothly, each time i need to upload a file i just use a thread to do the job.
and to make sure the program doesn't run too much threads, each topic is allowed to create one thread. and i'm using a variable (like mutex but it's not pthread_mutex_t because i don't need to block a thread, i want to skip that step and upload the next file). i though with this technique i'm safe but after running the program for 15 min i get this error 11 which said resource temporarily unavailable when the program try to create a thread (pthread_create).
one of my tries to figure out what could be the problem is.
i used the pthread_join() function which is not an option in my condition but just to make sure that each thread is finished and not running in permanent loop. the program was running over than an hour and the error didn't show up again. and of course each thread was finished as intended.
and i'm 90% sure that each topic is creating only on thread and the next one will be create only if the previous one finished. (i was tracking the status of the variable before and after the thread creation).
i set the max thread from here "/proc/sys/kernel/threads-max" to 2000 (2000 is more than enough since i don't have too much topics)
upload function (this will create the thread):
void uploadFile(<args...>, bool* locker_p){
*locker_p = true;
args->uploadLocker_p = uploadLocker_p;
<do something here>
pthread_t tid;
int error = pthread_create(&tid, NULL, uploadFileThread, (void*)args);
if(0 != error){
printf("Couldn't run thread,(%d) => %s\n", error, strerror(error));
}
else{
printf("Thread %d\n", tid);
}
}
upload thread:
void *uploadFileThread(void *arg){
typeArgs* args = (typeArgs*)arg;
<do something like upload the file>
*(args->uploadLocker_p) = false;
free(args);
return NULL;
//pthread_exit(0);
}

Answer 1

The default stack size for the created threads is eating too much virtual memory.
Essentially, the kernel is telling your process that it has so much virtual memory already in use that it doesn't dare give it any more, because there isn't enough RAM and swap to back it up if the process were to suddenly use it all.
To fix, create an attribute that limits the per-thread stack to something sensible. If your threads do not use arrays as local variables, or do deep recursion, then 2*PTHREAD_STACK_MIN (from <limits.h>) is a good size.
The attribute is not consumed by the pthread_create() call, it is just a configuration block, and you can use the same one for any number of threads you create, or create a new one for each thread.
Example:
pthread_attr_t attrs;
pthread_t tid;
int err;
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, 2 * PTHREAD_STACK_MIN);
err = pthread_create(&tid, &attrs, uploadFileThread, (void *)args);
pthread_attr_destroy(&attrs);
if (err) {
/* Failed, errno in err; use strerror(err) */
} else {
/* Succeeded */
}
Also remember that if your uploadFileThread() allocates memory, it will not be freed automatically when the thread exits. It looks like OP already knows this (as they have the function free the argument structure when it's ready to exit), but I thought it a good idea to point it out.
Personally, I like to use a thread pool instead. The idea is that the upload workers are created beforehand, and they'll wait for a new job. Here is an example:
pthread_mutex_t workers_lock;
pthread_mutex_t workers_wait;
volatile struct work *workers_work;
volatile int workers_idle;
volatile sig_atomic_t workers_exit = 0;
where struct work is a singly-linked list protected by workers_lock, workers_idle is initialized to zero and incremented when waiting for new work, workers_wait is a condition variable signaled when new work arrives under workers_lock, and workers_exit is a counter that when nonzero, tells that many workers to exit.
A worker would be basically something along
void worker_do(struct work *job)
{
/* Whatever handling a struct job needs ... */
}
void *worker_function(void *payload __attribute__((unused)))
{
/* Grab the lock. */
pthread_mutex_lock(&workers_lock);
/* Job loop. */
while (!workers_exit) {
if (workers_work) {
/* Detach first work in chain. */
struct work *job = workers_work;
workers_work = job->next;
job->next = NULL;
/* Work is done without holding the mutex. */
pthread_mutex_unlock(&workers_lock);
worker_do(job);
pthread_mutex_lock(&workers_lock);
continue;
}
/* We're idle, holding the lock. Wait for new work. */
++workers_idle;
pthread_cond_wait(&workers_wait, &workers_lock);
--workers_idle;
}
/* This worker exits. */
--workers_exit;
pthread_mutex_unlock(&workers_lock);
return NULL;
}
The connection handling process can use idle_workers() to check the number of idle workers, and either grow the worker thread pool, or reject the connection as being too busy. The idle_workers() is something like
static inline int idle_workers(void)
{
int result;
pthread_mutex_lock(&workers_lock);
result = workers_idle;
pthread_mutex_unlock(&workers_lock);
return result;
}
Note that each worker only holds the lock for very short durations, so the idle_workers() call won't block for long. (pthread_cond_wait() atomically releases the lock when it starts waiting for a signal, and only returns after it re-acquires the lock.)
When waiting for a new connection in accept(), set the socket nonblocking and use poll() to wait for new connections. If the timeout passes, examine the number of workers, and reduce them if necessary by calling reduce_workers(1) or similar:
void reduce_workers(int number)
{
pthread_mutex_lock(&workers_lock);
if (workers_exit < number) {
workers_exit = number;
pthread_cond_broadcast(&workers_wait);
}
pthread_mutex_unlock(&workers_lock);
}
To avoid having to call pthread_join() for each thread – and we really don't even know which threads have exited here! – to reap/free the kernel and C library metadata related to the thread, the worker threads need to be detached. After creating a worker thread tid successfully, just call pthread_detach(tid);.
When a new connection arrives and it is determined to be one that should be delegated to the worker threads, you can, but do not have to, check the number of idle threads, create new worker threads, reject the upload, or just append the work to the queue, so that it will "eventually" be handled.