I'm having some trouble in my first foray into threads in C. I'm trying (for now) to write a very simple server program that accepts a socket connection and starts a new thread to process it. It seems to work fine except that it will only create about 300 threads (303, sometimes 304) before pthread_create() fails with the EAGAIN code, which means:
"The system lacked the necessary resources to create another thread, or the system-imposed limit on the total number of threads in a process {PTHREAD_THREADS_MAX} would be exceeded."
This is not 303 threads at the same time - each thread exits which is confirmed by gdb. Each time the process request function is called there are two threads running.
So it means "the system lacked the necessary resources". My question is (and it may be a bit stupid) - what are these resources? Presumably it's a memory leak in my program (certainly possible, likely even), but I'd have thought that even so it could manage more than 300 considering the rest of the program does very little.
How can I find out how much memory my program has available to confirm that it's running out of it? There's plenty of memory and swap free so presumably there's an artificial limit imposed by the OS (Linux).
Thanks
If you are not creating the thread with the attribute PTHREAD_CREATE_DETACHED (or detaching them with pthread_detach(), you may need to call pthread_join() on each created thread after it exits to free up the resources associated with it.
Possibly a little overkill(?) but Valgrind can help you locate memleaks in Linux.
Could you perhaps post some code snippets? Preferably the parts where you allocate/free memory/sockets and where you create your threads.
Related
I have a little paging problem on my realtime system, and wanted to know how exactly linux should behave in my particular case.
Among various other things, my application spawns 2 threads using pthread_create(), which operate on a set of shared buffers.
The first thread, let's call it A, reads data from a device, performs some calculations on it, and writes the results into one of the buffers.
Once that buffer is full, thread B will read all the results and send them to a PC via ethernet, while thread A writes into the next buffer.
I have noticed that each time thread A starts writing into a previously unused buffer, i miss some interrupts and lose data (there is an id in the header of each packet, and if that increments by more than one, i have missed interrupts).
So if i use n buffers, i get exactly n bursts of missed interrupts at the start of my data acquisition (therefore the problem is definitely caused by paging).
To fix this, i used mlock() and memset() on all of the buffers to make sure they are actually paged in.
This fixed my problem, but i was wondering where in my code would be the correct place do this. In my main application, or in one/both of the threads? (currently i do it in both threads)
According to the libc documentation (section 3.4.2 "Locked Memory Details"), memory locks are not inherited by child processes created using fork().
So what about pthreads? Do they behave the same way, or would they inherit those locks from my main process?
Some background information about my system, even though i don't think it matters in this particular case:
It is an embedded system powered by a SoC with a dual-core Cortex-A9 running Linux 4.1.22 with PREEMPT_RT.
The interrupt frequency is 4kHz
The thread priorities (as shown in htop) are -99 for the interrupt, -98 for thread A (both of which are higher than the standard priority of -51 for all other interrupts) and -2 for thread B
EDIT:
I have done some additional tests, calling my page locking function from different threads (and in main).
If i lock the pages in main(), and then try to lock them again in one of the threads, i would expect to see a large amount of page faults for main() but no page faults for the thread itself (because the pages should already be locked). However, htop tells a different story: i see a large amount of page faults (MINFLT column) for each and every thread that locks those pages.
To me, that would suggest that pthreads actually do have the same limitation as child processes spawned using fork(). And if this is the case, locking them in both threads (but not in main) would be the correct procedure.
Threads share the same memory management context. If a page is resident for one thread, it's resident for all threads in the same process.
The implication of this is that memory locking is per-process, not per-thread.
You are probably still seeing minor faults on the first write because a fault is used to mark the page dirty. You can avoid this by also writing to each page after locking.
I have a Linux C program that runs well in a Raspberry 3. When I run it in a low memory situation in another sbc (Raspberry Zero) it runs about 2-3 days then freezes. I believe it's a stack overflow situation.
I've put a thread to check periodically when the main program has frozen. Unfortunately it appears that if the main process crashes, it takes down all of the other threads in the process.
I can avoid this by having another process checking upon the first process, but I'd prefer a thread. Is it possible to have thread that is safe and does not freeze it the main process freezes?
Easily no, it's not possible because per thread definition they share memory and they are part of the main process and it own them all. So everything afflict the main process afflict all the threads.
I have a multi-threaded application in a POSIX/Linux environment - I have no control over the code that creates the pthreads. At some point the process - owner of the pthreads - receives a signal.
The handler of that signal should abort,cancel or stop all the pthreads and log how many pthreads where running.
My problem is that I could not find how to list all the pthreads running in process.
There doesn't seem to be any portable way to enumerate the threads in a process.
Linux has pthread_kill_other_threads_np, which looks like a leftover from the original purely-userland pthreads implementation that may or may not work as documented today. It doesn't tell you how many threads there were.
You can get a lot of information about your process by looking in /proc/self (or, for other processes, /proc/123). Although many unices have a file or directory with that name, the layout is completely different, so any code using /proc will be Linux-specific. The documentation of /proc is in Documentation/filesystems/proc.txt in the kernel source. In particular, /proc/self/task has a subdirectory for each thread. The name of the subdirectory is the LWP id; unfortunately, [1][2][3] there doesn't seem to be a way to associate LWP ids with pthread ids (but you can get your own thread id with gettid(2) if you work for it). Of course, reading /proc/self/task is not atomic; the number of threads is available atomically through /proc/self/status (but of course it might change before you act on it).
If you can't achieve what you want with the limited support you get from Linux pthreads, another tactic is to play dynamic linking tricks to provide your own version of pthread_create that logs to a data structure you can inspect afterwards.
You could wrap ps -eLF (or another command that more closely reads just the process you're interested in) and read the NLWP column to find out how many threads are running.
Given that the threads are in your process, they should be under your control. You can record all of them in a data structure and keep track.
However, doing this won't be race-condition free unless it's appropriately managed (or you only ever create and join threads from one thread).
Any threads created by libraries you use are their business and you should not be messing with them directory, or the library may break.
If you are planning to exit the process of course, you can just leave the threads running anyway, as calling exit() terminates them all.
Remember that a robust application should be crash-safe anyway, so you should not depend upon shutdown behaviour to avoid data loss etc.
Suppose a thread A creates a thread B and after a duration the thread B crashes with an issue, Is there any possibility that the control moves back to the thread A in C language.
Sort of an exceptional handling.
No. "Control passes back" doesn't make a lot of sense at all, since they are executing independently anyway -- usually, Thread A isn't going to sit around waiting for Thread B to finish, but it will be doing something else.
Incidentally, threads can, of course, check whether another thread is still running. Check your thread library or the system functions that you are using.
However, that will only work for something one could call a "soft crash"; a lot of crashes screw up a lot more than just the thread doing the bad thing, such as hardware exceptions that kill the entire process, or corrupting memory. So, trying to catch crashes in another thread is going to be a good amount of work with little benefit, if any at all. Better spend that time fixing the crashes.
No. They're separate threads of execution. Once thread A has created and started thread B, both A and B can execute independently.
Of course if thread B crashes the whole process, thread A won't exist any more...
Threads cannot call other threads, only signal them. The 'normal' function/method call/return mechanism is stack-based and each thread has its own stack, (it is very common for several threads to run exactly the same code using different stack auto-variables).
If a thread cannot call another thread, then there is no 'return' from one thread to another either.
I have a multi-threaded application in a POSIX/Linux environment - I have no control over the code that creates the pthreads. At some point the process - owner of the pthreads - receives a signal.
The handler of that signal should abort,cancel or stop all the pthreads and log how many pthreads where running.
My problem is that I could not find how to list all the pthreads running in process.
There doesn't seem to be any portable way to enumerate the threads in a process.
Linux has pthread_kill_other_threads_np, which looks like a leftover from the original purely-userland pthreads implementation that may or may not work as documented today. It doesn't tell you how many threads there were.
You can get a lot of information about your process by looking in /proc/self (or, for other processes, /proc/123). Although many unices have a file or directory with that name, the layout is completely different, so any code using /proc will be Linux-specific. The documentation of /proc is in Documentation/filesystems/proc.txt in the kernel source. In particular, /proc/self/task has a subdirectory for each thread. The name of the subdirectory is the LWP id; unfortunately, [1][2][3] there doesn't seem to be a way to associate LWP ids with pthread ids (but you can get your own thread id with gettid(2) if you work for it). Of course, reading /proc/self/task is not atomic; the number of threads is available atomically through /proc/self/status (but of course it might change before you act on it).
If you can't achieve what you want with the limited support you get from Linux pthreads, another tactic is to play dynamic linking tricks to provide your own version of pthread_create that logs to a data structure you can inspect afterwards.
You could wrap ps -eLF (or another command that more closely reads just the process you're interested in) and read the NLWP column to find out how many threads are running.
Given that the threads are in your process, they should be under your control. You can record all of them in a data structure and keep track.
However, doing this won't be race-condition free unless it's appropriately managed (or you only ever create and join threads from one thread).
Any threads created by libraries you use are their business and you should not be messing with them directory, or the library may break.
If you are planning to exit the process of course, you can just leave the threads running anyway, as calling exit() terminates them all.
Remember that a robust application should be crash-safe anyway, so you should not depend upon shutdown behaviour to avoid data loss etc.