main doesn't continue after call pthread_join function - c

I'm new to pthread and multithreading, i have written a code like that.
#include <pthread.h>
#include <unistd.h>
void *nfc_read(void *arg)
{
int fd = -1;
int ret;
uint8_t read_data[24];
while(1){
ret = read_block(fd, 8, read_data);
if(!ret){
return (void)read_data;
}
}
}
int main(int argc, char *argv[])
{
pthread_t my_thread;
void *returnValue;
pthread_create(&my_thread, NULL, nfc_read, NULL);
pthread_join(my_thread, &returnValue);
printf("NFC card value is : %s \n", (char)returnValue);
printf("other process");
return 0;
}
Until the return value from nfc_read function, as I will have other code I need to run and I don't want to block in main. How can i do that?

This is a sample where a read thread runs concurrently to the "main" thread which is doing other work (in this case, printing dots and sleeping).
To keep things simple, a simulated the reading from an input device with copying a constant string character by character. I guess, this is reasonable as the synchronization of threads is focused.
For the synchronization of threads, I used atomic_bool with atomic_store() and atomic_load() which are provided by the Atomic Library (since C11).
My sample application test-concurrent-read.c:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <stdatomic.h>
#include <unistd.h>
/* sampe input */
const char sampleInput[]
= "This is sample input which is consumed as if it was read from"
" a (very slow) external device.";
atomic_bool readDone = ATOMIC_VAR_INIT(0);
void* threadRead(void *pArg)
{
char **pPBuffer = (char**)pArg;
size_t len = 0, size = 0;
int c; const char *pRead;
for (pRead = sampleInput; (c = *pRead++) > 0; sleep(1)) {
if (len + 1 >= size) {
if (!(*pPBuffer = realloc(*pPBuffer, (size + 64) * sizeof(char)))) {
fprintf(stderr, "ERROR! Allocation failed!\n");
break;
}
size += 64;
}
(*pPBuffer)[len++] = c; (*pPBuffer)[len] = '\0';
}
atomic_store(&readDone, 1);
return NULL;
}
int main()
{
/* start thread to read concurrently */
printf("Starting thread...\n");
pthread_t idThreadRead; /* thread ID for read thread */
char *pBuffer = NULL; /* pointer to return buffer from thread */
if (pthread_create(&idThreadRead, NULL, &threadRead, &pBuffer)) {
fprintf(stderr, "ERROR: Failed to start read thread!\n");
return -1;
}
/* start main loop */
printf("Starting main loop...\n");
do {
putchar('.'); fflush(stdout);
sleep(1);
} while (!atomic_load(&readDone));
putchar('\n');
void *ret;
pthread_join(idThreadRead, &ret);
/* after sync */
printf("\nReceived: '%s'\n", pBuffer ? pBuffer : "<NULL>");
free(pBuffer);
/* done */
return 0;
}
Compiled and tested with gcc in cygwin on Windows 10 (64 bit):
$ gcc -std=c11 -pthread -o test-concurrent-read test-concurrent-read.c
$ ./test-concurrent-read
Starting thread...
Starting main loop...
.............................................................................................
Received: 'This is sample input which is consumed as if it was read from a (very slow) external device.'
$
I guess, it is worth to mention why there is no mutex guarding for pBuffer which is used in main() as well as in threadRead().
pBuffer is initialized in main() before pthread_create() is called.
While thread_read() is running, pBuffer is used by it exclusively (via its passed address in pPBuffer).
It is accessed in main() again but not before pthread_join() which grants that threadRead() has ended.
I tried to find a reference by google to confirm that this procedure is well-defined and reasonable. The best, I could find was SO: pthread_create(3) and memory synchronization guarantee in SMP architectures which cites The Open Group Base Specifications Issue 7 - 4.12 Memory Synchronization.

Related

Failing to execute "make" in Ubuntu gcc

I have several files which I'm trying to compile using gcc in Ubuntu. I have vmware machine 16.04 ubuntu, which 64-bit computer. I have added the flags for 32bit compilation in the makefile. (-m32 flag).
This is the makefile:
all: clean binsem.a ut.a ph
FLAGS = -Wall -L./ -m32
ph: ph.c
gcc ${FLAGS} ph.c -lbinsem -lut -o ph
binsem.a:
gcc $(FLAGS) -c binsem.c
ar rcu libbinsem.a binsem.o
ranlib libbinsem.a
ut.a:
gcc $(FLAGS) -c ut.c
ar rcu libut.a ut.o
ranlib libut.a
clean:
rm -f *.o
rm -f a.out
rm -f *~
rm -f ph
rm -f *a
My attempts to compile the code were not successful, as I received the following error:
ph.c:126:12: warning: ‘main’ defined but not used [-Wunused-function]
/usr/lib/gcc/i686-linux-gnu/5/../../../i386-linux-gnu/crt1.o: In function `_start':
(.text+0x18): undefined reference to `main'
collect2: error: ld returned 1 exit status
Makefile:6: recipe for target 'ph' failed
make: *** [ph] Error 1
**I tried - **
Adding -c flags to the ph.c file in the make file. However it doesn't make much sense, since it has a main function. No success.
Adding sum -m32 flags in the makefile, in addition to what's already written. No Success.
The file "ph.c", is the following file -
#include <stdio.h>
#include <setjmp.h>
#include <signal.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <time.h>
#include <stdint.h>
#include <sys/time.h>
#include <inttypes.h>
#include "binsem.h"
#include "ut.h"
#define LEFT (i+N-1)%N
#define RIGHT (i+1)%N
#define THINKING 0
#define HUNGRY 1
#define EATING 2
int N;
volatile int *phil_state;
sem_t *s;
sem_t mutex;
int *tid;
uint64_t get_wall_time() {
struct timeval time;
gettimeofday(&time, NULL);
uint64_t millis = (time.tv_sec * (uint64_t)1000) + (time.tv_usec / 1000);
return millis;
}
void think(int p) {
int i, factor;
volatile int j;
printf("Philosopher (%d) - time %" PRId64 " - is thinking\n",p, get_wall_time()); fflush (stdout);
factor = 1 + random()%5;
for (i = 0; i < 100000000*factor; i++){
j += (int) i*i;
}
printf("Philosopher (%d) - time %" PRId64 " - is hungry\n", p, get_wall_time()); fflush (stdout);
}
void eat(int p){
int i, factor;
volatile int j;
printf("Philosopher (%d) - time %" PRId64 " - is eating\n",p, get_wall_time()); fflush (stdout);
factor = 1 + random()%5;
for (i = 0; i < 100000000*factor; i++){
j += (int) i*i;
}
//printf("Philosopher (%d) - time %" PRId64 " - is thinking\n",p, get_wall_time()); fflush (stdout);
}
void test(int i){
if (phil_state[i] == HUNGRY &&
phil_state[LEFT] != EATING &&
phil_state[RIGHT] != EATING){
phil_state[i] = EATING;
binsem_up(&(s[i]));
}
}
void take_forks(int i){
binsem_down(&mutex);
phil_state[i] = HUNGRY;
test(i);
binsem_up(&mutex);
binsem_down(&(s[i]));
}
void put_forks(int i){
binsem_down(&mutex);
phil_state[i] = THINKING;
test(LEFT);
test(RIGHT);
binsem_up(&mutex);
}
void int_handler(int signo) {
long int duration;
int i;
for (i = 0; i < N; i++) {
duration = ut_get_vtime(tid[i]);
printf("Philosopher (%d) used the CPU %ld.%ld sec.\n",
i+1,duration/1000,duration%1000);
}
exit(0);
}
void philosopher(int i){
while (1){
think(i);
take_forks(i);
eat(i);
put_forks(i);
}
}
static int main(int argc, char *argv[])
{
int c;
if (argc != 2){
printf("Usage: %s N\n", argv[0]);
exit(1);
}
N = atoi(argv[1]);
if (N < 2){
printf("Usage: %s N (N >=2)\n", argv[0]);
exit(1);
}
ut_init(N);
s = (sem_t *)malloc (N * sizeof(sem_t));
phil_state = (int *) malloc (N * sizeof(int));
tid = (int *) malloc (N * sizeof(int));
for (c = 0; c < N ; c++){
phil_state[c] = THINKING;
binsem_init(&(s[c]), 0);
}
for (c = 0; c < N ; c++){
tid[c] = ut_spawn_thread(philosopher,c);
printf("Spawned thread #%d\n", tid[c]);
}
binsem_init(&mutex, 1);
signal(SIGINT,int_handler);
ut_start();
return 0; // avoid warnings
}
The makefile also tries to compile these files - ut.c, binsem.c
#include <stdlib.h>
#ifndef _UT_H
#define _UT_H
#include "ut.h"
#include <sys/time.h> // for itimerval
#include <unistd.h> // for alarm
#include <stdlib.h> // for malloc
#include <stdio.h> // for perror
#include <ucontext.h>
# include "ut.h"
#define MAX_TAB_SIZE 128 // the maximal threads table size.
#define MIN_TAB_SIZE 2 // the minimal threads table size.
#define SYS_ERR -1 // system-related failure code
#define TAB_FULL -2 // full threads table failure code
/*This type defines a single slot (entry) in the threads table. Each slot describes a single
thread. Note that we don't need to keep the thread state since every thread is always ready
or running. We also don't have to support adding/stopping thread dynamically, so we also don't
have to manage free slots.*/
typedef struct ut_slot {
ucontext_t uc;
unsigned long vtime; /* the CPU time (in milliseconds) consumed by this thread.*/
void (*func)(int); /* the function executed by the thread.*/
int arg; /* the function argument.*/
} ut_slot_t, *ut_slot;
static ut_slot threads; // pointer to thread table
static volatile int numThreads = 0; // number of threads in the table
static volatile int currentThread = 0; // current thread
static ucontext_t mainThread;
#define STACKSIZE 8192 // the thread stack size.
/* The TID (thread ID) type. TID of a thread is actually the index of the thread in the
threads table. */
typedef short int tid_t;
void handler(int signal){
alarm(1); //the alarm every second as demanded in the assignment
currentThread = (currentThread +1 ) % numThreads;
printf("in signal handler: switching from %d to %d\n", currentThread, currentThread - 1);
swapcontext(&threads[currentThread].uc, &threads[currentThread].uc); /*save current thread,
* load next thread*/
if (signal == SIGVTALRM){ // increment the time stats
threads[currentThread].vtime+=100;
} else if(signal==SIGALRM) {
if (swapcontext(&threads[currentThread - 1].uc, &threads[currentThread].uc) == -1) {
perror("could not do swapping");
exit(1);
}
}
}
int ut_init(int tab_size){
/// (###)
if(tab_size < MAX_TAB_SIZE) {
threads = (ut_slot) malloc(tab_size * sizeof(int(ut_slot_t)));
}
else{
threads = (ut_slot) malloc(MAX_TAB_SIZE * sizeof(int(ut_slot_t)));
}
if (!threads) {
return SYS_ERR;
}
return 0;
}
tid_t ut_spawn_thread(void (*func)(int), int arg){
/*uc[1].uc_link = &uc[0];
uc[1].uc_stack.ss_sp = st1; //stack fro thread 1
uc[1].uc_stack.ss_size = sizeof st1; //size of stack for therad
makecontext(&uc[1], (void(*)(void)) f, 1, 1); */
int thread_stack_size = STACKSIZE/8; //no need for 8K in size
if (numThreads>=TAB_FULL){ //(*)
return TAB_FULL;
}
if (getcontext(&threads[numThreads].uc)==-1){
return SYS_ERR;
}
ut_slot_t newThread;
threads[numThreads] = newThread;
threads[numThreads].uc.uc_link = &mainThread;
threads[numThreads].uc.uc_stack.ss_sp = (void* *)malloc(thread_stack_size);
if(threads[numThreads].uc.uc_stack.ss_sp==NULL){
return SYS_ERR;
}
makecontext(&threads[numThreads].uc,(void(*)(void))func,1,arg);
numThreads++;
return numThreads - 1;
}
int ut_start(void){
struct sigaction sigaction1;
int firstThread = 0; /*represents the current thread*/
struct itimerval itv;
/* set up vtimer for accounting */
itv.it_interval.tv_sec = 0;
itv.it_interval.tv_usec = 10000;
itv.it_value = itv.it_interval;
/* Initialize the data structures for SIGALRM handling. */
sigaction1.sa_flags = SA_RESTART; //restart instead of throwing exception
sigfillset(&sigaction1.sa_mask); // don't throw exception for additional signals
sigaction1.sa_handler = handler; // specify handler for the sigaction declared
if (sigaction(SIGVTALRM, &sigaction1, NULL) < 0)
return SYS_ERR;
/*START THE TIMER */
if (setitimer(ITIMER_VIRTUAL, &itv, NULL) < 0)
return SYS_ERR;
if (sigaction(SIGINT, &sigaction1, NULL) < 0)
return SYS_ERR;
/* Start running. */
alarm(1); //alarm every second
if(swapcontext(&threads[firstThread].uc,&threads[firstThread+1].uc)==-1){ //swap first time
return SYS_ERR;
}
return -1;
}
unsigned long ut_get_vtime(tid_t tid){
return threads[tid].vtime;
}
#endif
binsem.c
#ifndef _BIN_SEM_H
#define _BIN_SEM_H
#include "binsem.h"
#include <signal.h>
#include "atomic.h"
typedef unsigned long sem_t;
void binsem_init(sem_t *s, int init_val){
if(init_val == 1){
*s = 1;
}
else{
*s = 0;
}
}
void binsem_up(sem_t *s){
xchg(s,1); /*send the pointer of s, and the new value 1*/
}
int binsem_down(sem_t *s){
int flag = 0;
while (flag == 0){
xchg(s,0); /*try changing the value - down*/
if (*s != 0){ /*no success in changing*/
if (raise(SIGALRM)!=0){ /*raise returns 0 on success*/
return -1;
}
} else{
flag = 1;
return 0;
}
}
}
#endif
Any ideas?
You declare the main function to be static. That means it will have internal linkage, and not be exported. Therefore the linker will not be able to find it.
Remove the static modifier and that should work.
However there are also many other problems in your Makefile that will most likely lead to problems:
The target ph only depends on ph.c, not the libraries you want to link with. You need to add the libraries as dependencies as well (their full file-names).
You should also make the libraries depend on the object files used to create them, and rely on the implicit rules to build object files from source files.
Also note that for a library named A then the file-name needs to be libA.a, otherwise the linker will not be able to find it with the -l (lower-case L) option.
The error message is pretty clear:
(.text+0x18): undefined reference to `main'
In ph.c you have (probably accustomed from Java?):
static int main(int argc, char *argv[])
In C:
static ([CPPReference]: C keywords: static) means internal linkage (only valid in current translation unit (ph.o in our case)), so a static symbol won't be "seen" by the linker (loader)
Code linked to an application (not a dynamic library or shared object (.so: linked with -shared)), must explicitly define main, and that should be visible by the linker
The combination of the above 2, lead to what you're experiencing.
Remove the static and you'll be fine.
Note: There might be other errors in the Makefile.

Understanding Context Switches in Xenomai-Linux POSIX skin

I'm running a RT program on BeagleBone Black with Xenomai and trying to figure how to monitor/understand context switches (I know the concept of context switches) so that I can determine when my program (in C using POSIX skin) switches from primary and secondary mode.
Here's the program main_posix.c
#ifndef __XENO_SIM__
#ifndef __KERNEL__
#include <stdio.h>
#define xnarch_printf printf
#endif
#include <time.h>
#include <errno.h>
#include <unistd.h>
#include <signal.h>
#include <sys/mman.h>
#include <pthread.h>
#include <mqueue.h>
#else /* __XENO_SIM */
#include <posix/posix.h>
#endif /* __XENO_SIM */
void warn_upon_switch(){
printf("Switched to Secondary Mode \n");
}
void *threadFunc(void *arg)
{
char *str;
int i = 0;
struct timespec delay, sleep;
unsigned long over;
int ret;
str=(char*)arg;
printf("In thread \n");
sleep.tv_sec = 1;
sleep.tv_nsec = 0;
#ifdef __XENO__
ret = pthread_set_mode_np(0, 0x00040000);
printf("Warn Bit Ret %d\n", ret);
#endif /* __XENO__ */
// run this for some arbitrary time
while(i < 110000000 )
{
clock_nanosleep(CLOCK_REALTIME, 0, &sleep, NULL);
printf("threadFunc says: %s\n",str);
++i;
}
return NULL;
}
int main(void)
{
signal(SIGXCPU, warn_upon_switch);
signal(SIGKILL, warn_upon_switch);
pthread_t pth;
double i = 0;
int ret;
pthread_attr_t tattr;
struct sched_param sparam;
sparam.sched_priority = 99;
ret = pthread_attr_init(&tattr);
printf("Init Return Val %d\n", ret);
ret = pthread_setschedparam(pth,SCHED_FIFO, &sparam);
printf("SetSchedParam Ret Value %d\n", ret);
pthread_create(&pth,&tattr,threadFunc,"foo");
printf("main waiting for thread to terminate...\n");
pthread_join(pth,NULL);
return 0;
}
I'm also monitoring /proc/xenomai/stat continuously through watch
I see that CSWand MSW for PID 3323 changes continuously.
Here's the output of ps -e -o class,rtprio,pri,nice,cmd | grep ./main_posix
The output is as follows
My questions are as follows
How do I know if my program is running in primary or secondary mode?
I get the return value of ret = pthread_setschedparam(pth,SCHED_FIFO, &sparam); as 16 which is EBUSY. Any idea why?
Tried catching the switch signal using signal(SIGXCPU, warn_upon_switch);. The function never gets called.
If the program can be seen in Linux (meaning it gets a PID through the Linux kernel), does it mean its running in secondary mode?
In proc/xenomai/stat, I see two processes for the same program. Is it the main and the thread?
Here are some resources I used
Periodic thread fails real-time in Xenomai
Xenomai clock_nanosleep in POSIX skin jumps to Linux Kernel
http://xenomai.org/2014/08/porting-a-linux-application-to-xenomai-dual-kernel/#Using_the_PTHREAD_WARNSW_bit
http://www.xenomai.org/documentation/xenomai-2.6/html/api/sigxcpu_8c-example.html

Does QEMU user mode emulation exit in a way that would prevent pthread_join from blocking?

I'm trying to run QEMU's user mode emulator as a thread in a larger program that I'm writing. I've modified the linux-user/main.c file so that the standard int main(int argc, char **argv, char **envp function is now called void *qemu_user_mode_func(void *arg). I've also added pthread_exit(NULL) to the end of that function, as is standard practice for pthreads (or so I've been told).
However, when I try to run a second thread that contains my own test function (shown below in void *test_func(void *arg)), the process exits before the second thread completes, even with a call to pthread_join(tid), which I've read blocks the calling thread until thread tid returns. Does QEMU's user mode emulation exit in such a way that would prevent pthread_join from exiting, or am I just using threads wrong?
Here's my code (not including the bulk of qemu_user_mode_func):
void *qemu_user_mode_func(void *arg)
{
thread_data_t *thread_data;
int argc;
char **argv;
char **envp;
/** QEMU's normal code **/
//return 0;
pthread_exit(NULL);
}
void *test_func(void *arg) {
struct timespec time;
time.tv_sec = 7;
time.tv_nsec = 0;
nanosleep(&time, NULL);
printf("hello, world - from a thread\n");
pthread_exit(NULL);
}
int main(int argc, char**argv, char **envp) {
//Initialize variables to create thread
int rc;
pthread_t threads[2];
thread_data_t main_args;
main_args.tid = 1;
main_args.argc = argc;
main_args.argv = argv;
main_args.envp = envp;
//Create thread
if ((rc = pthread_create(&(threads[0]), NULL, test_func, NULL))) {
fprintf(stderr, "error: pthread_create, rc: %d\n", rc);
return EXIT_FAILURE;
}
if ((rc = pthread_create(&(threads[1]), NULL, qemu_user_mode_func, (void *)&main_args))) {
fprintf(stderr, "error: pthread_create, rc: %d\n", rc);
return EXIT_FAILURE;
}
//Wait for thread to finish, then terminate process
for (rc = 0; rc < 2; rc++) {
pthread_join(threads[rc], NULL);
}
return 0;
}
EDIT: I've discovered in the void cpu_loop(CPUX86State *env) function that when the emulated program reaches its conclusion, QEMU calls syscall 231, which is sys_exit_group (as per 1). So I'm guessing this syscall is terminating the entire process that I'm running. I'd appreciate any tips on how to get around that!
If you turn a complicated preexisting application into thread there are going to be issues. One is that the application can call exit or its variants which will terminate your entire program. There are numerous other issues that could be causing a problem. I would suggest using gdb to determine what is making your program exit.
Problem was solved by editing the following section in void cpu_loop(CPUX86State *env). I capture either sys_exit_group and sys_exit system calls before they are executed, and just return from the function instead.
Original:
void cpu_loop(CPUX86State *env)
{
CPUState *cs = CPU(x86_env_get_cpu(env));
int trapnr;
abi_ulong pc;
target_siginfo_t info;
for(;;) {
cpu_exec_start(cs);
trapnr = cpu_x86_exec(env);
cpu_exec_end(cs);
switch(trapnr) {
case 0x80:
/* linux syscall from int $0x80 */
env->regs[R_EAX] = do_syscall(env,
env->regs[R_EAX],
env->regs[R_EBX],
env->regs[R_ECX],
env->regs[R_EDX],
env->regs[R_ESI],
env->regs[R_EDI],
env->regs[R_EBP],
0, 0);
break;
#ifndef TARGET_ABI32
case EXCP_SYSCALL:
/* linux syscall from syscall instruction */
env->regs[R_EAX] = do_syscall(env,
env->regs[R_EAX],
env->regs[R_EDI],
env->regs[R_ESI],
env->regs[R_EDX],
env->regs[10],
env->regs[8],
env->regs[9],
0, 0);
break;
#endif
Modified:
void cpu_loop(CPUX86State *env)
{
CPUState *cs = CPU(x86_env_get_cpu(env));
int trapnr;
abi_ulong pc;
target_siginfo_t info;
for(;;) {
cpu_exec_start(cs);
trapnr = cpu_x86_exec(env);
cpu_exec_end(cs);
switch(trapnr) {
case 0x80:
/* linux syscall from int $0x80 */
env->regs[R_EAX] = do_syscall(env,
env->regs[R_EAX],
env->regs[R_EBX],
env->regs[R_ECX],
env->regs[R_EDX],
env->regs[R_ESI],
env->regs[R_EDI],
env->regs[R_EBP],
0, 0);
break;
#ifndef TARGET_ABI32
case EXCP_SYSCALL:
/* linux syscall from syscall instruction */
----> if ((env->regs[R_EAX] == __NR_exit_group) || (env->regs[R_EAX] == __NR_exit)) {
return;
}
env->regs[R_EAX] = do_syscall(env,
env->regs[R_EAX],
env->regs[R_EDI],
env->regs[R_ESI],
env->regs[R_EDX],
env->regs[10],
env->regs[8],
env->regs[9],
0, 0);
break;
#endif

Simple C pthread test program hangs during execution

I'm new to using the pthread library in C and I have an assignment for my class to write a simple program using them. The basic description of the program is it takes 1 or more input files containing website names and 1 output file name. I then need to create 1 thread per input file to read in the website names and push them onto a queue. Then I need to create a couple of threads to pull those names off of the queue, find their IP Address, and then write that information out to the output file. The command line arguments are expected as follows:
./multi-lookup [one or more input files] [single output file name]
My issue is this. Whenever I run the program with only 1 thread to push information to the output file then everything works properly. When I make it two threads then the program hangs and none of my testing "printf" statements are even printed. My best guess is that deadlock is occurring somehow and that I'm not using my mutexes properly but I can't figure out how to fix it. Please help!
If you need any information that I'm not providing then just let me know. Sorry for the lack of comments in the code.
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <pthread.h>
#include "util.h"
#include "queue.h"
#define STRING_SIZE 1025
#define INPUTFS "%1024s"
#define USAGE "<inputFilePath> <outputFilePath>"
#define NUM_RESOLVERS 2
queue q;
pthread_mutex_t locks[2];
int requestors_finished;
void* requestors(void* input_file);
void* resolvers(void* output_file);
int main(int argc, char* argv[])
{
FILE* inputfp = NULL;
FILE* outputfp = NULL;
char errorstr[STRING_SIZE];
pthread_t requestor_threads[argc - 2];
pthread_t resolver_threads[NUM_RESOLVERS];
int return_code;
requestors_finished = 0;
if(queue_init(&q, 10) == QUEUE_FAILURE)
fprintf(stderr, "Error: queue_init failed!\n");
if(argc < 3)
{
fprintf(stderr, "Not enough arguments: %d\n", (argc - 1));
fprintf(stderr, "Usage:\n %s %s\n", argv[0], USAGE);
return 1;
}
pthread_mutex_init(&locks[0], NULL);
pthread_mutex_init(&locks[1], NULL);
int i;
for(i = 0; i < (argc - 2); i++)
{
inputfp = fopen(argv[i+1], "r");
if(!inputfp)
{
sprintf(errorstr, "Error Opening Input File: %s", argv[i]);
perror(errorstr);
break;
}
return_code = pthread_create(&(requestor_threads[i]), NULL, requestors, inputfp);
if(return_code)
{
printf("ERROR: return code from pthread_create() is %d\n", return_code);
exit(1);
}
}
outputfp = fopen(argv[i+1], "w");
if(!outputfp)
{
sprintf(errorstr, "Errord opening Output File: %s", argv[i+1]);
perror(errorstr);
exit(1);
}
for(i = 0; i < NUM_RESOLVERS; i++)
{
return_code = pthread_create(&(resolver_threads[i]), NULL, resolvers, outputfp);
if(return_code)
{
printf("ERROR: return code from pthread_create() is %d\n", return_code);
exit(1);
}
}
for(i = 0; i < (argc - 2); i++)
pthread_join(requestor_threads[i], NULL);
requestors_finished = 1;
for(i = 0; i < NUM_RESOLVERS; i++)
pthread_join(resolver_threads[i], NULL);
pthread_mutex_destroy(&locks[0]);
pthread_mutex_destroy(&locks[1]);
return 0;
}
void* requestors(void* input_file)
{
char* hostname = (char*) malloc(STRING_SIZE);
FILE* input = input_file;
while(fscanf(input, INPUTFS, hostname) > 0)
{
while(queue_is_full(&q))
usleep((rand()%100));
if(!queue_is_full(&q))
{
pthread_mutex_lock(&locks[0]);
if(queue_push(&q, (void*)hostname) == QUEUE_FAILURE)
fprintf(stderr, "Error: queue_push failed on %s\n", hostname);
pthread_mutex_unlock(&locks[0]);
}
hostname = (char*) malloc(STRING_SIZE);
}
printf("%d\n", queue_is_full(&q));
free(hostname);
fclose(input);
pthread_exit(NULL);
}
void* resolvers(void* output_file)
{
char* hostname;
char ipstr[INET6_ADDRSTRLEN];
FILE* output = output_file;
int is_empty = queue_is_empty(&q);
//while(!queue_is_empty(&q) && !requestors_finished)
while((!requestors_finished) || (!is_empty))
{
while(is_empty)
usleep((rand()%100));
pthread_mutex_lock(&locks[0]);
hostname = (char*) queue_pop(&q);
pthread_mutex_unlock(&locks[0]);
if(dnslookup(hostname, ipstr, sizeof(ipstr)) == UTIL_FAILURE)
{
fprintf(stderr, "DNSlookup error: %s\n", hostname);
strncpy(ipstr, "", sizeof(ipstr));
}
pthread_mutex_lock(&locks[1]);
fprintf(output, "%s,%s\n", hostname, ipstr);
pthread_mutex_unlock(&locks[1]);
free(hostname);
is_empty = queue_is_empty(&q);
}
pthread_exit(NULL);
}
Although I'm not familiar with your "queue.h" library, you need to pay attention to the following:
When you check whether your queue is empty you are not acquiring the mutex, meaning that the following scenario might happen:
Some requestors thread checks for emptiness (let's call it thread1) and just before it executes pthread_mutex_lock(&locks[0]); (and after if(!queue_is_full(&q)) ) thread1 gets contex switched
Other requestors threads fill the queue up and when out thread1 finally gets hold of the mutex if will try to insert to the full queue. Now if your queue implementation crashes when one tries to insert more elements into an already full queue thread1 will never unlock the mutex and you'll have a deadlock.
Another scenario:
Some resolver thread runs first requestors_finished is initially 0 so (!requestors_finished) || (!is_empty) is initially true.
But because the queue is still empty is_empty is true.
This thread will reach while(is_empty) usleep((rand()%100)); and sleep forever, because you pthread_join this thread your program will never terminate because this value is never updated in the loop.
The general idea to remember is that when you access some resource that is not atomic and might be accessed by other threads you need to make sure you're the only one performing actions on this resource.
Using a mutex is OK but you should consider that you cannot anticipate when will a context switch occur, so if you want to chech e.g whether the queue is empty you should do this while having the mutex locked and not unlock it until you're finished with it otherwise there's no guarantee that it'll stay empty when the next line executes.
You might also want to consider reading more about the consumer producer problem.
To help you know (and control) when the consumers (resolver) threads should run and when the producer threads produce you should consider using conditional variables.
Some misc. stuff:
pthread_t requestor_threads[argc - 2]; is using VLA and not in a good way - think what will happen if I give no parameters to your program. Either decide on some maximum and define it or create it dynamically after having checked the validity of the input.
IMHO the requestors threads should open the file themselves
There might be some more problems but start by fixing those.

How to "bypass" pthreads limit number

I have a little problem here. I know Linux limits the number of threads of an user can actually run.
I'm using pthread_create and an array of pthread_t limited with 50 ( pthread_t tid[50]; ). I have a for cycle that each time that limit reaches 50 every thread on pthread_t array is killed.
How? I tested almost everything. with pthread_kill(tid[w],SIGKILL); w is a simple cycle control variable goes from 0 to 50. I already tested pthread_cancel(tid[w]); and the problem keeps.
So what is the problem?
Everytime I reach 380 thread number I can't create more. But I'm killing with cancel or kill. So what is happening?
The objective of the program is a network scanner. To be faster I need like 500 threads with like 2 seconds of timeout to test IP's and ports.
Anyone knows how to "go arround" this problem?
I thought that I could kill the thread it would solve the problem but I was wrong :(
Without use ulimit or in /proc/sys/kernel/threads_max changing values, I looked at pthread_attr_setstacksize but I'm a bit confused :P
any ideas?
EDIT
The code as requested :P
I'm going to put ALL code here:
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <time.h>
#include <unistd.h>
#include <signal.h>
#ifndef SOL_TCP
#define SOL_TCP 6
#endif
#ifndef TCP_USER_TIMEOUT
#define TCP_USER_TIMEOUT 18 //retry
#endif
#define MAX_TH 250
struct ar_stc{
char* ip;
int port;
};
char* ret[2];
int porar[2];
pthread_t tid[MAX_TH];
void create_port_scan_th(char* host,int p,int j);
//cares about args.
//this is not helpful for the threads post on stackoverflow. skip this function
char** arguments_handle(int argc,char **arg)
{
char p[]="-p";
char h[]="-h";
size_t _p,_h;
_p=(size_t)strlen(p);
_h=(size_t)strlen(h);
if(argc!=5)
{
printf("Usage:./file -p PORT-RANGE -h HOST.IP\n");
exit(1);
}
if(strncmp(arg[1],p,_p)==0 || strncmp(arg[1],h,_h)==0 && strncmp(arg[3],p,_p)==0 || strncmp(arg[3],h,_h)==0)
{
if(strncmp(arg[1],p,_p)==0)
{
strncpy(ret[0],arg[2],strlen(arg[2]));
}
else
{
strncpy(ret[1],arg[2],strlen(arg[2]));
}
if(strncmp(arg[3],h,_h)==0)
{
strncpy(ret[1],arg[4],strlen(arg[4]));
}
else
{
strncpy(ret[0],arg[4],strlen(arg[4]));
}
}
return ret;
}
int* take_ports(char *arg)
{
char* ports[2];
ports[0] = malloc(5);
ports[1] = malloc(5);
memset(ports[0],0,5);
memset(ports[1],0,5);
char tmp[5];
int len = strlen(arg);
int i,j=0,x=0;
char min_p[5],max_p[5];
for(i=0;i<len;i++)
{
if(arg[i]=='-')
{
min_p[x]='\0';
j=1;
x=0;
continue;
}
else
{
if(j==0)
min_p[x]=arg[i];
else
max_p[x]=arg[i];
}
x++;
}
max_p[x]='\0';
porar[1]=atoi(max_p);
porar[0]=atoi(min_p);
free(ports[0]);
free(ports[1]);
return porar;
}
void *check_port(void* ar_p)
{
struct ar_stc *ar =ar_p;
char* ip = ar->ip;
int port = ar->port;
int s,conexao;
int timeout = 1000; //1 second timeout
s=socket(AF_INET,SOCK_STREAM,0);
struct sockaddr_in dst;
setsockopt(s,SOL_TCP,TCP_USER_TIMEOUT,(char*)&timeout,sizeof(timeout)); //NOT WORKING :(
if(s<0)
{
printf("\nCouldnt create socket\nPremissions maybe?\n");
exit(1);
}
dst.sin_family = AF_INET;
dst.sin_port = htons(port);
dst.sin_addr.s_addr = inet_addr(ip);
bzero(&(dst.sin_zero),8);
//printf("\nChecking: %d...",port);
conexao = connect(s,(struct sockaddr*)&dst,sizeof(dst));
if(conexao <0)
{
printf("TCP/%d:CLOSED!\n",port); //just to make sure the thread is running
close(s);
return;
}
else
{
printf("TCP/%d:OPEN!\n",port);
close(s);
return;
}
}
int main(int argc, char **argv)
{
int open_ports[65535];
int open_ports_count=0;
int min_p,max_p;
int* p;
ret[0] = malloc(20);
ret[1] = malloc(20);
memset(ret[0],0,20);
memset(ret[1],0,20);
char** ipnport;
ipnport = arguments_handle(argc,argv);
printf("The IP is :%s and the range is %s\n",ipnport[1],ipnport[0]);
p=take_ports(ipnport[0]);
min_p=p[0];
max_p=p[1];
printf("Min port:%d e max port:%d\n",min_p,max_p);
int i;
int thread_count=-1;
for(i=min_p;i<=max_p;i++)
{
thread_count++;
create_port_scan_th(ipnport[1],i,thread_count);
if(thread_count>=MAX_TH)
{
sleep(1);
thread_count=0;
int w;
for(w=0;w<=MAX_TH;w++)
{
pthread_kill(tid[w],SIGKILL);
}
}
}
free(ret[0]);
free(ret[1]);
return 0x0;
}
void create_port_scan_th(char* host,int p,int j)
{
int error;
struct ar_stc *ar;
ar = malloc(sizeof(*ar));
ar->ip=host;
ar->port=p;
error = pthread_create(&(tid[j]),NULL,&check_port,(void*)ar);
if(error!=0)
printf("\nError creating thread:%s\n",strerror(error));
}
But I'm killing with cancel or kill.
First of all, pthread_kill does not kill or end a thread.
(see more at pthread_kill doesnt kill thread C linux or When to use pthread_cancel and not pthread_kill).
If you send SIGKILL to a thread, the entire process will end.
To end a thread, you need to
Make the thread end.
by returning from the thread function, or
calling pthread_exit or
pthread_cancel the thread
Dispose the resources tied to the thread by:
Call pthread_join() on the thread or
make the thread a detached thread.
If you opt for the last point by making the thread detached - which will automatically release the thread when it ends , you can call pthread_detach(pthread_Self()) at the start of your thread function.
Or supply a pthread_attr_t when you call pthread_create(), where you set the thread to a detached state.
As for the total number of threads you can use, linux have a limit on the total number of threads/processes any user can have running.
You can view this with the command ulimit -u

Resources