Reading from named pipe freezes when piping tshark output - c

I'm trying to implement a web prefetching system. The purpose of a system like this is to “predict” future requests and prefetch them.
The system builds a predictive model from web navigation logs (Squid access.log files). The model is a dependency graph, where a node representing URL A has an arc to a node representing URL B if URL B has been requested immediately after URL A.
Once the model is built, the system receives queries of URLs requested by users, and make “predictions” based on the graph. Predictions are resources (URLs) very likely to be requested in the future. So, based on predictions, the system prefetches these resources to store them in cache prior to users' requests.
I'm using the following testing scenario:
A process simulate multiple clients, requesting URLs in a file using libcurl. The process runs in a different PC from the prefetching system. PCs are connected directly via an ethernet cable
Requests made by the client simulator are always the same URLs in the same relative time from the first request made. All requests are going to port 3128 (Prefetch PC Squid listen port) (port 80 DNAT to port 3128 in the client).
The prefetching system runs in a CentOS 6.3 box, kernel 2.6.32-71.el6.i686, 2 core Intel Pentium 4 3.00GHz processor, 4 GB RAM.
The prefetching system is one process with multiple threads. The main thread creates the predictive model and generates predictions based on queries. A “listener” thread reads URLs requested by users and prefetches predicted URLs using libcurl. “Listening” means reading from a named pipe (called url_fifo) URLs captured live on an interface using tshark:
stdbuf -o0 tshark -i eth1 tcp port 3128 and "tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420" -T fields -e http.request.full_uri >> url_fifo
Each 10 minutes (1 cycle) the model is updated based on requests from the last cycle. The client tells the system when a cycle ends and so the model is updated. Once the model is updated, the system tells the client to start requesting URLs from the next cycle.
Here is the situation: Sometimes reading from the named pipe freezes. No URLs are read from the pipe even though tshark keeps capturing URLs and redirecting them to the named pipe. After an hour (or a couple of hours) all “buffered” URLs are read in less than 10 minutes. After that, reading from the pipe keeps going ok again. This situation doesn't happen always (50% of times freezes, 50% no).
It seems that there is a buffering issue, since tshark keeps capturing URLs and all requests are correctly logged in Squid's access.log.
In the beginning, I ran tshark with the -l option, so that its output becomes line buffered. Then I started using stdbuf -o0 (no buffering). Anyway the situation still happens.
In the system code, I also tried opening and reading the named pipe as a stream (FILE *) and set the stream as no buffered or line buffered (using setvbuf() function). The situation still happened.
In some cycles requests are faster than in other cycles. Anyway, it doesn't seems to be a fast producer slow consumer issue, since in many repetitions of the test all URLs are correctly read and processed without any freezes.
Is there something am I missing related to named pipes and buffering? I'd really appreciate some guidance.
Assume networking (interfaces, routing, iptables, squid) is ok. I've not had any issues related to it.
Code (assume necessary header files are included):
functions.c
#define BUFLEN 512
#define QUEUE_LEN 64
#define THREADS_LEN 2
pthread_mutex_t model_lock;
pthread_cond_t model_cond, listen_cond;
pthread_t queries_thread, listen_thread;
short int model_is_updating, model_can_update, program_shutdown;
/* Program execution statistics */
Status * program_status;
/* Thread pool */
threadpool_t *pool;
/* program execution */
int
run(void)
{
Graph_Adj_List * gr = NULL; /* Graph as an adjacency list */
char ** reports = NULL;
unsigned report_counter = 0;
/* Init program status */
program_status = status_init();
/* Load list of custom web navigation reports to be used to build the initial
* version of the predictive model */
reports = file_load_reports(program_config.reports_file);
if (!reports)
return 0;
/* Init mutex and cond */
pthread_mutex_init(&model_lock, NULL);
pthread_cond_init(&model_cond, NULL);
pthread_cond_init(&listen_cond, NULL);
/* Lock */
pthread_mutex_lock (&model_lock);
/* Start first cycle */
status_start_cycle(program_status);
/* Create initial version of the predictive model */
gr = create_model_from_files(reports, &report_counter, program_config.reports_limit);
if (!gr)
{
/* Unlock */
pthread_mutex_unlock (&model_lock);
return 0;
}
/* Unlock */
pthread_mutex_unlock (&model_lock);
/* Start threads */
if (pthread_create(&queries_thread, NULL, fifo_predictions_threaded, (void *)gr) ||
pthread_create(&listen_thread, NULL, listen_end_of_cycle, NULL))
program_shutdown = 1;
/* main loop */
while(!program_shutdown)
{
/* lock */
pthread_mutex_lock (&model_lock);
/* wait for clients' announcement of the end of requests from current cycle */
while (!model_can_update)
pthread_cond_wait(&model_cond, &model_lock);
/* set updating flag */
model_is_updating = 1;
/* Update predictive model, based on Squid's access.log from (about to finish)
* current cycle */
adj_list_update_access(gr, program_config.access_file);
/* Save statistics related to the current cycle and finish it */
status_finish_cycle(program_status);
/* Check if last custom report has been read */
if (!reports[report_counter])
{
program_shutdown = 1;
pthread_mutex_unlock (&model_lock);
break;
}
/* Start a new cycle */
status_start_cycle(program_status);
/* Read a new custom report and update the predictive model */
update_model(gr, reports[report_counter]);
report_counter++;
/* Updating is done */
model_is_updating = 0;
/* Model can't be updated until client announces the end of the cycle
* that has just started */
model_can_update = 0;
/* Tell client to start sending requests from the new cycle */
if (!signal_start_cycle())
{
program_shutdown = 1;
pthread_mutex_unlock (&model_lock);
break;
}
/* Signal listener thread that a new cycle has begin */
pthread_cond_signal(&listen_cond);
/* Unlock */
pthread_mutex_unlock (&model_lock);
}
/* Finish threads */
pthread_cancel(listen_thread);
pthread_cancel(queries_thread);
pthread_join(listen_thread, NULL);
pthread_join(queries_thread, NULL);
/* Free memory */
adj_list_free_all2(&gr);
file_reports_free_all(&reports);
pthread_cond_destroy(&model_cond);
pthread_cond_destroy(&listen_cond);
pthread_mutex_destroy(&model_lock);
status_free(&program_status);
return 1;
}
void *
fifo_predictions_threaded(void * data)
{
Graph_Adj_List * gr = (Graph_Adj_List *) data;
/* Set thread cancel type */
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
while (!program_shutdown)
{
pthread_mutex_lock(&model_lock);
/* Pause reading from named pipe while the model is being updated */
while(model_is_updating)
pthread_cond_wait(&listen_cond, &model_lock);
pthread_mutex_unlock(&model_lock);
/* Read URLs from named pipe */
fifo_predictions(gr, program_config.fifo);
}
pthread_exit(NULL);
return NULL;
}
int
fifo_predictions(Graph_Adj_List * gr, const u8 * fifo)
{
u8 cad[BUFLEN] = { '\0' };
u8 * ini = NULL, * fin = NULL, * fullurl = NULL;
int i, fifo_descriptor, read_urls = 0, fullurl_len = 0, incomplete_url = 1;
FILE * fifo_file = NULL;
/* Open fifo in blocking mode */
fifo_descriptor = open(CHAR_CAST fifo, O_RDONLY);
/* Open fifo as a stream */
// fifo_file = fopen(fifo, "r");
// if (!fifo_file)
if (fifo_descriptor == -1)
return 0;
/* If fifo is opened as a stream, set it line buffered */
// setlinebuf(fifo_file);
do
{
if ((i = read(fifo_descriptor, cad, BUFLEN - 1)) == -1)
// if ( fgets(cad, BUFLEN-1, fifo_file) == NULL)
ERROR(__FILE__, __FUNCTION__, __LINE__, "Fifo read error");
else
{
// i = strlen(cad);
cad[i] = '\0';
read_urls = 0;
if (i > 0)
{
int j = 0;
for (j = 0, ini = cad, fin = NULL ; cad[j] != '\0'; j++)
{
if (cad[j] == '\n')
{
/* Save URL */
fin = &cad[j];
ini = (*ini == '\n' ? ini + 1 : ini);
/* Check if string is a continuation of the previously read URL */
read_urls = fin - ini;
read_urls = read_urls >= 0 ? read_urls : 0;
/* Save URL in fullurl string */
fullurl = realloc(fullurl, fullurl_len + read_urls + 1);
memcpy(&fullurl[fullurl_len], ini, read_urls);
fullurl[fullurl_len + read_urls] = '\0';
ini = fin;
incomplete_url = fullurl_len = 0;
/* Ask the model for predictions and fetch them */
fetch_url_predictions2(gr, fullurl);
u8_free(&fullurl);
} else
incomplete_url = 1;
}
if (incomplete_url)
{
ini = (*ini == '\n' ? ini + 1 : ini);
read_urls = &cad[j] - ini;
read_urls = read_urls >= 0 ? read_urls : 0;
fullurl = realloc(fullurl, fullurl_len + read_urls);
memcpy(&fullurl[fullurl_len], ini, read_urls);
fullurl_len += read_urls;
}
}
}
} while (i > 0);
close(fifo_descriptor);
// fclose (fifo_file);
return 1;
}
int
fetch_url_predictions2(Graph_Adj_List * gr, u8 * in_url)
{
String * string_url = NULL;
Headnode * head = NULL;
LinkedList * list = NULL;
LinkedListElem * elem = NULL;
/* Use custom string type */
string_url = string_create_no_len(in_url);
if (!string_url)
return 0;
pthread_mutex_lock(&model_lock);
/* Get URL node */
head = adj_list_get_node(gr, string_url);
if (head)
{
/* Get predictions (URLs) as a linked list */
list = adj_list_predictions_to_list(head);
if (!list)
{
string_free_all(&string_url);
return 0;
}
pthread_mutex_unlock(&model_lock);
/* Callback fetches URLs */
list->callback = &curl_callback_void;
if (!pool)
pool = threadpool_create(THREADS_LEN, QUEUE_LEN, 0);
/* Load URLs to be fetched to threadpool's task queue */
for (elem = list->first; elem; elem = elem->next)
{
CallbackArg arg;
arg.data = arg.copy(elem->data);
threadpool_add_copy_arg(pool, list->callback, &arg, 1, sizeof(arg), 0);
}
linked_list_free_all(&list);
}
pthread_mutex_unlock(&model_lock);
string_free_all(&string_url);
return 1;
}
fetch.c
void
curl_callback_void(void * data)
{
CallbackArg * arg = (CallbackArg *) data;
char * url = (char *) arg->data;
fetch_url(url);
}
static size_t
write_data(void *buffer, size_t size, size_t nmemb, void *userp)
{
return size * nmemb;
}
int
fetch_url(char * url)
{
CURL *curl;
CURLcode res;
struct timeval time;
char * time_string = NULL;
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, NULL);
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 15);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
gettimeofday(&time, NULL);
time_string = timeval_to_str(&time);
/* Check for errors */
if (res != CURLE_OK)
{
fprintf(stderr, "\ntime %s curl_easy_perform() (url %s) failed: %s\n",
time_string, url, curl_easy_strerror(res));
}
else
{
fprintf(stderr, "\ntime %s curl_easy_perform() (url %s) fetched ok\n",
time_string, url);
}
fflush(stderr);
free (time_string);
curl_easy_cleanup(curl);
}
return 0;
}
network.c
/*
* Code based on Beej's Networking Guide
*/
#define MSG_LEN 5
#define QUEUE_SIZE 5
extern pthread_mutex_t model_lock;
extern pthread_cond_t model_cond;
extern short int model_can_update, program_shutdown;
extern Config program_config;
// get sockaddr, IPv4 or IPv6:
static void *
get_in_addr(struct sockaddr *sa) {
if (sa->sa_family == AF_INET) {
return &(((struct sockaddr_in*) sa)->sin_addr);
}
return &(((struct sockaddr_in6*) sa)->sin6_addr);
}
void *
listen_end_of_cycle(void * data)
{
int sockfd, new_fd; // listen on sock_fd, new connection on new_fd
struct addrinfo hints, *servinfo, *p;
struct sockaddr_storage their_addr; // connector's address information
socklen_t sin_size;
int yes = 1;
char s[INET_ADDRSTRLEN], msg[MSG_LEN], *str = NULL;
int rv;
int read_bytes;
struct timeval actual_time;
/* Set thread cancel type */
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL );
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE; // use my IP
if ((rv = getaddrinfo(NULL, program_config.listen_port, &hints, &servinfo))
!= 0) {
fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(rv));
return "error";
}
// loop through all the results and bind to the first we can
for (p = servinfo; p != NULL ; p = p->ai_next) {
if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol))
== -1) {
perror("server: socket");
continue;
}
if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int))
== -1) {
perror("setsockopt");
return "error";
}
if (bind(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
close(sockfd);
perror("server: bind");
continue;
}
break;
}
if (p == NULL ) {
fprintf(stderr, "server: failed to bind\n");
return "error";
}
freeaddrinfo(servinfo); // all done with this structure
if (listen(sockfd, QUEUE_SIZE) == -1) {
perror("listen");
return "error";
}
while (!program_shutdown)
{
sin_size = sizeof their_addr;
new_fd = accept(sockfd, (struct sockaddr *) &their_addr, &sin_size);
if (new_fd == -1) {
perror("accept");
continue;
}
inet_ntop(their_addr.ss_family,
get_in_addr((struct sockaddr *) &their_addr), s, sizeof s);
if ((read_bytes = recv(new_fd, msg, MSG_LEN - 1, 0)) == -1) {
perror("recv");
continue;
}
close(new_fd);
msg[read_bytes] = '\0';
/* Check received message */
if (strcmp(msg, "DONE")) {
perror("Not valid message");
continue;
}
printf("\ngot \"DONE\" from %s\n", s);
fflush(stdout);
/* Lock */
pthread_mutex_lock(&model_lock);
/* Flag used by main thread to allow model update */
model_can_update = 1;
/* Signal model can be updated */
pthread_cond_signal(&model_cond);
/* Unlock */
pthread_mutex_unlock(&model_lock);
}
close(sockfd);
pthread_exit(NULL);
return "ok";
}
int signal_start_cycle(void) {
int sockfd;
struct addrinfo hints, *servinfo, *p;
int rv;
char s[INET6_ADDRSTRLEN], *str = NULL;
struct timeval actual_time, aux_time;
struct timeval connect_timeout = { 15, 0 }, max_connect_time = { 0, 0 };
short int connected = 0;
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
if ((rv = getaddrinfo(program_config.client_ip, program_config.client_port,
&hints, &servinfo)) != 0) {
fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(rv));
return 0;
}
gettimeofday(&aux_time, NULL);
timeval_add(aux_time, connect_timeout, &max_connect_time);
/* Try several times to connect to the remote side */
do {
// loop through all the results and connect to the first we can
for (p = servinfo; p != NULL ; p = p->ai_next) {
if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol))
== -1) {
perror("client: socket");
continue;
}
gettimeofday(&actual_time, NULL )
printf("\ntrying to connect %s\n", program_config.client_ip);
fflush(stdout);
if (connect(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
close(sockfd);
perror("client: connect");
continue;
}
connected = 1;
break;
}
} while (!connected && !timeval_greater_than(actual_time, max_connect_time));
if (p == NULL ) {
fprintf(stderr, "client: failed to connect\n");
return 0;
}
inet_ntop(p->ai_family, get_in_addr((struct sockaddr *) p->ai_addr), s,
sizeof s);
printf("\nMAIN THREAD: connecting to %s\n", s);
fflush(stdout);
freeaddrinfo(servinfo); // all done with this structure
if (send(sockfd, "DONE", 4, 0) == -1)
{
perror("send");
return 0;
}
printf("\nsent \"DONE\" to %s\n", s);
fflush(stdout);
close(sockfd);
return 1;
}

Related

Recieve a message from server asynchronously

I have a client program and a server program. There could be multiple servers and multiple
clients that can connect to multiple servers of there choice
The client program lists a menu
connect 4000 // connects to server on port 4000
bid 1000 4000 // send a bid value of 1000 to the server at port 4000
Now a server may recieve bids from several clients connected to it and keeps track of the highest
bid till now. Whenever a new bid is placed the server sends a broadcast to each client connected
to it one by one like - write(users[i].sock_fd, msg, size).
How do I listen to this message on the client side ?
There are two things here
The client needs to listen to the message sent by server.
The client is also reading the text or menu items (connect and bid) from command line from the user.
I have coded the part 2) But confused how to code 1) into client and simultaneously make the 2) also working
Client code :
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>
#define BUF_SIZE 128
#define MAX_AUCTIONS 5
#ifndef VERBOSE
#define VERBOSE 0
#endif
#define ADD 0
#define SHOW 1
#define BID 2
#define QUIT 3
/* Auction struct - this is different than the struct in the server program
*/
typedef struct auction_data
{
int sock_fd;
char item[BUF_SIZE];
int current_bid;
} auction_data;
auction_data *auction_data_ptr;
/* Displays the command options available for the user.
* The user will type these commands on stdin.
*/
void print_menu()
{
printf("The following operations are available:\n");
printf(" show\n");
printf(" add <server address> <port number>\n");
printf(" bid <item index> <bid value>\n");
printf(" quit\n");
}
/* Prompt the user for the next command
*/
void print_prompt()
{
printf("Enter new command: ");
fflush(stdout);
}
/* Unpack buf which contains the input entered by the user.
* Return the command that is found as the first word in the line, or -1
* for an invalid command.
* If the command has arguments (add and bid), then copy these values to
* arg1 and arg2.
*/
int parse_command(char *buf, int size, char *arg1, char *arg2)
{
int result = -1;
char *ptr = NULL;
if (strncmp(buf, "show", strlen("show")) == 0)
{
return SHOW;
}
else if (strncmp(buf, "quit", strlen("quit")) == 0)
{
return QUIT;
}
else if (strncmp(buf, "add", strlen("add")) == 0)
{
result = ADD;
}
else if (strncmp(buf, "bid", strlen("bid")) == 0)
{
result = BID;
}
ptr = strtok(buf, " "); // first word in buf
ptr = strtok(NULL, " "); // second word in buf
if (ptr != NULL)
{
strncpy(arg1, ptr, BUF_SIZE);
}
else
{
return -1;
}
ptr = strtok(NULL, " "); // third word in buf
if (ptr != NULL)
{
strncpy(arg2, ptr, BUF_SIZE);
return result;
}
else
{
return -1;
}
return -1;
}
/* Connect to a server given a hostname and port number.
* Return the socket for this server
*/
int add_server(char *hostname, int port)
{
// Create the socket FD.
int sock_fd = socket(AF_INET, SOCK_STREAM, 0);
if (sock_fd < 0)
{
perror("client: socket");
exit(1);
}
// Set the IP and port of the server to connect to.
struct sockaddr_in server;
server.sin_family = AF_INET;
server.sin_port = htons(port);
struct addrinfo *ai;
/* this call declares memory and populates ailist */
if (getaddrinfo(hostname, NULL, NULL, &ai) != 0)
{
close(sock_fd);
return -1;
}
/* we only make use of the first element in the list */
server.sin_addr = ((struct sockaddr_in *)ai->ai_addr)->sin_addr;
// free the memory that was allocated by getaddrinfo for this list
freeaddrinfo(ai);
// Connect to the server.
if (connect(sock_fd, (struct sockaddr *)&server, sizeof(server)) == -1)
{
perror("client: connect");
close(sock_fd);
return -1;
}
if (VERBOSE)
{
fprintf(stderr, "\nDebug: New server connected on socket %d. Awaiting item\n", sock_fd);
}
return sock_fd;
}
/* ========================= Add helper functions below ========================
* Please add helper functions below to make it easier for the TAs to find the
* work that you have done. Helper functions that you need to complete are also
* given below.
*/
/* Print to standard output information about the auction
*/
void print_auctions(struct auction_data *a, int size)
{
printf("Current Auctions:\n");
for (int i = 0; i < size; i++)
{
struct auction_data auction_data = a[i];
printf("(%d) %s bid = %d\n", i, auction_data.item, auction_data.current_bid);
}
/* TODO Print the auction data for each currently connected
* server. Use the follosing format string:
* "(%d) %s bid = %d\n", index, item, current bid
* The array may have some elements where the auction has closed and
* should not be printed.
*/
}
/* Process the input that was sent from the auction server at a[index].
* If it is the first message from the server, then copy the item name
* to the item field. (Note that an item cannot have a space character in it.)
*/
void update_auction(char *buf, int size, struct auction_data *a, int index)
{
// TODO: Complete this function
// fprintf(stderr, "ERROR malformed bid: %s", buf);
// printf("\nNew bid for %s [%d] is %d (%d seconds left)\n", );
}
int main(void)
{
char name[BUF_SIZE];
int size = 0;
// Declare and initialize necessary variables
// TODO
// Get the user to provide a name.
printf("Please enter a username: ");
fflush(stdout);
int num_read = read(STDIN_FILENO, name, BUF_SIZE);
printf("%s-name\n", name);
if (num_read <= 0)
{
fprintf(stderr, "ERROR: read from stdin failed\n");
exit(1);
}
print_menu();
// TODO
char server_reply[2000];
while (1)
{
print_prompt();
char *command;
scanf("%m[^\n]s", &command);
getchar();
char arg1[100];
char arg2[100];
int commandNumber = parse_command(command, 1000, arg1, arg2);
char dest[100] = "";
strcpy(dest, name);
dest[strlen(dest) - 1] = '\0';
if (commandNumber == ADD)
{
printf("%s-name4\n", dest);
int port = atoi(arg2);
int sock_fd = add_server(arg1, port);
printf("%s-server\n", server_reply);
write(sock_fd, dest, strlen(dest));
auction_data_ptr = (auction_data *)realloc(auction_data_ptr, (size + 1) * sizeof(auction_data_ptr));
auction_data_ptr[size].sock_fd = sock_fd;
size++;
}
else if (commandNumber == SHOW)
{
print_auctions(auction_data_ptr, size);
}
else if (commandNumber == BID)
{
int itemIndex = atoi(arg1);
int bidValue = atoi(arg2);
printf("%d-test\n", auction_data_ptr[itemIndex].sock_fd);
send(auction_data_ptr[itemIndex].sock_fd, arg2, strlen(arg2), 0);
}
else if (commandNumber == QUIT)
{
}
// TODO
}
return 0; // Shoud never get here
}
Server Code :
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#ifndef PORT
#define PORT 30000
#endif
#define MAX_BACKLOG 5
#define MAX_CONNECTIONS 20
#define BUF_SIZE 128
#define MAX_NAME 56
int verbose = 0;
struct user
{
int sock_fd;
char name[MAX_NAME];
int bid;
};
typedef struct
{
char *item;
int highest_bid; // value of the highest bid so far
int client; // index into the users array of the top bidder
} Auction;
/*
* Accept a connection. Note that a new file descriptor is created for
* communication with the client. The initial socket descriptor is used
* to accept connections, but the new socket is used to communicate.
* Return the new client's file descriptor or -1 on error.
*/
int accept_connection(int fd, struct user *users)
{
int user_index = 0;
while (user_index < MAX_CONNECTIONS && users[user_index].sock_fd != -1)
{
user_index++;
}
if (user_index == MAX_CONNECTIONS)
{
fprintf(stderr, "server: max concurrent connections\n");
return -1;
}
int client_fd = accept(fd, NULL, NULL);
if (client_fd < 0)
{
perror("server: accept");
close(fd);
exit(1);
}
users[user_index].sock_fd = client_fd;
users[user_index].name[0] = '\0';
return client_fd;
}
/* Remove \r\n from str if the characters are at the end of the string.
* Defensively assuming that \r could be the last or second last character.
*/
void strip_newline(char *str)
{
if (str[strlen(str) - 1] == '\n' || str[strlen(str) - 1] == '\r')
{
if (str[strlen(str) - 2] == '\r')
{
str[strlen(str) - 2] = '\0';
}
else
{
str[strlen(str) - 1] = '\0';
}
}
}
/*
* Read a name from a client and store in users.
* Return the fd if it has been closed or 0 otherwise.
*/
int read_name(int client_index, struct user *users)
{
int fd = users[client_index].sock_fd;
/* Note: This is not the best way to do this. We are counting
* on the client not to send more than BUF_SIZE bytes for the
* name.
*/
int num_read = read(fd, users[client_index].name, MAX_NAME);
if (num_read == 0)
{
users[client_index].sock_fd = -1;
return fd;
}
users[client_index].name[num_read] = '\0';
strip_newline(users[client_index].name);
if (verbose)
{
fprintf(stderr, "[%d] Name: %s\n", fd, users[client_index].name);
}
/*
if (num_read == 0 || write(fd, buf, strlen(buf)) != strlen(buf)) {
users[client_index].sock_fd = -1;
return fd;
}
*/
return 0;
}
/* Read a bid from a client and store it in bid.
* If the client does not send a number, bid will be set to -1
* Return fd if the socket is closed, or 0 otherwise.
*/
int read_bid(int client_index, struct user *users, int *bid)
{
printf("inside bid\n");
int fd = users[client_index].sock_fd;
char buf[BUF_SIZE];
char *endptr;
int num_read = read(fd, buf, BUF_SIZE);
if (num_read == 0)
{
return fd;
}
buf[num_read] = '\0';
if (verbose)
{
fprintf(stderr, "[%d] bid: %s", fd, buf);
}
// Check if the client sent a valid number
// (We are not checking for a good bid here.)
errno = 0;
*bid = strtol(buf, &endptr, 10);
if (errno != 0 || endptr == buf)
{
*bid = -1;
}
return 0;
}
void broadcast(struct user *users, char *msg, int size)
{
for (int i = 0; i < MAX_CONNECTIONS; i++)
{
if (users[i].sock_fd != -1)
{
if (write(users[i].sock_fd, msg, size) == -1)
{
// Design flaw: can't remove this socket from select set
close(users[i].sock_fd);
users[i].sock_fd = -1;
}
}
}
}
int prep_bid(char *buf, Auction *a, struct timeval *t)
{
// send item, current bid, time left in seconds
printf("robin2-%s-%d\n", a->item, a->highest_bid);
printf("robin-%ld\n", t->tv_sec);
sprintf(buf, "%s %d %ld", a->item, a->highest_bid, t->tv_sec);
printf("robin-bid2\n");
return 0;
}
/* Update auction if new_bid is higher than current bid.
* Write to the client who made the bid if it is lower
* Broadcast to all clients if the bid is higher
*/
int update_bids(int client_index, struct user *users,
int new_bid, Auction *auction, struct timeval *t)
{
char buf[BUF_SIZE];
if (new_bid > auction->highest_bid)
{
auction->highest_bid = new_bid;
auction->client = client_index;
prep_bid(buf, auction, t);
if (verbose)
{
fprintf(stderr, "[%d] Sending to %d:\n %s\n",
getpid(), users[client_index].sock_fd, buf);
}
broadcast(users, buf, strlen(buf) + 1);
}
else
{
fprintf(stderr, "Client %d sent bid that was too low. Ignored\n",
client_index);
}
return 0;
}
int main(int argc, char **argv)
{
argc = 7;
argv[1] = "-v";
argv[2] = "-t";
argv[3] = "5";
argv[4] = "-p";
argv[5] = "4000";
argv[6] = "robin";
Auction auction;
int opt;
int port = PORT;
struct timeval timeout;
struct timeval *time_ptr = NULL;
int minutes = 0;
while ((opt = getopt(argc, argv, "vt:p:")) != -1)
{
switch (opt)
{
case 'v':
verbose = 1;
break;
case 't':
minutes = atoi(optarg);
timeout.tv_sec = minutes * 60;
timeout.tv_usec = 0;
time_ptr = &timeout;
break;
case 'p':
port = atoi(optarg);
break;
default:
fprintf(stderr, "Usage: auction_server [-v] [-t timeout] [-p port] item\n");
exit(1);
}
}
if (optind >= argc)
{
fprintf(stderr, "Expected argument after options\n");
exit(1);
}
auction.item = argv[optind];
auction.client = -1;
auction.highest_bid = -1;
struct user users[MAX_CONNECTIONS];
for (int index = 0; index < MAX_CONNECTIONS; index++)
{
users[index].sock_fd = -1;
users[index].name[0] = '\0';
}
// Create the socket FD.
int sock_fd = socket(AF_INET, SOCK_STREAM, 0);
if (sock_fd < 0)
{
perror("server: socket");
exit(1);
}
// Set information about the port (and IP) we want to be connected to.
struct sockaddr_in server;
server.sin_family = AF_INET;
server.sin_port = htons(port);
server.sin_addr.s_addr = INADDR_ANY;
// This sets an option on the socket so that its port can be reused right
// away. Since you are likely to run, stop, edit, compile and rerun your
// server fairly quickly, this will mean you can reuse the same port.
int on = 1;
int status = setsockopt(sock_fd, SOL_SOCKET, SO_REUSEADDR,
(const char *)&on, sizeof(on));
if (status == -1)
{
perror("setsockopt -- REUSEADDR");
}
// This should always be zero. On some systems, it won't error if you
// forget, but on others, you'll get mysterious errors. So zero it.
memset(&server.sin_zero, 0, 8);
// Bind the selected port to the socket.
if (bind(sock_fd, (struct sockaddr *)&server, sizeof(server)) < 0)
{
perror("server: bind");
close(sock_fd);
exit(1);
}
// Announce willingness to accept connections on this socket.
if (listen(sock_fd, MAX_BACKLOG) < 0)
{
perror("server: listen");
close(sock_fd);
exit(1);
}
if (verbose)
{
fprintf(stderr, "[%d] Ready to accept connections on %d\n",
getpid(), port);
}
// The client accept - message accept loop. First, we prepare to listen
// to multiple file descriptors by initializing a set of file descriptors.
int max_fd = sock_fd;
fd_set all_fds;
FD_ZERO(&all_fds);
FD_SET(sock_fd, &all_fds);
while (1)
{
// select updates the fd_set it receives, so we always use a copy
// and retain the original.
fd_set listen_fds = all_fds;
int nready;
if ((nready = select(max_fd + 1, &listen_fds, NULL, NULL, time_ptr)) == -1)
{
perror("server: select");
exit(1);
}
if (nready == 0)
{
char buf[BUF_SIZE];
sprintf(buf, "Auction closed: %s wins with a bid of %d\r\n",
users[auction.client].name, auction.highest_bid);
printf("%s", buf);
broadcast(users, buf, BUF_SIZE);
exit(0);
}
// Is it the original socket? Create a new connection ...
if (FD_ISSET(sock_fd, &listen_fds))
{
int client_fd = accept_connection(sock_fd, users);
if (client_fd != -1)
{
if (client_fd > max_fd)
{
max_fd = client_fd;
}
FD_SET(client_fd, &all_fds);
if (verbose)
{
fprintf(stderr, "[%d] Accepted connection on %d\n",
getpid(), client_fd);
}
}
}
// Next, check the clients.
for (int index = 0; index < MAX_CONNECTIONS; index++)
{
if (users[index].sock_fd > -1 && FD_ISSET(users[index].sock_fd, &listen_fds))
{
int client_closed = 0;
int new_bid = 0;
if (users[index].name[0] == '\0')
{
client_closed = read_name(index, users);
if (client_closed == 0)
{
char buf[BUF_SIZE];
prep_bid(buf, &auction, time_ptr);
if (verbose)
{
fprintf(stderr, "[%d] Sending to %d:\n %s\n",
getpid(), users[index].sock_fd, buf);
}
if (write(users[index].sock_fd, buf, strlen(buf) + 1) == -1)
{
fprintf(stderr, "Write to %d failed\n", sock_fd);
close(sock_fd);
}
}
}
else
{ // read a bid
client_closed = read_bid(index, users, &new_bid);
if (client_closed == 0)
{
update_bids(index, users, new_bid, &auction, time_ptr);
}
}
if (client_closed > 0)
{
FD_CLR(client_closed, &all_fds);
printf("Client %d disconnected\n", client_closed);
}
}
}
}
// Should never get here.
return 1;
}
Caveat: Because you've only posted partial code for server and client, this will be some suggestions.
Your client can attach/connect to multiple bid servers simultaneously. As such, it must be able to keep track of the multiple connections in a manner similar to a server.
Your main [stated] issue is that you're blocking the client on a user prompt (e.g. from stdin via scanf et. al.). Presently, this means that the client is "stuck" at user input prompt and can not field messages from the servers it is connected to. More on how to fix this below.
So, you'll have a bunch of code from the server that needs to be in the client with some minor differences. You may wish to generalize some of the server code a bit, so it can work both in server and client (e.g. you may want to move it to common.c).
You already have code in the server to handle multiple connections. The server needs a select mask that is the OR of the listen fd and all active client fds.
Likewise, your client needs a select mask that is the OR of the fd for user input (e.g. 0) and all active server connections.
Doing select on fd 0 and using stdio.h streams won't work too well. So, replace access to stdin with (e.g.) read(0,line_buffer,sizeof(line_buffer)). You do this if fd 0 is set in the select mask. The role is very similar to what your server does for the accept on sock_fd.
You'll need to allow for partial reads and append to the buffer until you see a newline. So, you'll have to do the work that fgets would normally do in assembling a whole line. Then, you can call parse_command.
Because read doesn't understand newline demarcations, the user could enter more than one line before you can do a read.
So, for user input of:
connect 4000\n
bid 100 4000\n
connect 5000\n
You may get partial reads of:
conn
ect
4000\nbid 100 4000
\nconnect
5000\n
You may also need to use the FIONREAD ioctl on the fd 0 to prevent blocking. And, you may need to set the kernel TTY layer into raw mode via termios calls.
The client now becomes very similar to your server code. It will handle [asynchronously] actions by any connected servers and user input.
A tip: Under the DRY principle ["don't repeat yourself"] ...
You already have a struct user in the server. The client will need something similar/identical, such as struct server. When generalizing the code, rather than having two distinct structs that do essentially the same thing, consider renaming the existing struct to (e.g.) struct connection

Trouble with converting single threaded HTTP server into multithreaded using pthreads

My single threaded HTTP Server works just fine, but I'm having trouble multithreading it. I know I am supposed to use pthreads, locks, and condition variables, but I can't get the logic set up properly. The trouble starts after listening to the server. Currently I have a struct that contains a client socket variable, a lock variable, a condition variable, and some variables necessary for parsing and storing headers. I create a struct array sized with the amount of threads, then create a pthread array sized with the amount of threads. I go into a while(1) loop which goes into a for loop and iterates through all the threads accepting each connection, calling pthread_create and passing them to my handle connections function, then closing the client socket. My handle connections then does the request handling that my single threaded http server did (reading, parsing, processing, constructing), then returns NULL. No request gets read when I run this using pthread_create, but if I run handle connections without the pthreads, it works just fine. And below I'll attach my code. Any help is appreciated
Thank you for commenting so well ...
Okay, I coded up, but not tested the changes.
Your loop is inherently single threaded, so a bit of refactoring is in order
You have to scan for an unused thread control slot after doing accept.
You have to pthread_join completed/done threads [from any prior invocations].
The thread function has to close the per-client socket [not main thread]
You need a global (file scope) mutex.
I've coded it up, but not tested it. I put #if 0 around most of what I clipped out and #if 1 around new code.
Note that number of simultaneous connections [second arg to listen], herein 5 has to be less than or equal to threadNum. Although I didn't do it, I'd just do listen(...,threadNum) instead of hardwiring it.
Here's the short code with just the relevant changes:
#if 1
pthread_mutex_t global_mutex = PTHREAD_MUTEX_INITIALIZER;
#endif
struct threadObject {
char method[5]; // PUT, HEAD, GET. HEAD==4 letters+null terminator
char filename[28]; // what is the file we are worried about. Max 27 ASCII characters (NULL terminated on 28)
char httpversion[9]; // HTTP/1.1
ssize_t content_length; // example: 13
uint16_t status_code; // status code for the request
char buffer[BUFFER_SIZE]; // buffer to transfer data
char rest_of_PUT[BUFFER_SIZE]; // incase client send part of PUT message in header
int client_sockd;
pthread_mutex_t *dispatch_lock;
const pthread_cond_t *job_pool_empty;
// pthread_mutex_t* log_lock;
// const pthread_cond_t* log_pool_empty;
pthread_mutex_t *read_write_lock;
pthread_cond_t *file_list_update;
// JobQueue* job_pool;
// LogQueue log_pool;
// bool is_logging;
#if 1
pthread_t tsk_threadid;
int tsk_inuse;
int tsk_done;
#endif
};
void *
handle_connections(void *ptr_thread)
{
// create a mutual exclusion to lock out any other threads from the function
// pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
// pthread_mutex_lock(&mutex);
// operations go here
struct threadObject *thread = (struct threadObject *) ptr_thread;
// reset message after each loop
memset(thread->buffer, '\0', BUFFER_SIZE);
memset(thread->method, '\0', 5);
memset(thread->filename, '\0', 28);
memset(thread->httpversion, '\0', 9);
thread->content_length = 0;
thread->status_code = 0;
memset(thread->rest_of_PUT, '\0', BUFFER_SIZE);
// read message
if (read_http_response(thread) == true) {
// process message
process_request(thread);
}
// construct a response
construct_http_response(thread);
// unlock the function
// pthread_mutex_unlock(&mutex);
#if 1
close(thread->client_sockd);
pthread_mutex_lock(&global_mutex);
thread->tsk_done = 1;
pthread_mutex_unlock(&global_mutex);
#endif
return NULL;
}
int
main(int argc, char **argv)
{
// Create sockaddr_in with server information
if (argc < 2) {
perror("No arguments passed\n");
return -1;
}
// make sure port number is above 1024 and set the port # to it
if (atoi(argv[1]) < 1024) {
return 1;
}
char *port = argv[1];
// parse the command line args for options -l and -N. -l specifies it will use a log and the following parameter is the filename. -N specifies the number of threads it will use and the following parameter will be a number
int opt;
uint8_t threadNum = 1;
char *logName = NULL;
while ((opt = getopt(argc - 1, argv + 1, "N:l:")) != -1) {
if (opt == 'N') {
threadNum = atoi(optarg);
}
else if (opt == 'l') {
logName = optarg;
}
}
struct sockaddr_in server_addr;
memset(&server_addr, 0, sizeof(server_addr));
server_addr.sin_family = AF_INET;
server_addr.sin_port = htons(atoi(port));
server_addr.sin_addr.s_addr = htonl(INADDR_ANY);
socklen_t addrlen = sizeof(server_addr);
// Create server socket
int server_sockd = socket(AF_INET, SOCK_STREAM, 0);
// Need to check if server_sockd < 0, meaning an error
if (server_sockd < 0) {
perror("socket");
return 1;
}
// Configure server socket
int enable = 1;
// This allows you to avoid: 'Bind: Address Already in Use' error
int ret = setsockopt(server_sockd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(enable));
if (ret < 0) {
return EXIT_FAILURE;
}
// Bind server address to socket that is open
ret = bind(server_sockd, (struct sockaddr *) &server_addr, addrlen);
if (ret < 0) {
return EXIT_FAILURE;
}
// Listen for incoming connections
ret = listen(server_sockd, 5); // 5 should be enough, if not use SOMAXCONN
if (ret < 0) {
return EXIT_FAILURE;
}
struct threadObject thread[threadNum];
// Connecting with a client
struct sockaddr client_addr;
socklen_t client_addrlen = sizeof(client_addr);
// create a pthread array of size (number of threads). specify this will be using the handle connections function. join the threads together
#if 0
pthread_t thread_id[threadNum];
#endif
#if 1
struct threadObject *tsk = NULL;
int tskidx;
// clear out the thread structs
for (tskidx = 0; tskidx < threadNum; tskidx++) {
tsk = &thread[tskidx];
memset(tsk,0,sizeof(struct threadObject));
}
while (true) {
// accept connection
int client_sockd = accept(server_sockd, &client_addr, &client_addrlen);
pthread_mutex_lock(&global_mutex);
// join any previously completed threads
for (tskidx = 0; tskidx < threadNum; tskidx++) {
tsk = &thread[tskidx];
if (tsk->tsk_done) {
pthread_join(tsk->tsk_threadid,NULL);
tsk->tsk_inuse = 0;
tsk->tsk_done = 0;
}
}
// find unused task slot
for (tskidx = 0; tskidx < threadNum; tskidx++) {
tsk = &thread[tskidx];
if (! tsk->tsk_inuse)
break;
}
memset(tsk,0,sizeof(struct threadObject));
tsk->client_sockd = client_sockd;
tsk->tsk_inuse = 1;
pthread_mutex_unlock(&global_mutex);
// fire in the hole ...
pthread_create(&tsk->tsk_threadid, NULL, handle_connections, tsk);
}
#endif
#if 0
for (int i = 0; i < threadNum; i++) {
printf("\n[+] server is waiting...\n");
thread[i].client_sockd = accept(server_sockd, &client_addr, &client_addrlen);
handle_connections(&thread[i]);
// pthread_create(&thread_id[i], NULL, handle_connections, &thread[i]);
printf("Response Sent\n");
// close the current client socket
close(thread[i].client_sockd);
}
}
#endif
return EXIT_SUCCESS;
}
Here's the complete code [just in case I clipped out too much]:
#include <sys/socket.h>
#include <sys/stat.h>
#include <stdio.h>
#include <netinet/in.h>
#include <netinet/ip.h>
#include <fcntl.h>
#include <unistd.h> // write
#include <string.h> // memset
#include <stdlib.h> // atoi
#include <stdbool.h> // true, false
#include <errno.h>
#include <sys/types.h>
#include <ctype.h>
#include <pthread.h>
#define BUFFER_SIZE 4096
#if 1
pthread_mutex_t global_mutex = PTHREAD_MUTEX_INITIALIZER;
#endif
struct threadObject {
char method[5]; // PUT, HEAD, GET. HEAD==4 letters+null terminator
char filename[28]; // what is the file we are worried about. Max 27 ASCII characters (NULL terminated on 28)
char httpversion[9]; // HTTP/1.1
ssize_t content_length; // example: 13
uint16_t status_code; // status code for the request
char buffer[BUFFER_SIZE]; // buffer to transfer data
char rest_of_PUT[BUFFER_SIZE]; // incase client send part of PUT message in header
int client_sockd;
pthread_mutex_t *dispatch_lock;
const pthread_cond_t *job_pool_empty;
// pthread_mutex_t* log_lock;
// const pthread_cond_t* log_pool_empty;
pthread_mutex_t *read_write_lock;
pthread_cond_t *file_list_update;
// JobQueue* job_pool;
// LogQueue log_pool;
// bool is_logging;
#if 1
pthread_t tsk_threadid;
int tsk_inuse;
int tsk_done;
#endif
};
//read in the header and store it in the appropriate places
bool
read_http_response(struct threadObject *thread)
{
printf("\nThis function will take care of reading message\n");
// how many bytes we're receiving from the header. also puts the message into the buffer
ssize_t bytes = recv(thread->client_sockd, thread->buffer, BUFFER_SIZE, 0);
// if nothing or too much gets sent in the header, return
if (bytes <= 0 || bytes >= BUFFER_SIZE) {
thread->status_code = 400;
printf("Too long or nothing in here\n");
return false;
}
// NULL terminate the last spot on the buffer
thread->buffer[bytes] = '\0';
// how many bytes we received
printf("[+] received %ld bytes from client\n[+] response: \n", bytes);
printf("those bytes are: %s\n", thread->buffer);
// make a char pointer pointer to the buffer to easily traverse it and parse it into the right spots
char *traverse = thread->buffer;
// first stop. sgnals the beginning of the filename
char *file = strstr(traverse, "/");
// 2nd stop. signls the beginning of the HTTP version. only 1.1 is accepted
char *http = strstr(traverse, "HTTP/1.1");
// 3rd stop. Signals the beginning of the content length
char *contlength1 = strstr(traverse, "Content-Length");
char *chunked = strstr(traverse, "chunked");
if (chunked != NULL) {
printf("MESSAGE NOT A FILE PUT\n");
thread->status_code = 403;
return false;
}
// store the method
sscanf(traverse, "%s", thread->method);
printf("method:%s\n", thread->method);
// if its not 1 of the 3 valid requests, throw 400 error
if (strcmp(thread->method, "GET") != 0 &&
strcmp(thread->method, "PUT") != 0 &&
strcmp(thread->method, "HEAD") != 0) {
thread->status_code = 400;
printf("Invalid Method:%s\n", thread->method);
return false;
}
// if the filename doesnt start with /, its invalid throw 400 error
if (*file != '/') {
thread->status_code = 400;
printf("bad filename\n");
return false;
}
// only store the filename portion after the required /
traverse = file + 1;
// to make sure the filename isnt too long
uint8_t size_check = 0;
// traverse filename until first whitespace
while (*traverse != ' ') {
// if any character in the filename isnt 1 of these, its invalid. throw 400 error
if (!isalnum(*traverse) && *traverse != '_' && *traverse != '-') {
// if theres no filename at all, throw a 404 error
if (size_check == 0) {
thread->status_code = 404;
printf("No file specified\n");
return thread->status_code;
}
thread->status_code = 400;
printf("Invalid filename character:%c\n", *traverse);
return false;
}
sscanf(traverse++, "%c", thread->filename + size_check++);
// if the filename breaks the 27 character limit, return a 400 error
if (size_check > 27) {
thread->status_code = 400;
printf("filename too long\n");
return false;
}
}
printf("filename:%s\n", thread->filename);
// if HTTP/1.1 isnt given, throw a 400 error
if (http == NULL) {
printf("HTTP/1.1 400 Bad Request\r\n\r\n");
thread->status_code = 400;
return false;
}
traverse = http;
// read in the http version until the first \r\n. this signals the end of the given version name
sscanf(traverse, "%[^\r\n]s", thread->httpversion);
printf("HTTP:%s\n", thread->httpversion);
// if its not a put request, this is the end of the header. return
if (strcmp(thread->method, "PUT") != 0) {
return true;
}
// for put requests only. traverse until the beginning of the content length
traverse = contlength1;
// last stop. signals the end of a normal PUT header. if a client wants to put some of the message in the header, it gets stored after this
char *end = strstr(traverse, "\r\n\r\n");
// if theres no \r\n\r\n, the header is bad. return 400
if (end == NULL) {
printf("bad header\n");
thread->status_code = 400;
return false;
}
// traverse to the next digit
while (!isdigit(*traverse)) {
// if theres no next digit after "content length", the header is bad. return 400
if (traverse == end) {
printf("bad header\n");
thread->status_code = 400;
return false;
}
traverse++;
}
// set to traverse to be sure fit the entire content length. use size_check to traverse through
char *temp = traverse;
size_check = 0;
// while its taking in digits, put them into the char array.
while (isdigit(*traverse)) {
sscanf(traverse++, "%c", temp + size_check++);
}
// convert the new string into numbers
thread->content_length = atoi(temp);
// if the content length is < 0 throw a 400 error
if (thread->content_length < 0) {
thread->status_code = 400;
printf("bad content length:%ld\n", thread->content_length);
return false;
}
// printf("Content Length:%ld\n", thread->content_length);
// move +4 spots to get to the end of this. if its a normal PUT, this will be the last spot. If the client puts part of the message in the header, it goes after this
traverse = end + 4;
// put the rest of the header into a char array to append later. if theres nothing, itll do nothing
strcpy(thread->rest_of_PUT, traverse);
// printf("Rest of PUT:%s\n", thread->rest_of_PUT);
// will only get here if status code is 0
return true;
}
//process the message we just recieved
void
process_request(struct threadObject *thread)
{
printf("\nProcessing Request\n");
// server side file descriptor
int fd;
// if the method is PUT
if (strcmp(thread->method, "PUT") == 0) {
// open the file for read only to check if its already there or not to set proper status code
fd = open(thread->filename, O_WRONLY);
// if it doesnt exist, set 201 status code
struct stat checkExist;
if (stat(thread->filename, &checkExist) != 0) {
thread->status_code = 201;
}
// if it exists, set 200 and overwrite
else {
struct stat fileStat;
fstat(fd, &fileStat);
// check write permission
if ((S_IWUSR & fileStat.st_mode) == 0) {
printf("MESSAGE NOT WRITEABLE PUT\n");
thread->status_code = 403;
return;
}
thread->status_code = 200;
}
// close it
close(fd);
// reopen it. this time for writing to or overwriting. if its there, overwrite it. if not, create it. cant use for status codes since it will always create a new file
fd = open(thread->filename, O_WRONLY | O_CREAT | O_TRUNC);
// printf("fd in process is:%d\n", fd);
// if theres a bad fd, throw a 403
if (fd < 0) {
printf("ERROR\n\n");
thread->status_code = 403;
return;
}
// to check that the amount of bytes sent = the amount received
ssize_t bytes_recv,
bytes_send;
// if theres no body, put an empty file on the server
if (thread->content_length == 0) {
bytes_send = write(fd, '\0', 0);
}
// if there is a body, put it onto the new file created on the server and make sure the received bytes = the sent ones
else {
ssize_t total = 0,
len_track = thread->content_length;
while (thread->content_length != 0) {
bytes_recv = recv(thread->client_sockd, thread->buffer, BUFFER_SIZE, 0);
bytes_send = write(fd, thread->buffer, bytes_recv);
total += bytes_send;
// if the received bytes != the sent byes, send a 500 error
if (bytes_recv != bytes_send) {
thread->status_code = 500;
printf("Recieved != sent for put request\n");
return;
}
thread->content_length -= bytes_recv;
// printf("Bytes read:%ld\nBytes sent:%ld\nMessage content length:%ld\n", bytes_recv, bytes_send, message->content_length);
}
// if the content length != bytes sent, throw a 403 error
if (len_track != total) {
thread->status_code = 403;
printf("Content length != sent for put request\n");
return;
}
}
printf("Message status code:%d\n", thread->status_code);
// close the fd
close(fd);
return;
}
// if the method is GET or HEAD
else if (strcmp(thread->method, "GET") == 0 || strcmp(thread->method, "HEAD") == 0) {
// open the file for reading only
fd = open(thread->filename, O_RDONLY);
// if bad fd, throw a 404
struct stat fileStat;
fstat(fd, &fileStat);
// check read permission and if it exists
if (((S_IRUSR & fileStat.st_mode) == 0) || stat(thread->filename, &fileStat) != 0) {
printf("BAD GET\n");
thread->status_code = 404;
return;
}
else {
thread->status_code = 200;
thread->content_length = lseek(fd, 0, SEEK_END);
}
// close the fd
close(fd);
return;
}
}
void
construct_http_response(struct threadObject *thread)
{
printf("Constructing Response\n");
// size 22 since the largest code is 21 characters + NULL
char response[22];
// 200=OK, 201=CREATED, 400=BAD REQUEST, 403=FORBIDDEN, 404=NOT FOUND, 500=INTERNAL SERVER ERROR
if (thread->status_code == 200) {
strcpy(response, "OK");
}
else if (thread->status_code == 201) {
strcpy(response, "CREATED");
}
else if (thread->status_code == 400) {
strcpy(response, "BAD REQUEST");
}
else if (thread->status_code == 403) {
strcpy(response, "FORBIDDEN");
}
else if (thread->status_code == 404) {
strcpy(response, "NOT FOUND");
}
else if (thread->status_code == 500) {
strcpy(response, "INTERNAL SERVER ERROR");
}
else {
printf("Bad response...\n");
return;
}
dprintf(thread->client_sockd, "%s %d %s\r\nContent-Length: %ld\r\n\r\n", thread->httpversion, thread->status_code, response, thread->content_length);
if (strcmp(thread->method, "GET") == 0 && thread->status_code == 200) {
int fd = open(thread->filename, O_RDONLY);
ssize_t total = 0,
len_track = thread->content_length,
bytes_recv,
bytes_send;
while (thread->content_length != 0) {
bytes_recv = read(fd, thread->buffer, BUFFER_SIZE);
bytes_send = send(thread->client_sockd, thread->buffer, bytes_recv, 0);
if (bytes_recv != bytes_send) {
thread->status_code = 500;
close(fd);
printf("Recieved != sent for GET request\nReceived:%ld\nSent:%ld\n", bytes_recv, bytes_send);
dprintf(thread->client_sockd, "%s %d %s\r\nContent-Length: %ld\r\n\r\n", thread->httpversion, thread->status_code, response, thread->content_length);
close(fd);
return;
}
total += bytes_send;
thread->content_length -= bytes_recv;
}
if (total != len_track) {
thread->status_code = 403;
printf("Content length != recvd for GET request\n");
dprintf(thread->client_sockd, "%s %d %s\r\nContent-Length: %ld\r\n\r\n", thread->httpversion, thread->status_code, response, thread->content_length);
close(fd);
return;
}
close(fd);
}
}
void *
handle_connections(void *ptr_thread)
{
// create a mutual exclusion to lock out any other threads from the function
// pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
// pthread_mutex_lock(&mutex);
// operations go here
struct threadObject *thread = (struct threadObject *) ptr_thread;
// reset message after each loop
memset(thread->buffer, '\0', BUFFER_SIZE);
memset(thread->method, '\0', 5);
memset(thread->filename, '\0', 28);
memset(thread->httpversion, '\0', 9);
thread->content_length = 0;
thread->status_code = 0;
memset(thread->rest_of_PUT, '\0', BUFFER_SIZE);
// read message
if (read_http_response(thread) == true) {
// process message
process_request(thread);
}
// construct a response
construct_http_response(thread);
// unlock the function
// pthread_mutex_unlock(&mutex);
#if 1
close(thread->client_sockd);
pthread_mutex_lock(&global_mutex);
thread->tsk_done = 1;
pthread_mutex_unlock(&global_mutex);
#endif
return NULL;
}
int
main(int argc, char **argv)
{
// Create sockaddr_in with server information
if (argc < 2) {
perror("No arguments passed\n");
return -1;
}
// make sure port number is above 1024 and set the port # to it
if (atoi(argv[1]) < 1024) {
return 1;
}
char *port = argv[1];
// parse the command line args for options -l and -N. -l specifies it will use a log and the following parameter is the filename. -N specifies the number of threads it will use and the following parameter will be a number
int opt;
uint8_t threadNum = 1;
char *logName = NULL;
while ((opt = getopt(argc - 1, argv + 1, "N:l:")) != -1) {
if (opt == 'N') {
threadNum = atoi(optarg);
}
else if (opt == 'l') {
logName = optarg;
}
}
struct sockaddr_in server_addr;
memset(&server_addr, 0, sizeof(server_addr));
server_addr.sin_family = AF_INET;
server_addr.sin_port = htons(atoi(port));
server_addr.sin_addr.s_addr = htonl(INADDR_ANY);
socklen_t addrlen = sizeof(server_addr);
// Create server socket
int server_sockd = socket(AF_INET, SOCK_STREAM, 0);
// Need to check if server_sockd < 0, meaning an error
if (server_sockd < 0) {
perror("socket");
return 1;
}
// Configure server socket
int enable = 1;
// This allows you to avoid: 'Bind: Address Already in Use' error
int ret = setsockopt(server_sockd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(enable));
if (ret < 0) {
return EXIT_FAILURE;
}
// Bind server address to socket that is open
ret = bind(server_sockd, (struct sockaddr *) &server_addr, addrlen);
if (ret < 0) {
return EXIT_FAILURE;
}
// Listen for incoming connections
ret = listen(server_sockd, 5); // 5 should be enough, if not use SOMAXCONN
if (ret < 0) {
return EXIT_FAILURE;
}
struct threadObject thread[threadNum];
// Connecting with a client
struct sockaddr client_addr;
socklen_t client_addrlen = sizeof(client_addr);
// create a pthread array of size (number of threads). specify this will be using the handle connections function. join the threads together
#if 0
pthread_t thread_id[threadNum];
#endif
#if 1
struct threadObject *tsk = NULL;
int tskidx;
// clear out the thread structs
for (tskidx = 0; tskidx < threadNum; tskidx++) {
tsk = &thread[tskidx];
memset(tsk,0,sizeof(struct threadObject));
}
while (true) {
// accept connection
int client_sockd = accept(server_sockd, &client_addr, &client_addrlen);
pthread_mutex_lock(&global_mutex);
// join any previously completed threads
for (tskidx = 0; tskidx < threadNum; tskidx++) {
tsk = &thread[tskidx];
if (tsk->tsk_done) {
pthread_join(tsk->tsk_threadid,NULL);
tsk->tsk_inuse = 0;
tsk->tsk_done = 0;
}
}
// find unused task slot
for (tskidx = 0; tskidx < threadNum; tskidx++) {
tsk = &thread[tskidx];
if (! tsk->tsk_inuse)
break;
}
memset(tsk,0,sizeof(struct threadObject));
tsk->client_sockd = client_sockd;
tsk->tsk_inuse = 1;
pthread_mutex_unlock(&global_mutex);
// fire in the hole ...
pthread_create(&tsk->tsk_threadid, NULL, handle_connections, tsk);
}
#endif
#if 0
for (int i = 0; i < threadNum; i++) {
printf("\n[+] server is waiting...\n");
thread[i].client_sockd = accept(server_sockd, &client_addr, &client_addrlen);
handle_connections(&thread[i]);
// pthread_create(&thread_id[i], NULL, handle_connections, &thread[i]);
printf("Response Sent\n");
// close the current client socket
close(thread[i].client_sockd);
}
}
#endif
return EXIT_SUCCESS;
}

How to programmatically get PID of process connecting to my proxy via AF_INET sockets, on the same machine?

I am writing a small http proxy server(in C) on a linux machine, Ubuntu 18.04.1 to be specific, and I've been trying to find a way to get the pid of the process that is connecting to it.
It might be of use to mention that the proxy is intended to proxy connections only for processes running on the same machine, so I guess this should make this task possible.
The server uses AF_INET family sockets along with read/write operations in order to do it's job; I am mentioning this because after some research I did encounter threads about "ancillary data",for example: Is there a way to get the uid of the other end of a unix socket connection
Ancillary data contain credentials of the connecting socket(such as PID), but only work on AF_UNIX sockets, used for local IPC, and requires us to explicitly send/receive it on both sides(client/server). In my case, although, as I mentioned, the server will only proxy traffic on the same machine as the server, I need to use AF_INET sockets, so everyone(e.g. web browser) is able to connect to it.
Performance is not so critical; so any suggestions(including workarounds using system calls etc.) are very welcome.
We can use netstat -nptW output to see which local processes' TCP connections. As the output may be security sensitive, superuser privileges are required to see processes belonging to all users.
Since there is no reason to run a proxy service with elevated privileges (expect perhaps CAP_NET_BIND_SERVICE), a privileged helper program is needed.
I pondered a suitable security model for a bit, and came to the conclusion that a helper which examines the connected socket given to it (as say standard input), and outputs just the peer PID(s), would be safest: it would be extremely hard to misuse it, and even if possible, only the peer process ID is revealed.
Here is the example helper, tcp-peer-pids.c:
#define _POSIX_C_SOURCE 200809L
#define _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>
#define EXITCODE_OK 0
#define EXITCODE_STDIN_INVALID 1
#define EXITCODE_UNKNOWN_ADDRESS 2
#define EXITCODE_NETSTAT 3
#define EXITCODE_NETSTAT_OUTPUT 4
#define EXITCODE_WRITE_ERROR 5
#define EXITCODE_PRIVILEGES 6
static pid_t *pids = NULL;
static size_t num_pids = 0;
static size_t max_pids = 0;
static int add_pid(const pid_t p)
{
size_t i;
/* Check if already listed. */
for (i = 0; i < num_pids; i++)
if (pids[i] == p)
return 0;
/* Ensure enough room in pids array. */
if (num_pids >= max_pids) {
const size_t max_temp = (num_pids | 1023) + 1025 - 8;
pid_t *temp;
temp = realloc(pids, max_temp * sizeof pids[0]);
if (!temp)
return ENOMEM;
pids = temp;
max_pids = max_temp;
}
pids[num_pids++] = p;
return 0;
}
int main(void)
{
struct sockaddr_storage sock_addr;
socklen_t sock_addrlen = sizeof sock_addr;
char sock_match[128], sock_host[64], sock_port[32];
struct sockaddr_storage peer_addr;
socklen_t peer_addrlen = sizeof peer_addr;
char peer_match[128], peer_host[64], peer_port[32];
FILE *cmd;
char *line = NULL;
size_t size = 0;
ssize_t len;
int status;
/* Socket address is *remote*, and peer address is *local*.
This is because the variables are named after their matching netstat lines. */
if (getsockname(STDIN_FILENO, (struct sockaddr *)&sock_addr, &sock_addrlen) == -1) {
fprintf(stderr, "Standard input is not a valid socket.\n");
exit(EXITCODE_STDIN_INVALID);
}
if (getpeername(STDIN_FILENO, (struct sockaddr *)&peer_addr, &peer_addrlen) == -1) {
fprintf(stderr, "Standard input is not a connected socket.\n");
exit(EXITCODE_STDIN_INVALID);
}
if ((sock_addr.ss_family != AF_INET && sock_addr.ss_family != AF_INET6) ||
(peer_addr.ss_family != AF_INET && peer_addr.ss_family != AF_INET6)) {
fprintf(stderr, "Standard input is not an IP socket.\n");
exit(EXITCODE_STDIN_INVALID);
}
/* For security, we close the standard input descriptor, */
close(STDIN_FILENO);
/* and redirect it from /dev/null, if possible. */
{
int fd = open("/dev/null", O_RDONLY);
if (fd != -1 && fd != STDIN_FILENO) {
dup2(fd, STDIN_FILENO);
close(fd);
}
}
/* Convert sockets to numerical host and port strings. */
if (getnameinfo((const struct sockaddr *)&sock_addr, sock_addrlen,
sock_host, sizeof sock_host, sock_port, sizeof sock_port,
NI_NUMERICHOST | NI_NUMERICSERV)) {
fprintf(stderr, "Unknown socket address.\n");
exit(EXITCODE_UNKNOWN_ADDRESS);
}
if (getnameinfo((const struct sockaddr *)&peer_addr, peer_addrlen,
peer_host, sizeof peer_host, peer_port, sizeof peer_port,
NI_NUMERICHOST | NI_NUMERICSERV)) {
fprintf(stderr, "Unknown peer address.\n");
exit(EXITCODE_UNKNOWN_ADDRESS);
}
/* Combine to the host:port format netstat uses. */
snprintf(sock_match, sizeof sock_match, "%s:%s", sock_host, sock_port);
snprintf(peer_match, sizeof peer_match, "%s:%s", peer_host, peer_port);
/* Switch to privileged user, if installed as setuid. */
{
uid_t real_uid = getuid();
gid_t real_gid = getgid();
uid_t effective_uid = geteuid();
gid_t effective_gid = getegid();
if (real_gid != effective_gid || real_uid != effective_uid) {
/* SetUID or SetGID in effect. Switch privileges. */
if (setresgid(effective_gid, effective_gid, effective_gid) == -1 ||
setresuid(effective_uid, effective_uid, effective_uid) == -1) {
fprintf(stderr, "Error in privileges: %s.\n", strerror(errno));
exit(EXITCODE_PRIVILEGES);
}
}
}
/* Run netstat to obtain the data; redirect standard error to standard output. */
cmd = popen("LANG=C LC_ALL=C /bin/netstat -nptW 2>&1", "r");
if (!cmd) {
fprintf(stderr, "Cannot run netstat.\n");
exit(EXITCODE_NETSTAT);
}
/* Input line loop. */
while (1) {
char *field[8], *ends;
long val;
pid_t p;
len = getline(&line, &size, cmd);
if (len < 1)
break;
/* Split each line into fields. */
field[0] = strtok(line, "\t\n\v\f\r "); /* Protocol */
/* We are only interested in tcp ("tcp" and "tcp6" protocols). */
if (strcmp(field[0], "tcp") && strcmp(field[0], "tcp6"))
continue;
field[1] = strtok(NULL, "\t\n\v\f\r "); /* Recv-Q */
field[2] = strtok(NULL, "\t\n\v\f\r "); /* Send-Q */
field[3] = strtok(NULL, "\t\n\v\f\r "); /* Local address (peer) */
field[4] = strtok(NULL, "\t\n\v\f\r "); /* Remote address (sock) */
field[5] = strtok(NULL, "\t\n\v\f\r "); /* State */
field[6] = strtok(NULL, "\t\n\v\f\r /"); /* PID */
field[7] = strtok(NULL, "\t\n\v\f\r "); /* Process name */
/* Local address must match peer_match, and foreign/remote sock_match. */
if (strcmp(field[3], peer_match) || strcmp(field[4], sock_match))
continue;
/* This line corresponds to the process we are looking for. */
/* Missing PID field is an error at this point. */
if (!field[6])
break;
/* Parse the PID. Parsing errors are fatal. */
ends = field[6];
errno = 0;
val = strtol(field[6], &ends, 10);
if (errno || ends == field[6] || *ends != '\0' || val < 1)
break;
p = (pid_t)val;
if ((long)p != val)
break;
/* Add the pid to the known pids list. */
if (add_pid(p))
break;
}
/* The line buffer is no longer needed. */
free(line);
/* I/O error? */
if (!feof(cmd) || ferror(cmd)) {
fprintf(stderr, "Error reading netstat output.\n");
exit(EXITCODE_NETSTAT_OUTPUT);
}
/* Reap the netstat process. */
status = pclose(cmd);
if (status == -1) {
fprintf(stderr, "Error reading netstat output: %s.\n", strerror(errno));
exit(EXITCODE_NETSTAT_OUTPUT);
}
if (!WIFEXITED(status)) {
fprintf(stderr, "Netstat died unexpectedly.\n");
exit(EXITCODE_NETSTAT_OUTPUT);
}
if (WEXITSTATUS(status)) {
fprintf(stderr, "Netstat failed with exit status %d.\n", WEXITSTATUS(status));
exit(EXITCODE_NETSTAT_OUTPUT);
}
/* Output the array of pids as binary data. */
if (num_pids > 0) {
const char *head = (const char *)pids;
const char *const ends = (const char *)(pids + num_pids);
ssize_t n;
while (head < ends) {
n = write(STDOUT_FILENO, head, (size_t)(ends - head));
if (n > 0)
head += n;
else
if (n != -1)
exit(EXITCODE_WRITE_ERROR);
else
if (errno != EINTR)
exit(EXITCODE_WRITE_ERROR);
}
}
/* Discard the pids array. */
free(pids);
exit(EXITCODE_OK);
}
It can be run using ordinary user privileges (in which case it'll only know about processes owned by that user), root privileges, or as setuid root.
If used with sudo, ensure you use rule proxyuser ALL = NOPASSWD: /path/to/helper, because sudo has no way of asking a password there. I would probably just install the helper as setuid root at /usr/lib/yourproxy/tcp-peer-pid, owner root, group your proxy service group, and no access to other users (root:proxygroup -r-sr-x---).
The helper is tightly coupled to netstat -nptW output format, but does explicitly set the C locale to avoid getting localized output.
The comparison address:port strings to match to "Local Address" and "Foreign Address" in netstat output are constructed from the addresses returned by getpeername() and getsockname(), respectively, using [getnameinfo()(http://man7.org/linux/man-pages/man3/getnameinfo.3.html) in numerical form (using NI_NUMERICHOST | NI_NUMERICSERV flags).
The helper provides the PIDs in binary form to the server, because the server code would have been too long to fit in a single post here otherwise.
Here is an example TCP service, server.c, which uses the above helper to find out the PID of the peer end of the socket on the local computer. (To avoid denial-of-service attacks, you should set an IP filter that rejects accesses to your proxy service port from outside the computer.)
#define _POSIX_C_SOURCE 200809L
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <netdb.h>
#include <signal.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>
#ifndef HELPER_PATH
#define HELPER_PATH "./tcp-peer-pids"
#endif
#ifndef HELPER_NAME
#define HELPER_NAME "tcp-peer-pids"
#endif
#ifndef SUDO_PATH
#define SUDO_PATH "/usr/bin/sudo"
#endif
#ifndef SUDO_NAME
#define SUDO_NAME "sudo"
#endif
/*
* Signal handler, to detect INT (Ctrl+C), HUP, and TERM signals.
*/
static volatile sig_atomic_t done = 0;
static void handle_done(int signum)
{
/* In Linux, all signals have signum > 0. */
__atomic_store_n(&done, (sig_atomic_t)signum, __ATOMIC_SEQ_CST);
}
static int install_done(int signum)
{
struct sigaction act;
memset(&act, 0, sizeof act);
sigemptyset(&act.sa_mask);
act.sa_flags = SA_RESTART; /* Do not interrupt slow syscalls. */
act.sa_handler = handle_done;
if (sigaction(signum, &act, NULL) == -1)
return -1; /* errno set by getpeername() */
return 0;
}
/* Helper function: Move descriptors away from STDIN/STDOUT/STDERR.
Returns 0 if successful, -1 with errno set if an error occurs. */
static inline int normalfds(int fd[], const size_t n)
{
unsigned int closemask = 0;
int err = 0;
size_t i;
int newfd;
for (i = 0; i < n; i++)
while (fd[i] == STDIN_FILENO || fd[i] == STDOUT_FILENO || fd[i] == STDERR_FILENO) {
newfd = dup(fd[i]);
if (newfd == -1) {
err = errno;
break;
}
closemask |= 1u << fd[i];
fd[i] = newfd;
}
/* Close temporary descriptors. */
if (closemask & (1u << STDIN_FILENO)) close(STDIN_FILENO);
if (closemask & (1u << STDOUT_FILENO)) close(STDOUT_FILENO);
if (closemask & (1u << STDERR_FILENO)) close(STDERR_FILENO);
/* Success? */
if (!err)
return 0;
/* Report error. */
errno = err;
return -1;
}
/* Return the number of peer processes.
If an error occurs, returns zero; examine errno. */
size_t peer_pids(const int connfd, pid_t *const pids, size_t maxpids)
{
char *in_data = NULL;
size_t in_size = 0;
size_t in_used = 0;
size_t n;
int binpipe[2], status;
pid_t child, p;
/* Sanity check. */
if (connfd == -1) {
errno = EBADF;
return 0;
}
/* Create a pipe to transfer the PIDs (in binary). */
if (pipe(binpipe) == -1)
return 0; /* errno set by pipe(). */
/* Make sure the binary pipe descriptors do not conflict with standard descriptors. */
if (normalfds(binpipe, 2) == -1) {
const int saved_errno = errno;
close(binpipe[0]);
close(binpipe[1]);
errno = saved_errno;
return 0;
}
/* Fork a child process. */
child = fork();
if (child == -1) {
const int saved_errno = errno;
close(binpipe[0]);
close(binpipe[1]);
errno = saved_errno;
return 0;
}
if (!child) {
/* This is the child process. */
#ifdef USE_SUDO
const char *cmd_path = SUDO_PATH;
char *const cmd_args[3] = { SUDO_NAME, HELPER_PATH, NULL };
#else
const char *cmd_path = HELPER_PATH;
char *const cmd_args[2] = { HELPER_NAME, NULL };
#endif
/* The child runs in its own process group, for easier management. */
setsid();
/* Close read end of pipe. */
close(binpipe[0]);
/* Move established connection to standard input. */
if (connfd != STDIN_FILENO) {
if (dup2(connfd, STDIN_FILENO) != STDIN_FILENO)
_Exit(99);
close(connfd);
}
/* Move write end of pipe to standard output. */
if (dup2(binpipe[1], STDOUT_FILENO) != STDOUT_FILENO)
_Exit(99);
else
close(binpipe[1]);
/* Execute helper. */
execv(cmd_path, cmd_args);
/* Failed to execute helper. */
_Exit(98);
}
/* Parent process. */
/* Close write end of pipe, so we detect when child exits. */
close(binpipe[1]);
/* Read all output from child. */
status = 0;
while (1) {
ssize_t bytes;
if (in_used >= in_size) {
const size_t size = (in_used | 1023) + 1025 - 8;
char *temp;
temp = realloc(in_data, in_size);
if (!temp) {
status = ENOMEM;
break;
}
in_data = temp;
in_size = size;
}
bytes = read(binpipe[0], in_data + in_used, in_size - in_used);
if (bytes > 0) {
in_used += bytes;
} else
if (bytes == 0) {
/* End of input condition. */
break;
} else
if (bytes != -1) {
status = EIO;
break;
} else
if (errno != EINTR) {
status = errno;
break;
}
}
/* Close the pipe. */
close(binpipe[0]);
/* Abort, if an error occurred. */
if (status) {
free(in_data);
kill(-child, SIGKILL);
do {
p = waitpid(child, NULL, 0);
} while (p == -1 && errno == EINTR);
errno = status;
return 0;
}
/* Reap the child process. */
do {
status = 0;
p = waitpid(child, &status, 0);
} while (p == -1 && errno == EINTR);
if (p == -1) {
const int saved_errno = errno;
free(in_data);
errno = saved_errno;
return 0;
}
if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) {
free(in_data);
errno = ESRCH; /* The helper command failed, really. */
return 0;
}
/* We expect an integer number of pid_t's. Check. */
n = in_used / sizeof (pid_t);
if ((in_used % sizeof (pid_t)) != 0) {
free(in_data);
errno = EIO;
return 0;
}
/* None found? */
if (!n) {
free(in_data);
errno = ENOENT; /* Not found, really. */
return 0;
}
/* Be paranoid, and verify the pids look sane. */
{
const pid_t *const pid = (const pid_t *const)in_data;
size_t i;
for (i = 0; i < n; i++)
if (pid[i] < 2) {
free(in_data);
errno = ESRCH; /* Helper failed */
return 0;
}
}
/* Copy to user buffer, if specified. */
if (maxpids > n)
memcpy(pids, in_data, n * sizeof (pid_t));
else
if (maxpids > 0)
memcpy(pids, in_data, maxpids * sizeof (pid_t));
/* The pid buffer is no longer needed. */
free(in_data);
/* Return the number of pids we actually received. */
return n;
}
int main(int argc, char *argv[])
{
struct addrinfo hints, *list, *curr;
const char *node, *serv;
int service_fd, err;
struct sockaddr_storage client_addr;
socklen_t client_addrlen;
int client_fd;
if (argc != 3) {
fprintf(stderr, "\n");
fprintf(stderr, "Usage: %s [ -h | --help ]\n", argv[0]);
fprintf(stderr, " %s HOST PORT\n", argv[0]);
fprintf(stderr, "\n");
return EXIT_FAILURE;
}
/* Install signal handers for Ctrl+C, HUP, and TERM. */
if (install_done(SIGINT) ||
install_done(SIGHUP) ||
install_done(SIGTERM)) {
fprintf(stderr, "Cannot install signal handlers: %s.\n", strerror(errno));
return EXIT_FAILURE;
}
/* Empty or - or * is a wildcard host. */
if (argv[1][0] == '\0' || !strcmp(argv[1], "-") || !strcmp(argv[1], "*"))
node = NULL;
else
node = argv[1];
serv = argv[2];
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_UNSPEC; /* IPv4 or IPv6 */
hints.ai_socktype = SOCK_STREAM; /* TCP */
hints.ai_flags = AI_PASSIVE;
hints.ai_protocol = 0;
hints.ai_canonname = NULL;
hints.ai_addr = NULL;
hints.ai_next = NULL;
list = NULL;
err = getaddrinfo(node, serv, &hints, &list);
if (err) {
fprintf(stderr, "Invalid host and/or port: %s.\n", gai_strerror(err));
return EXIT_FAILURE;
}
service_fd = -1;
err = 0;
for (curr = list; curr != NULL; curr = curr->ai_next) {
service_fd = socket(curr->ai_family, curr->ai_socktype, curr->ai_protocol);
if (service_fd == -1)
continue;
errno = 0;
if (bind(service_fd, curr->ai_addr, curr->ai_addrlen) == -1) {
if (!err)
if (errno == EADDRINUSE || errno == EADDRNOTAVAIL || errno == EACCES)
err = errno;
close(service_fd);
service_fd = -1;
continue;
}
if (listen(service_fd, 5) == -1) {
if (!err)
if (errno == EADDRINUSE)
err = errno;
close(service_fd);
service_fd = -1;
continue;
}
/* This socket works. */
break;
}
freeaddrinfo(list);
list = curr = NULL;
if (service_fd == -1) {
if (err)
fprintf(stderr, "Cannot listen for incoming connections on the specified host and port: %s.\n", strerror(err));
else
fprintf(stderr, "Cannot listen for incoming connections on the specified host and port.\n");
return EXIT_FAILURE;
}
/* Do not leak the listening socket to child processes. */
fcntl(service_fd, F_SETFD, FD_CLOEXEC);
/* We also want the listening socket to be nonblocking. */
fcntl(service_fd, F_SETFL, O_NONBLOCK);
fprintf(stderr, "Process %ld is waiting for incoming TCP connections.\n", (long)getpid());
/* Incoming connection loop. */
while (!done) {
struct timeval t;
char client_host[64]; /* 64 for numeric, 1024 for non-numeric */
char client_port[32];
pid_t client_pid;
fd_set fds;
t.tv_sec = 0;
t.tv_usec = 100000; /* Max. 0.1s delay to react to done signal. */
FD_ZERO(&fds);
FD_SET(service_fd, &fds);
if (select(service_fd + 1, &fds, NULL, NULL, &t) < 1)
continue;
client_addrlen = sizeof client_addr;
client_fd = accept(service_fd, (struct sockaddr *)&client_addr, &client_addrlen);
if (client_fd == -1) {
if (errno == EINTR || errno == ECONNABORTED)
continue;
fprintf(stderr, "Error accepting an incoming connection: %s.\n", strerror(errno));
continue;
}
if (getnameinfo((const struct sockaddr *)&client_addr, client_addrlen,
client_host, sizeof client_host, client_port, sizeof client_port,
NI_NUMERICHOST | NI_NUMERICSERV) != 0) {
fprintf(stderr, "Cannot resolve peer address for incoming connection, so dropping it.\n");
close(client_fd);
continue;
}
printf("Incoming connection from %s:%s", client_host, client_port);
fflush(stdout);
if (peer_pids(client_fd, &client_pid, 1) != 1) {
printf(", but cannot determine process ID. Dropped.\n");
close(client_fd);
continue;
}
printf(" from local process %ld.\n", (long)client_pid);
fflush(stdout);
/*
* Handle connection.
*/
printf("Closing connection.\n");
fflush(stdout);
close(client_fd);
}
/* Close service socket. */
close(service_fd);
switch (__atomic_load_n(&done, __ATOMIC_SEQ_CST)) {
case SIGINT:
fprintf(stderr, "Received INT signal.\n");
break;
case SIGHUP:
fprintf(stderr, "Received HUP signal.\n");
break;
case SIGTERM:
fprintf(stderr, "Received TERM signal.\n");
break;
}
return EXIT_SUCCESS;
}
The peer_pids() function communicates with the helper process. It is very straightforward, albeit careful to not return unreliable data: instead of ignoring errors or trying to recover from them, it reports failure. This allows the main program do if (peer_pids(client_fd, &pid, 1) != 1) /* Don't know! */ and drop any connection the server is unsure of -- an approach I consider the sane one here.
The normalfds() helper function is often ignored. It helps avoid issues if any of the standard streams are/get closed. It simply moves the set of descriptors away from the three standard streams, using at most three extra descriptors.
You can define USE_SUDO at compile time to have it use sudo when executing the helper. Define HELPER_PATH and HELPER_NAME to the absolute path to the helper and its file name, respectively. (As it is now, they default to ./tcp-peer-pid and tcp-peer-pid, for easier testing.)
The server does install a signal handler for INT (Ctrl+C), HUP (sent when the user closes the terminal), or TERM signals, which all cause it to stop accepting new connections and exit in a controlled manner. (Because the signal handler is installed using SA_RESTART flag, its delivery will not interrupt slow syscalls or cause errno == EINTR. This also means that accept() should not block, or the signal delivery will not be noticed. So, blocking in select() for 0.1s, and checking if a signal was delivered in between, is a good compromise, at least in an example server.)
On my machine, I compiled and tested the service in one terminal window using
gcc -Wall -O2 tcp-peer-pids.c -o tcp-peer-pids
gcc -Wall -O2 "-DHELPER_PATH=\"$PWD/tcp-peer-pids\"" server.c -o server
./server - 2400
That will report Process # is waiting for incoming TCP connections. In another window, using Bash or POSIX shell, I run one or more test netcat commands:
nc localhost 2400 & wait
It might look silly to run a command in the background, and immediately wait for it, but that way you can see the PID of the nc process.
On my system, all loopback (127.x.y.z), TCP/IPv4, and TCP/IPv6 (the addresses of my ethernet and WiFi interfaces) worked fine, and reliably reported the correct PID of the process connecting to the example server.
There are a number of cases where the number of PIDs reported might vary: For example, if the program has executed a child process, but left the connected descriptor open in the child as well. (This should be considered a bug.) Another typical case is the program having exited before the netstat command executes.
If you find any typos or errors or strange behaviour, let me know in a comment so I can verify and fix. I wrote both programs in one sitting, so they are quite likely to contain bugs. As I mentioned, I would not trust either in production before having a colleague (or myself a few times, later on, with fresh eyes) going through it with a critical/paranoid eye.
I would personally only use this approach for logging and statistics, not access control per se. By access control, I mean that you should configure an IP filter (the firewall built in to the Linux kernel) to limit access to only trusted hosts; and specifically allow no incoming proxy connections to the proxy service if only local applications are to be proxied, rather than rely on this detecting all remote connections.
For application-specific logging/limiting, use readlink() on the /proc/PID/exe pseudosymlink. This cannot be faked, but the call may fail if the executable is not accessible, or is too deep in the directory tree. (In those cases I'd reject the proxy connection altogether.)
Note that it is usually trivial for an user to copy an executable to any directory they own, and execute it from there. This means that for application-specific limiting to work at all, you should have tight limits for all applications by default, and relax the limits for specific executables.

Windows API: Wait for data to be available on non-GUI console input (PIPE-based STDIN)

Background
I am currently working on a Windows select-like function that not only supports SOCKET handles, but also other kinds of waitable handles. My goal is to wait on standard console handles in order to provide select-functionality to the curl testsuite.
The related program can be found in the curl git repository: sockfilt.c
Question
Is it possible to wait for data to be available on a non-GUI-based console input? The issue is that WaitFor* methods do not support PIPE handles and therefore STDIN is not supported if the process input is fed from another process, e.g. using the pipe | functionality of cmd.
The following example program illustrates the problem: select_ws.c
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <windows.h>
#include <winsock2.h>
#include <malloc.h>
#include <conio.h>
#include <fcntl.h>
#define SET_SOCKERRNO(x) (WSASetLastError((int)(x)))
typedef SOCKET curl_socket_t;
/*
* select function with support for WINSOCK2 sockets and all
* other handle types supported by WaitForMultipleObjectsEx.
* http://msdn.microsoft.com/en-us/library/windows/desktop/ms687028.aspx
* http://msdn.microsoft.com/en-us/library/windows/desktop/ms741572.aspx
*/
static int select_ws(int nfds, fd_set *readfds, fd_set *writefds,
fd_set *exceptfds, struct timeval *timeout)
{
long networkevents;
DWORD milliseconds, wait, idx, avail, events, inputs;
WSAEVENT wsaevent, *wsaevents;
WSANETWORKEVENTS wsanetevents;
INPUT_RECORD *inputrecords;
HANDLE handle, *handles;
curl_socket_t sock, *fdarr, *wsasocks;
int error, fds;
DWORD nfd = 0, wsa = 0;
int ret = 0;
/* check if the input value is valid */
if(nfds < 0) {
SET_SOCKERRNO(EINVAL);
return -1;
}
/* check if we got descriptors, sleep in case we got none */
if(!nfds) {
Sleep((timeout->tv_sec * 1000) + (timeout->tv_usec / 1000));
return 0;
}
/* allocate internal array for the original input handles */
fdarr = malloc(nfds * sizeof(curl_socket_t));
if(fdarr == NULL) {
SET_SOCKERRNO(ENOMEM);
return -1;
}
/* allocate internal array for the internal event handles */
handles = malloc(nfds * sizeof(HANDLE));
if(handles == NULL) {
SET_SOCKERRNO(ENOMEM);
return -1;
}
/* allocate internal array for the internal socket handles */
wsasocks = malloc(nfds * sizeof(curl_socket_t));
if(wsasocks == NULL) {
SET_SOCKERRNO(ENOMEM);
return -1;
}
/* allocate internal array for the internal WINSOCK2 events */
wsaevents = malloc(nfds * sizeof(WSAEVENT));
if(wsaevents == NULL) {
SET_SOCKERRNO(ENOMEM);
return -1;
}
/* loop over the handles in the input descriptor sets */
for(fds = 0; fds < nfds; fds++) {
networkevents = 0;
handles[nfd] = 0;
if(FD_ISSET(fds, readfds))
networkevents |= FD_READ|FD_ACCEPT|FD_CLOSE;
if(FD_ISSET(fds, writefds))
networkevents |= FD_WRITE|FD_CONNECT;
if(FD_ISSET(fds, exceptfds))
networkevents |= FD_OOB;
/* only wait for events for which we actually care */
if(networkevents) {
fdarr[nfd] = (curl_socket_t)fds;
if(fds == fileno(stdin)) {
handles[nfd] = GetStdHandle(STD_INPUT_HANDLE);
}
else if(fds == fileno(stdout)) {
handles[nfd] = GetStdHandle(STD_OUTPUT_HANDLE);
}
else if(fds == fileno(stderr)) {
handles[nfd] = GetStdHandle(STD_ERROR_HANDLE);
}
else {
wsaevent = WSACreateEvent();
if(wsaevent != WSA_INVALID_EVENT) {
error = WSAEventSelect(fds, wsaevent, networkevents);
if(error != SOCKET_ERROR) {
handles[nfd] = wsaevent;
wsasocks[wsa] = (curl_socket_t)fds;
wsaevents[wsa] = wsaevent;
wsa++;
}
else {
handles[nfd] = (HANDLE)fds;
WSACloseEvent(wsaevent);
}
}
}
nfd++;
}
}
/* convert struct timeval to milliseconds */
if(timeout) {
milliseconds = ((timeout->tv_sec * 1000) + (timeout->tv_usec / 1000));
}
else {
milliseconds = INFINITE;
}
/* wait for one of the internal handles to trigger */
wait = WaitForMultipleObjectsEx(nfd, handles, FALSE, milliseconds, FALSE);
/* loop over the internal handles returned in the descriptors */
for(idx = 0; idx < nfd; idx++) {
fds = fdarr[idx];
handle = handles[idx];
sock = (curl_socket_t)fds;
/* check if the current internal handle was triggered */
if(wait != WAIT_FAILED && (wait - WAIT_OBJECT_0) >= idx &&
WaitForSingleObjectEx(handle, 0, FALSE) == WAIT_OBJECT_0) {
/* try to handle the event with STD* handle functions */
if(fds == fileno(stdin)) {
/* check if there is no data in the input buffer */
if(!stdin->_cnt) {
/* check if we are getting data from a PIPE */
if(!GetConsoleMode(handle, &avail)) {
/* check if there is no data from PIPE input */
if(!PeekNamedPipe(handle, NULL, 0, NULL, &avail, NULL))
avail = 0;
if(!avail)
FD_CLR(sock, readfds);
} /* check if there is no data from keyboard input */
else if (!_kbhit()) {
/* check if there are INPUT_RECORDs in the input buffer */
if(GetNumberOfConsoleInputEvents(handle, &events)) {
if(events > 0) {
/* remove INPUT_RECORDs from the input buffer */
inputrecords = (INPUT_RECORD*)malloc(events *
sizeof(INPUT_RECORD));
if(inputrecords) {
if(!ReadConsoleInput(handle, inputrecords,
events, &inputs))
inputs = 0;
free(inputrecords);
}
/* check if we got all inputs, otherwise clear buffer */
if(events != inputs)
FlushConsoleInputBuffer(handle);
}
}
/* remove from descriptor set since there is no real data */
FD_CLR(sock, readfds);
}
}
/* stdin is never ready for write or exceptional */
FD_CLR(sock, writefds);
FD_CLR(sock, exceptfds);
}
else if(fds == fileno(stdout) || fds == fileno(stderr)) {
/* stdout and stderr are never ready for read or exceptional */
FD_CLR(sock, readfds);
FD_CLR(sock, exceptfds);
}
else {
/* try to handle the event with the WINSOCK2 functions */
error = WSAEnumNetworkEvents(fds, NULL, &wsanetevents);
if(error != SOCKET_ERROR) {
/* remove from descriptor set if not ready for read/accept/close */
if(!(wsanetevents.lNetworkEvents & (FD_READ|FD_ACCEPT|FD_CLOSE)))
FD_CLR(sock, readfds);
/* remove from descriptor set if not ready for write/connect */
if(!(wsanetevents.lNetworkEvents & (FD_WRITE|FD_CONNECT)))
FD_CLR(sock, writefds);
/* remove from descriptor set if not exceptional */
if(!(wsanetevents.lNetworkEvents & FD_OOB))
FD_CLR(sock, exceptfds);
}
}
/* check if the event has not been filtered using specific tests */
if(FD_ISSET(sock, readfds) || FD_ISSET(sock, writefds) ||
FD_ISSET(sock, exceptfds)) {
ret++;
}
}
else {
/* remove from all descriptor sets since this handle did not trigger */
FD_CLR(sock, readfds);
FD_CLR(sock, writefds);
FD_CLR(sock, exceptfds);
}
}
for(idx = 0; idx < wsa; idx++) {
WSAEventSelect(wsasocks[idx], NULL, 0);
WSACloseEvent(wsaevents[idx]);
}
free(wsaevents);
free(wsasocks);
free(handles);
free(fdarr);
return ret;
}
int main(void)
{
WORD wVersionRequested;
WSADATA wsaData;
SOCKET sock[4];
struct sockaddr_in sockaddr[4];
fd_set readfds;
fd_set writefds;
fd_set exceptfds;
SOCKET maxfd = 0;
int selfd = 0;
void *buffer = malloc(1024);
ssize_t nread;
setmode(fileno(stdin), O_BINARY);
wVersionRequested = MAKEWORD(2, 2);
WSAStartup(wVersionRequested, &wsaData);
sock[0] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
sockaddr[0].sin_family = AF_INET;
sockaddr[0].sin_addr.s_addr = inet_addr("74.125.134.26");
sockaddr[0].sin_port = htons(25);
connect(sock[0], (struct sockaddr *) &sockaddr[0], sizeof(sockaddr[0]));
sock[1] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
sockaddr[1].sin_family = AF_INET;
sockaddr[1].sin_addr.s_addr = inet_addr("74.125.134.27");
sockaddr[1].sin_port = htons(25);
connect(sock[1], (struct sockaddr *) &sockaddr[1], sizeof(sockaddr[1]));
sock[2] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
sockaddr[2].sin_family = AF_INET;
sockaddr[2].sin_addr.s_addr = inet_addr("127.0.0.1");
sockaddr[2].sin_port = htons(1337);
printf("bind = %d\n", bind(sock[2], (struct sockaddr *) &sockaddr[2], sizeof(sockaddr[2])));
printf("listen = %d\n", listen(sock[2], 5));
sock[3] = INVALID_SOCKET;
while(1) {
FD_ZERO(&readfds);
FD_ZERO(&writefds);
FD_ZERO(&exceptfds);
FD_SET(sock[0], &readfds);
FD_SET(sock[0], &exceptfds);
maxfd = maxfd > sock[0] ? maxfd : sock[0];
FD_SET(sock[1], &readfds);
FD_SET(sock[1], &exceptfds);
maxfd = maxfd > sock[1] ? maxfd : sock[1];
FD_SET(sock[2], &readfds);
FD_SET(sock[2], &exceptfds);
maxfd = maxfd > sock[2] ? maxfd : sock[2];
FD_SET((SOCKET)fileno(stdin), &readfds);
maxfd = maxfd > (SOCKET)fileno(stdin) ? maxfd : (SOCKET)fileno(stdin);
printf("maxfd = %d\n", maxfd);
selfd = select_ws(maxfd + 1, &readfds, &writefds, &exceptfds, NULL);
printf("selfd = %d\n", selfd);
if(FD_ISSET(sock[0], &readfds)) {
printf("read sock[0]\n");
nread = recv(sock[0], buffer, 1024, 0);
printf("read sock[0] = %d\n", nread);
}
if(FD_ISSET(sock[0], &exceptfds)) {
printf("exception sock[0]\n");
}
if(FD_ISSET(sock[1], &readfds)) {
printf("read sock[1]\n");
nread = recv(sock[1], buffer, 1024, 0);
printf("read sock[1] = %d\n", nread);
}
if(FD_ISSET(sock[1], &exceptfds)) {
printf("exception sock[1]\n");
}
if(FD_ISSET(sock[2], &readfds)) {
if(sock[3] != INVALID_SOCKET)
closesocket(sock[3]);
printf("accept sock[2] = %d\n", sock[2]);
nread = sizeof(sockaddr[3]);
printf("WSAGetLastError = %d\n", WSAGetLastError());
sock[3] = accept(sock[2], (struct sockaddr *) &sockaddr[3], &nread);
printf("WSAGetLastError = %d\n", WSAGetLastError());
printf("accept sock[2] = %d\n", sock[3]);
}
if(FD_ISSET(sock[2], &exceptfds)) {
printf("exception sock[2]\n");
}
if(FD_ISSET(fileno(stdin), &readfds)) {
printf("read fileno(stdin)\n");
nread = read(fileno(stdin), buffer, 1024);
printf("read fileno(stdin) = %d\n", nread);
}
}
WSACleanup();
free(buffer);
}
Compile using MinGW with the following command:
mingw32-gcc select_ws.c -Wl,-lws2_32 -g -o select_ws.exe
Running the program directly from the console using the following command works:
select_ws.exe
But doing the same with a pipe will constantly signal WaitForMultipleObjectsEx:
ping -t 8.8.8.8 | select_ws.exe
The pipe is ready to read until the parent process is finished, e.g.:
ping 8.8.8.8 | select_ws.exe
Is there a compatible way to simulate a blocking wait on the PIPE-based console input handle together with the other handles? The use of threads should be avoided.
You are welcome to contribute changes to the example program in this gist.
Thanks in advance!
I actually found a way to make it work using a separate waiting-thread. Please see the following commit in the curl repository on github.com.
Thanks for your comments!
Use GetStdHandle(STD_INPUT_HANDLE) to get the STDIN pipe handle, then use ReadFile/Ex() with an OVERLAPPED structure whose hEvent member is set to a manual-reset event from CreateEvent(). You can then use any of the WaitFor*() functions to wait on the event. If it times out, call CancelIo() to abort the read operation.

SIGPIPE With Running Program

I have two daemons, and A is speaking to B. B is listening on a port, and A opens a tcp connection to that port. A is able to open a socket to B, but when it attempts to actually write said socket, I get a SIGPIPE, so I'm trying to figure out where B could be closing the open socket.
However, if I attach to both daemons in gdb, the SIGPIPE happens before any of the code for handling data is called. This kind of makes sense, because the initial write is never successful, and the listeners are triggered from receiving data. My question is - what could cause daemon B to close the socket before any data is sent? The socket is closed less than a microsecond after opening it, so I'm thinking it can't be a timeout or anything of the sort. I would love a laundry list of possibilities to track down, as I've been chewing on this one for a few days and I'm pretty much out of ideas.
As requested, here is the code that accepts and handles communication:
{
extern char *PAddrToString(pbs_net_t *);
int i;
int n;
time_t now;
fd_set *SelectSet = NULL;
int SelectSetSize = 0;
int MaxNumDescriptors = 0;
char id[] = "wait_request";
char tmpLine[1024];
struct timeval timeout;
long OrigState = 0;
if (SState != NULL)
OrigState = *SState;
timeout.tv_usec = 0;
timeout.tv_sec = waittime;
SelectSetSize = sizeof(char) * get_fdset_size();
SelectSet = (fd_set *)calloc(1,SelectSetSize);
pthread_mutex_lock(global_sock_read_mutex);
memcpy(SelectSet,GlobalSocketReadSet,SelectSetSize);
/* selset = readset;*/ /* readset is global */
MaxNumDescriptors = get_max_num_descriptors();
pthread_mutex_unlock(global_sock_read_mutex);
n = select(MaxNumDescriptors, SelectSet, (fd_set *)0, (fd_set *)0, &timeout);
if (n == -1)
{
if (errno == EINTR)
{
n = 0; /* interrupted, cycle around */
}
else
{
int i;
struct stat fbuf;
/* check all file descriptors to verify they are valid */
/* NOTE: selset may be modified by failed select() */
for (i = 0; i < MaxNumDescriptors; i++)
{
if (FD_ISSET(i, GlobalSocketReadSet) == 0)
continue;
if (fstat(i, &fbuf) == 0)
continue;
/* clean up SdList and bad sd... */
pthread_mutex_lock(global_sock_read_mutex);
FD_CLR(i, GlobalSocketReadSet);
pthread_mutex_unlock(global_sock_read_mutex);
} /* END for each socket in global read set */
free(SelectSet);
log_err(errno, id, "Unable to select sockets to read requests");
return(-1);
} /* END else (errno == EINTR) */
} /* END if (n == -1) */
for (i = 0; (i < max_connection) && (n != 0); i++)
{
pthread_mutex_lock(svr_conn[i].cn_mutex);
if (FD_ISSET(i, SelectSet))
{
/* this socket has data */
n--;
svr_conn[i].cn_lasttime = time(NULL);
if (svr_conn[i].cn_active != Idle)
{
void *(*func)(void *) = svr_conn[i].cn_func;
netcounter_incr();
pthread_mutex_unlock(svr_conn[i].cn_mutex);
func((void *)&i);
/* NOTE: breakout if state changed (probably received shutdown request) */
if ((SState != NULL) &&
(OrigState != *SState))
break;
}
else
{
pthread_mutex_lock(global_sock_read_mutex);
FD_CLR(i, GlobalSocketReadSet);
pthread_mutex_unlock(global_sock_read_mutex);
close_conn(i, TRUE);
pthread_mutex_unlock(svr_conn[i].cn_mutex);
pthread_mutex_lock(num_connections_mutex);
sprintf(tmpLine, "closed connections to fd %d - num_connections=%d (select bad socket)",
i,
num_connections);
pthread_mutex_unlock(num_connections_mutex);
log_err(-1, id, tmpLine);
}
}
else
pthread_mutex_unlock(svr_conn[i].cn_mutex);
} /* END for i */
/* NOTE: break out if shutdown request received */
if ((SState != NULL) && (OrigState != *SState))
return(0);
/* have any connections timed out ?? */
now = time((time_t *)0);
for (i = 0;i < max_connection;i++)
{
struct connection *cp;
pthread_mutex_lock(svr_conn[i].cn_mutex);
cp = &svr_conn[i];
if (cp->cn_active != FromClientDIS)
{
pthread_mutex_unlock(svr_conn[i].cn_mutex);
continue;
}
if ((now - cp->cn_lasttime) <= PBS_NET_MAXCONNECTIDLE)
{
pthread_mutex_unlock(svr_conn[i].cn_mutex);
continue;
}
if (cp->cn_authen & PBS_NET_CONN_NOTIMEOUT)
{
pthread_mutex_unlock(svr_conn[i].cn_mutex);
continue; /* do not time-out this connection */
}
/* NOTE: add info about node associated with connection - NYI */
snprintf(tmpLine, sizeof(tmpLine), "connection %d to host %s has timed out after %d seconds - closing stale connection\n",
i,
PAddrToString(&cp->cn_addr),
PBS_NET_MAXCONNECTIDLE);
log_err(-1, "wait_request", tmpLine);
/* locate node associated with interface, mark node as down until node responds */
/* NYI */
close_conn(i, TRUE);
pthread_mutex_unlock(svr_conn[i].cn_mutex);
} /* END for (i) */
return(0);
}
NOTE: I didn't write this code.
Is it possible you messed up and somewhere else in the program you try to close the same handle twice?
That could do this to you very easily.
HINT: systrace can determine if this is happening.

Resources