Why is there a long pause after ~16370 socket connections? - c

Question

I've been playing a bit with the sockets API to get a feel for how it works.
I wrote two little programs:
The server listens on 8080 for a stream connection. It sends anyone that connects to it a simple message.
The client connects to 127.0.0.1:8080, and dumps what it receives to stdout. It repeats this 20000 times in sequence.
The messages flow blazingly fast up to ~16370 times, then it pauses for tens of seconds before it starts moving really fast again to finish out the 20000 connections.
I've repeated this experiment a few times, and I've caught it at 16370, 16371 and 16372. It's surprisingly consistent on repeat experiments.
My question is: why does it need to pause after ~16370 iterations? What's the bottleneck here?
FWIW, I'm on macOS Sierra.
I run the server code like this:
clang -Wall -Werror -Wpedantic server.c -o server.out && ./server.out
and the client code like this:
clang -Wall -Werror -Wpedantic client.c -o client.out && time ./client.out
Here are the two programs:
server.c
#include <errno.h>
#include <netdb.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#define PORT 8080
#define MAXMSG 512
int make_socket(int port) {
int sock;
struct sockaddr_in name;
sock = socket(PF_INET, SOCK_STREAM, 0);
if (sock < 0) {
perror("socket");
exit(1);
}
name.sin_family = AF_INET;
name.sin_port = htons(port);
name.sin_addr.s_addr = htonl(INADDR_ANY);
if (bind(sock, (struct sockaddr*) &name, sizeof(name)) < 0) {
perror("bind");
exit(1);
}
return sock;
}
int main(int argc, char** argv) {
const char hello[] = "Hello visitor ";
char buffer[MAXMSG];
int sk;
unsigned long count = 0;
strcpy(buffer, hello);
sk = make_socket(PORT);
listen(sk, 10);
printf("ready\n");
for (;;) {
count++;
sprintf(buffer + strlen(hello), "%lu", count);
int s = accept(sk, NULL, NULL);
if (send(s, buffer, strlen(buffer) + 1, 0) < 0) {
perror("send");
exit(1);
}
close(s);
printf("data socket (%d) message sent (%s)\n", s, buffer);
}
}
client.c
#include <arpa/inet.h>
#include <errno.h>
#include <netdb.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#define PORT 8080
#define MAXMSG 512
int make_socket() {
int sock;
sock = socket(PF_INET, SOCK_STREAM, 0);
if (sock < 0) {
perror("socket");
exit(1);
}
return sock;
}
int main(int argc, char** argv) {
char buffer[MAXMSG];
int sk;
size_t i;
struct sockaddr_in addr;
strcpy(buffer, "Hello world!");
for (i = 0; i < 20000; i++) {
sk = make_socket();
addr.sin_family = AF_INET;
addr.sin_port = htons(PORT);
addr.sin_addr.s_addr = inet_addr("127.0.0.1");
connect(sk, (struct sockaddr*) &addr, sizeof(addr));
recv(sk, buffer, strlen(buffer) + 1, 0);
close(sk);
printf("socket (%d) message = %s\n", sk, buffer);
}
}
This is some of the last stdout I get client side:
socket (3) message = Hello visitor 16369
socket (3) message = Hello visitor 16370
socket (3) message = Hello visitor 16371
socket (3) message = Hello visitor 16372

Answer 1

Most likely you are hitting a limit called the Ephemeral Port Range on your operating system. The same principle applies on all IP based operating systems.
Whenever a socket connection is made a port is allocated for the request and associated with the interface that the connection is made on. Once this socket is closed the port is put into a state called TIME_WAIT. Effectively putting the port on the bench for a period of time to ensure that is it not reused too soon. This is to prevent latent packets in the internet from coming in late and causing issues.
The ephermal port range is specified on Linux as /proc/sys/net/ipv4/ip_local_port_range.
You can display these on MacOS using:
sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535
Which is 16,383 available ports in the ephemeral range.
To see all network parameters you can execute:
sysctl net.inet.tcp
You can change the TIME_WAIT values but for your high stress application it would simply lower the threshold at which your slowdown waits.
You can see the number of open connections using netstat -an. Sockets may be stuck in TIME_WAIT state if you are opening and closing a lot of connections. In some places this is unavoidable, but you may need to consider if you need a pool of connection if this is the case.
If TIME_WAIT is the issue there are system settings you can adjust. you can set net.ipv4.tcp_tw_reuse / net.ipv4.tcp_tw_recycle to speed up connection turnover.
One quick test would be to switch to another interface and try again. If you were using localhost and then experienced the slowdown you could then switch to your external IP on another interface and you should run like gangbusters until you hit the limit again.
This is not an issue with the language your using but an issue in socket based network usage.

Answer 2

You are probably dealing with the TIME_WAIT state of the sockets. After actively closing each connection socket in your server, the socket remains in that state for a long (tens of seconds) period. So, as #Blaze suggests, your program is reaching a resource limit and has to wait for this sockets to be definitively closed.
This feature is there to prevent two cases, the first is the chance that a delayed packet sent in one connection to be interpreted as part of a later connection. The second is the chance that the last ACK of the connection close is not received by the passive closing side of the connection, causing it to retransmit the FIN/ACK. If this happens and the active closing side has already closed the socket it will answer with a RST, causing the other side to receive an error in spite that all the information was correctly sent.
Take a look at this: http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html
If you really want to completely close the socket (having the risk of the previous events to happen) you have to tune the SO_LINGER option before trying to close it.
struct linger immediate_linger;
immediate_linger.l_onoff = 1; /* Do linger on closing */
immediate_linger.l_linger = 0; /* Wait 0 seconds to linger after closing */
if (setsockopt(comm_socket, SOL_SOCKET, SO_LINGER, &immediate_linger, sizeof (immediate_linger))) {
err = errno;
printf("Error setting immediate linger for socket %d: %s.", comm_socket, strerror(err));
}
else {
printf("Set immediate linger after close for the socket %d.", comm_socket);
}