How to avoid Undertow Connection RESET in apache benchmark test? - nio

Question

Using apache benchmarking 100K request 20K concurrent users:
$ ab -n 100000 -c 20000 http://localhost:8080/mrs/ping
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
apr_socket_recv: Connection reset by peer (104) <<< HOW to overcome??
Below is the Undertow (version 1.2.6 + xnio-api 3.3.1) PingServer:
public class UndertowPingServer {
private static Logger log = Logger.getLogger(UndertowPingServer.class);
public static void main(String[] args) throws ServletException {
PathHandler path = Handlers.path()
.addPrefixPath("/mrs/ping", new HttpHandler() {
#Override
public void handleRequest(HttpServerExchange exchange) throws Exception {
exchange.getResponseHeaders().put(
Headers.CONTENT_TYPE, "text/plain");
exchange.getResponseSender().send("Server Time:" + new Date().toString() + "\n\n");
}
});
Undertow.Builder builder = Undertow.builder()
.setHandler(path)
.addHttpListener(8080, "0.0.0.0")
.setBufferSize(1024 * 16)
//this seems slightly faster in some configurations
.setIoThreads(Runtime.getRuntime().availableProcessors() * 2)
.setSocketOption(Options.BACKLOG, 500000)
.setWorkerThreads(2000)
//don't send a keep-alive header for HTTP/1.1 requests, as it is not required
.setServerOption(UndertowOptions.ALWAYS_SET_KEEP_ALIVE, false);
Undertow server = builder.build();
server.start();
log.info("micro-service running!");
}
}
All the needed linux kernel sockets and thread settings via sysctl are already done. That is why it can do the first 90K request with 20k users without issue.

Answer 1

You can add -r parameter to ab to prevent it exitting.
from https://httpd.apache.org/docs/2.4/programs/ab.html
-r
Don't exit on socket receive errors.
Connections will start to timeout and can prolong testing time by a timeout period.

Answer 2

I just FOUND OUT similar issue from:
blog.scene.ro/posts/apache-benchmark-apr_socket_recv
sudo sysctl -w net.ipv4.tcp_syncookies=0
This did the job. NO MORE Connection reset by peer (104).
Guess this might not be a undertow or xnio-api issue.

Answer 3

Another observation:
Undertow.Builder builder = Undertow.builder()
.setSocketOption(Options.BACKLOG, 100000) // << Does impact the REST
and also MAY related to:
$ sysctl -a | grep -i sync
...
net.ipv4.tcp_max_syn_backlog = 100000