I am doing some load testing on a service run with Apache2 and my load testing tool has a default timeout of 30 seconds. When I run the tool for a minute with 1 request per second load, it reports that 40 succeeded with 200 OK response and 20 requests were cancelled because client timeout exceeded while awaiting headers.
Now, I was trying to spot this on the server side. I can't see the timeouts logged either in apache access logs or gunicorn access logs. Note that I am interested in connections that weren't accepted as well as that are accepted and times out.
I have some experience working on similar services on Windows. The http.sys error logs would show connection dropped errors and we would know if our server was dropping connections.
When a client times out, all the server knows is that the client has aborted the connection. In mod_log's config, the %X format specifier is used to log the status of the client connection after the request has completed, which is exactly what you want to know in this case.
Configure your logs to use %X, and look for the X character in the log lines.
Bonus: I even found the discussion about this feature in apache's dev forum, from 20 years ago
Update:
Regarding refused connections, these cannot be logged by apache. Connection refusal is done by the kernel, in the tcp stack, and not by apache. The closest solution including only apache that I can think of is keeping track of the amount of open connections (using mod_status). If it reaches the maximum you know you might be refusing connections. Otherwise, you'd need to set up some monitoring solution to track tcp resets sent by the kernel.
Related
I have an application where a user can upload a PDF using angular-file-upload.js
This library does not support file chunking: https://github.com/nervgh/angular-file-upload/issues/41
My elastic load balancer is configured to have an idle timeout of 10 seconds and other parts of the application depend on keeping this parameter.
The issue is if the file upload takes longer than 10 seconds the user receives a 504 Gateway Timeout in the browser and an error message. However, the file still reaches the server after some time.
How can I ignore or not show the user this 504 Gateway Timeout that comes from the ELB? Is there another way around this issue?
The issue you have is that an ELB is always going to close the connection unless it gets some traffic back from your server. See below from AWS docs. It's the same behaviour for an ALB or a Classic load balancer.
By default, Elastic Load Balancing sets the idle timeout to 60 seconds
for both connections. Therefore, if the instance doesn't send some
data at least every 60 seconds while the request is in flight, the
load balancer can close the connection. To ensure that lengthy
operations such as file uploads have time to complete, send at least 1
byte of data before each idle timeout period elapses, and increase the
length of the idle timeout period as needed.
So to get around this, you have two options:
Change the server processing to start sending some data back as soon as the connection is established, on an interval of less than 10 seconds.
Use another library for doing your uploads, or use vanilla javascript. There are plenty of examples out there, e.g. this one.
Edit: Third option
Thanks to #colde for making the valid point that you can simply work around your load balancer altogether. This has the added benefit of freeing up your server resources which get tied up with lengthy uploads. In our implementation of this we used pre-signed urls to securely achieve this.
The OPTION/POST Request is failing inconsistently with a console Error as err_timed_out. We get the issue inconsistently, it's only observed sometimes. Otherwise the request gets proper response from the back end. When it's timing out, the request doesn't even reach the server.
I have done some research on the stuff and found that due to maximum 6 connections to a resource restrictions it may wait for getting a connection released. But, I don't see any other requests pending ,all the other requests were completed.
In the timeline I can always see that it stalled for 20.00 seconds. Most of the time the time is same. But, it only shows that its been stalled for some time nothing else in the timeline.
The status of the request shows failed ERR_Connection_Timed_Out. Please help.
The Network Timing
Console Error
I've seen this issue when I use an authenticated proxy server and usually a refresh of the page fixes it.
Are you using an authenticated proxy server where you are seeing this behavior? Have you tried on a pc with direct access (i.e. without proxy) to the Internet?
I've got the same problem when I choose another ISP. I thought I would have only to put my new ID and password, but it wasn't the case.
I have an ADSL modem with a dry loop.
All others services were fine (DNS resolution, IP telephony, FTP, etc).
I did a lot of tests (disable firewall, try some others navigator, try under Linux, modem default factory, etc) none of those tests were successful.
To resolve the problem ERR_TIMED_OUT, I had to adjust the MTU and MRU values. I put 1458 rather than 1492, which is the default value.
It works for me. Maybe some ISPs use different values. Good luck.
Our performance testing team is running test on our WPF-WCF-Sql Server application and they are facing connection timeout after the load goes above 75 users
Error -27796: Failed to connect to server "81.171.180.119:4567": [10060] Connection timed out
I would like to know what can be steps to look at bottlenecks which may be causing issues like maybe some setting in Load Runner or identify the code bottlenecks.
Thanks
Based on what you wrote my first guess would be a configuration on the server that only allows 75 active sessions to connect to it. I would start by looking into the server's configuration for concurrent threads/sessions/connections.
you need to change the runtime settings in load runner{press F4} , in miscellaneous >options and change connection timeout to around 240 seconds depending how your application is responding.
now loadrunner will wait for about 4 minutes until response is received.
Check similar settings for step download timeout in case of downloads are there in your application.
we wrote in C++ a screen sharing application based on sending screenshots.
It works by establishing a TCP connection btw the server and client, where the server forwards every new screenshot received for a user through the connection, and this is popped-up by the client.
Now, we are trying to host this on google app engine, and therefore need 'servlet'-ize and 'sandbox' the server code, so to implement this forwarding through HTTP requests.
I immagine the following:
1. Post request with the screenshot as multiple-data form (apache uploads ..).
But now the server needs to contact the specified client (who is logged in) to send it/forward the screenshot.
I'm not sure how to 'initiate' such connection from the servlet to the client. The client doesn't run any servlet environment (of course).
I know HTTP 1.1 mantains a TCP connection, but it seems gapps won't let me use it.
1 approaches that comes to mind is to send a CONTINUE 100 to every logged in user at login, and respond with the screenshot once it arrives. Upon receival the client makes another request, and so on.
an alternative (insipired from setting the refresh header for a browser) would be to have the app pool on a regular basis (every 5 secs).
You're not going to be able to do this effectively on GAE.
Problem 1: All output is buffered until your handler returns.
Problem 2: Quotas & Limits:
Some features impose limits unrelated
to quotas to protect the stability of
the system. For example, when an
application is called to serve a web
request, it must issue a response
within 30 seconds. If the application
takes too long, the process is
terminated and the server returns an
error code to the user. The request
timeout is dynamic, and may be
shortened if a request handler reaches
its timeout frequently to conserve
resources.
Comet support is on the product roadmap, but to me your app still seems like a poor fit for a GAE application.
Long Polling is the concept used for such asynchronous communications between server and client.
In Long Polling, servlet keeps a map of client and associated messages. Key of Map being client id and value being list of messages to be sent to the client. When a client opens a connection with server (sends request to a servlet), the servlet checks the Map if there are any messages to be sent to it. If found, it sends the messages to the client exits from the method. On receiving messages, the client opens a new connection to the server. If the servlet does not find any messages for given client, it waits till the Map gets updated with messages for given client.
This is a late reply, I'm aware, but I believe that Google have an answer for this requirement: the Channel API.
I've read that Silverlight 2.0 imposes by design an asynchronous model when communicating with the web server. I haven't had a chance to experiment with Silverlight, but I assume that it uses a thread-pool to manage threads like in the .NET Framework.
Now, since some browsers, most notably Internet Explorer, have an hard-coded limit of maximum two concurrent HTTP connections that can be made on the web server, what happens if I make a bunch of asynchronous requests from Silverlight?
Does Silverlight bypass this limitation in the web browser and open as many HTTP connections as there are threads available, or do the asynchronous requests queue up and wait for one of the two connections to become available?
In IE (haven't tested others) Silverlight is restricted to 2 connections at a time.
The behavior in Silverlight is to simply not make the request. So if you make 5 Async web service requests right in a row, the first 2 will happen, the other three won't. No exception is thrown that i've seen...
Fiddler is a big help here :)
Create a messaging manager interface for your client. Any outgoing request are posted to a queue that this manager processes against. It would serially process queued messages (i.e., when the call back of the last message sent to the server is invoked, can then safely proceed to process the next queued message).
You can consume the other connection resource by keeping a Comet connection open to the server. The server would push any return messages to the client via this Comet connection. You'll need to stamp out-going messages with a unique number that can be embedded as a property on in-coming messages - so that results can be correlated to request. The messaging manager would dispatch a result message to the appropriate handler for that result.
Essentially you end up using two connection resources to establish bi-directional messaging. But there is no artificial limit on the number of requesters on the client (though request will get serially transmitted to the server). The act of sending is always fast, though, because you don't wait for any result to be computed - you just need to deliver the message reliably to the server and return. Results come back asynchronously on the other Comet connection.
We do something along these lines with our Flex client apps in conjunction to Adobe BlazeDS running in our Tomcat web server:
A Flex-based asynchronous stack
Firefox is also limited to two connections, in addition to IE as stated already.
Note that the limit is per hostname.
If you add entries to your hosts file, or use dns aliases you can get more connections. For example in testing, add lines like '127.0.0.1 test1' to your hosts file, and then you can open two connections to http://localhost and two more to http://test1
I guess, being a .NET application Silverlight 2 has an independent from browser limit.
I would assume It is maxconnection attribute in Machine.config as mentioned in http://support.microsoft.com/kb/828219
Firstly the Machine.config file would not be used as the Silverlight control is sandboxed with its own version of the CoreCLR.
I believe that the Silverlight control actually makes use of the underlying browser to make the asynchronous HTTP requests. This is most likely the case considering how the Silverlight control can't gain access to SOAP fault information as the SOAP specification requires that the server returns an HTTP 500 response code and the Silverlight control doesn't get that from the browser hosting the control.
This post here serves to confirm this.
As to the limit of concurrent HTTP connections, I believe IE5 and later limit the number of connections to the same site based on HTTP protocol version - HTTP/1.0 it limits to 4 connections and HTTP/1.1 to 3 connections. Most of the time the web server will limit the number of connections to 2 per client, queueing or discarding the remainder.