How can one handle RequestTimeoutException in Gatling so a scenario doesn't get marked as failed?
I have looked into https://github.com/gatling/gatling/blob/master/gatling-core/src/main/resources/gatling-defaults.conf but couldn't find a corresponding parameter.
You can change timeout in gatling.conf of your project, but I don't think you can make Gatling completely ignore it. I wonder what is the reason behind this goal, cause catching failures and put it in report is smth you are looking for?
If you have chained requests and with timeout the following requests are useless, you can use exitBlockOnFail to stop on failed (time outed)request.
http {
#fetchedCssCacheMaxCapacity = 200 # Cache size for CSS parsed content, set to 0 to disable
#fetchedHtmlCacheMaxCapacity = 200 # Cache size for HTML parsed content, set to 0 to disable
#perUserCacheMaxCapacity = 200 # Per virtual user cache size, set to 0 to disable
#warmUpUrl = "https://gatling.io" # The URL to use to warm-up the HTTP stack (blank means disabled)
#enableGA = true # Very light Google Analytics (Gatling and Java version), please support
#pooledConnectionIdleTimeout = 60000 # Timeout in millis for a connection to stay idle in the pool
requestTimeout = 120000 # Timeout in millis for performing an HTTP request
#enableHostnameVerification = false # When set to true, enable hostname verification: SSLEngine.setHttpsEndpointIdentificationAlgorithm("HTTPS")
dns {
#queryTimeout = 5000 # Timeout in millis of each DNS query in millis
#maxQueriesPerResolve = 6 # Maximum allowed number of DNS queries for a given name resolution
}
This is it: https://i.stack.imgur.com/SaZIq.png
You can tweak it as much as you want.
Related
I am trying to get scrape information.
I goto the introduction page to determine the number of search results. Often, the results occur over >1 page, and as such, I need to refresh and run another requests in a loop. On a few occasions an error occurs in the extra requests, or it hangs.
I am curious if there is a way to check a request, if it fails, then try again, and if it still fails, log it and go to the next one
Here is a sample script:
t=.3
urls=['https://stackoverflow.com/','https://www.google.com/'] #list of upto 200 urls
for url in urls:
print(url)
response = requests.get(url, timeout=t)
t=t+10
i=i-1
running this returns a timeout occasionally, and the processing stops. My workaround is to print the url that failed, and then rerun, updating the list manually.
I would like to find a way that if a request error occurs, the response retries 5x, and if it fails, logs and stores the failed url, then goes onto the next one, so that I can try the failed urls at a later stage
Any suggestions?
I haven't used Python in a while, but I'm pretty sure that this will work.
t=.3
urls=['https://stackoverflow.com/','https://www.google.com/'] #list of upto 200 urls
for url in urls:
print(url)
try:
response = requests.get(url, timeout=t)
except:
# if the request fails: try again
try:
response = requests.get(url, timeout=t)
except:
# if the request fails again: do nothing (continue to the next url)
pass
t=t+10
i=i-1
`Akka http 10.0.6;
max-content-length is set to 6000m`
I am using Akka file streaming to upload huge files( sent as octet-stream to my service ) to accept incoming bytes and write it to a file sink. This is what I observe from my experimentations.. My limited understand from reading the documents is the client should be able keep sending the data unless we tell explicitly from akka http for back pressure mechanisms.. Been searching online to understand this behavior, not able to get an understanding yet on this to explain the following behavior.. Is there something I am missing in the code? How can I debug more on this? Also, via scalatest, able to do it. If someone can throw more might on what’s the difference in behavior w.r.t scalatest and via curl/http clients.. Thanks
Through "curl", I can stream max of 1KB sized file. Anything more than this, it will just hang and this message is given after timeout no matter how long I wait(20 seconds, 5 mins, 10 mins, etc) "Sending an 2xx 'early' response before end of request was received... Note that the connection will be closed after this response. Also, many clients will not read early responses! Consider only issuing this response after the request data has been completely read!"
Through "Apache http client", max of 128KB sized file streaming happens. Anything more than this, it will hang and same message as in #1 from akka http service
Through "python client", max of ~26KB sized file streaming happens. Anything more than this, it will hang and same message as in #1 from akka http service
Through scalatest inside the service, was able to upload files like 200MB, 400MB and more also
Here’s the code..
`put {
withoutSizeLimit {
extractDataBytes { bytes =>
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// Tried system dispatcher also
implicit val executionContext = system.dispatchers.lookup("dispatcher")
val sink = FileIO.toPath(Paths.get("/file.out"))
val action = bytes.runWith(sink).map {
case ior if ior.wasSuccessful => {
complete(StatusCodes.OK, s"${ior.count} bytes written")
}
case ior => complete(StatusCodes.EnhanceYourCalm, ior.getError.toString)
}
Await.result(action, 300.seconds)
}
}
}`
I'm running GmailApp.search to find un-labeled email, classify it, and then label it if it matches various rules. The script triggers every 10 minutes, but is getting 'user-rate limit exceeded' warnings. Below is the GmailApp search I'm running. Typically I only have less than 100 unlabeled emails in my inbox, so I wouldn't expect this would take a lot of resources if the search is in any way efficient.
function RunRules()
{
var threads = GmailApp.search("label:inbox has:nouserlabels");
if (threads.length > 0)
{
for (var idxThread=threads.length-1; idxThread>=0; idxThread--)
{
var messages = threads[idxThread].getMessages();
if (messages)
{
for (var idxMsg=messages.length-1; idxMsg>=0; idxMsg--)
{
if (messages[idxMsg].isInInbox())
{
RunRulesOnMessage(messages[idxMsg]);
}
}
}
}
}
}
Any suggestions how to avoid the user-rate limit?
Thanks,
Dave
Based from Usage Limits, Gmail API have a per user rate limit of 250 quota per second.
Reading further, please note that there's also a corresponding quota units for every method that you can use and as stated, the number of quota units consumed by a request varies depending on the method called.
So, it is possible that you will be exceeding your per user rate limit if, for example, a client is requesting to get 75 unlabeled messages using the messages.get which is equivalent to 5 quota units per request.
For this, I suggest that you please try the following:
implement retries with exponential backoff in case of temporary errors such as, HTTP 429 errors, HTTP 403 quota errors, or HTTP 5xx errors.
batching requests which allows your client to put several API calls into a single HTTP request is encouraged. However, larger batch sizes are likely to trigger rate limiting and sending batches larger than 50 requests is not recommended.
Lastly, you may want to also check Gmail Per-User Limits which cannot be increased for any reason.
Hope above info helps!
I have a backends that is normally invoked by a cron to run a few times every day. Yesterday, I noticed it was restarting without stopping. I dont see a place in my code where that invocation is happening. Rather, the task queue seems to indicate it is running due to re-tries due to errors. One error is that status is saved to bigQuery and that is failing because a quoto is exceeded. But this seems to generate an infinite loop. Is this a bug in app engine or I am doing something wrong? Is there a way to indicate to not restart a task if it fails? My other app engine tasks that terminate without 200 status dont do that...
Here is a trace of the queue from which the restarts keep happening:
Here is the logging showing continous running
And here is the http header inside the logging
UPDATE1
Here is the cron:
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
<cron>
<url>/uploadToBigQueryStatus</url>
<description>Check fileNameSaved Status</description>
<schedule>every 15 minutes from 02:30 to 03:30</schedule>
<timezone>US/Pacific</timezone>
<target>checkuploadstatus-backend</target>
</cron>
</cronentries>
UPDATE 2
As for the comment about catching the error: The error I believe is that the biqQuery job fails because a quota has been hit. Strange thing is that it happened yesterday, and the quota should have been reset, so the error should have good away for at least a while. I dont understand why the task retries, I never selected that option that I am aware of.
I killed the servlet and emptied the task queue so at least it is stopped. But I dont know the root cause. IF BQ table quota was the reason, that shouldnt cause an infinite retry!
UPDATE 3
I have not trapped the servlet call that produced the error that led to the infinite retry. But I checked this cron activated servlet today and found I had another non-200 result. The return value this time was 500 and it is caused by a DataStore time-out exception.
Here is the screen shot of the return that show 500 return code.
Here is the exception info page 1
And the following data
The offending code line is the for loop iterating on the data store query
if (keys[0] != null) {
/* Define the query */
q = new Query(bucket).setAncestor(keys[0]);
pq = datastore.prepare(q);
gotResult = false;
// First system time stamp
Date date= new Timestamp(new Date().getTime());
Timestamp timeStampNow = new Timestamp(date.getTime());
for (Entity result : pq.asIterable()) {
I will add a try-catch on this for loop as it is crashing in this iteration.
if (keys[0] != null) {
/* Define the query */
q = new Query(bucket).setAncestor(keys[0]);
pq = datastore.prepare(q);
gotResult = false;
// First system time stamp
Date date= new Timestamp(new Date().getTime());
Timestamp timeStampNow = new Timestamp(date.getTime());
try {
for (Entity result : pq.asIterable()) {
Hopefully, the data store read will not crash the servlet but it will render a failure. At leas the cron will run again and pickup other non-handled results.
By the way, is this a java error or app engine? I see a lot of these data store time outs and I will add a try-catch around all the result loops. Still, it should not cause the infinite retry that I experienced. I will see if I can find the actual crash..problem is that it overloaded my logging...More later.
UPDATE 4
I went back to the logs to see when the inifinite loop began. In the logs below, I opened the run that is at the head of the continuous running. YOu can see that it fails with 500 every 5th time. It is not the cron that invoked it, it was me calling the servlet to check biq query upload status (I write to the data store the job info, then read it back in servlet and write to bigQuery the job status and if done, erase the data store entry.) I cannot explain the steady 500 errors every 5th call, but it is always the Data Store Timeout exception.
UPDATE 5
Can the infinite retries be happening because of the queue configuration?
CheckUploadStatus
20/s
10
100
10
200
2
I just noticed another task queue had a 500 return code and it was continuously retrying. I did some search and found some people have tried to configure
the queues for no retry. They said that didnt work.
See this link:
Google App Engine: task_retry_limit doesn't work?
But one re-try is possible? That is far better than infinite.
It is contradictory that Google enforces quotas but seems to prefer infinite retries. I would much prefer block the retries by default on non-200 return code and then have NO QUOTAS!!!
According to Retrying cron jobs that fail:
If a cron job's request handler returns a status code that is not in
the range 200–299 (inclusive) App Engine considers the job to have
failed. By default, failed jobs are not retried.
To set failed jobs to be retried:
Include a retry-parameters block in your cron.xml file.
Choose and set the retry parameters in the retry-parameters block.
Your cron config doesn't specify the necessary retry parameters, so the jobs returning the 500 code should, indeed, not be retried, as you expect.
So this looks like a bug. Possibly a variant of the (older) known issue 10075 - the 503 code mentioned there might have changed in the mean time - but it is also a quota-related failure.
The suggestion from GAEfan's comment is likely a good workaround:
You will need to catch the error, and send a 200 response to stop the
task queue from retrying. – GAEfan 1 hour ago
I perform a cron job where I need to update my search indices. As part of updating, I delete old documents with this code:
while True:
results = index.search(search.Query(
query_string="locationID="+location_id,
options=search.QueryOptions(
limit=100,
cursor=cursor,
ids_only=True)))
cursor = results.cursor
doc_ids = [tmp_result.doc_id for tmp_result in results]
index.delete(doc_ids)
if not cursor: # if cursor is None, meaning no more results
break
I am relatively routinely seeing this error in my logs:
DeadlineExceededError: The API call search.DeleteDocument() took too long to respond
and was cancelled.
Is there something I'm doing wrong with my deletion code that I'm seeing this error pop up?
Edit:
Is this just a random error that will show up from time to time? If so, should I just implement a redo with an exponential backoff like so:
def delete_doc_ids(doc_ids, retries):
success = False
time_to_sleep = 2**retries*0.1 #100 ms
time.sleep(time_to_sleep)
retries+=1
try:
index.delete(doc_ids)
success = True
return success, retries
except:
logging.info("Failure to delete documents. Retrying in %s seconds"%time_to_sleep)
return success, retries
# because this step fails a lot, keep running in a while loop until it works with exponential backoff
deletion_finished = False
retries = 0
#keep trying until deletion_finished returns true on an expontential backoff
while not deletion_finished:
deletion_finished, retries = delete_doc_ids(doc_ids,retries)
Edit 2:
What is the default deadline alluded to here? I dug through the RPC source files and can't find it.
Try sending the job to the taskqueue, which has a much longer deadline (10 minutes), or to a custom module, which can run indefinitely.