We have a Azure Function that is called from the Azure Scheduler. It is a long running function. We are using the Durable Functions Framework. It is returning to the Azure Scheduler 202 Accepted, the Location for the callback to check status of the function. For some reason it seems the Azure Scheduler is not calling back. Based on the documentation of "Limits" you are to implement the "HTTP asynchronous protocols" if you have an action that takes longer than 1 minute. This is what we have done. We have verified the response headers have Location and Retry-After with values.
From Docs:
There’s a static (not configurable) request timeout of 60 seconds for HTTP actions. For longer running operations, follow HTTP asynchronous protocols; for example, return a 202 immediately but continue working in the background.
Any one else encounter this behavior?
Related
While working with snowflake external functions we found a issue, and we believe Snowflake is not respecting Retry-After response header.
The external function invokes the AWS lambda function via AWS API gateway which in-turn invokes a 3rd party API (does some data enrichment), and return the response back to the snowflake client with status 200 (for happy case scenario)
For a batch request of 50, if we try to process a 1 million records, then after certain time, our 3rd party API starts sending HTTP 429 and we send the same to the snowflake client, along with Retry-After header in response of 10 seconds. (As the third party API tends to get overloaded with requests after a certain time). This is an indicator which tells Snowflake to scale back the rate at which it sends rows, and retries sending batches of rows that were not processed successfully.
To our surprise Snowflake is not actually respecting the parameter and retrying the request again within 1-3 seconds. Please find attached the cloud watch logs for the lambda function:
How can we slow down the request rate, so snowflake starts invoking after 10 seconds thereby giving some time for the 3rd party API to recover from 429?
I have a very simple Azure Logic App that makes a REST call to an SAP web server and translates the response JSON before sending a response back to the caller of the Logic App. What is baffling me is that when the SAP call takes just over 1 minute, the Response action throws this error:
ActionResponseTimedOut. The execution of template action 'Response' is
failed: the client application timed out waiting for a response from
service. This means that workflow took longer to respond than the
alloted timeout value. The connection maintained between the client
application and service will be closed and client application will get
an HTTP status code 504 Gateway Timeout.
According to Microsoft documentation, the time-out HTTP calls is supposed to be 120 seconds (2 minutes). Unless the Logic App history display is completely wrong, the entire Logic App never takes any where near 120 seconds to complete, it keeps failing at just over 60 seconds.
The SAP GET CustomerCredit action shown in the sample below is a Logic Apps Customer Connector, not the built-in SAP action. The Logic App is the current production version, not a preview version.
Am I doing something wrong? I'd be fine if the Logic App actually timed-out after 2 minutes, but a 1 minute time-out is a bit extreme.
I don't know why your logic app shows ActionResponseTimedOut error even if it doesn't execute more than 120 seconds. I test it in my side and it works fine it the execution time less than 120 seconds. Here I can provide a workaround which may help with your problem.
1. Click "..." --> "Setting" of your "Response" action.
2. Enable "Asynchronous Response"
3. Then when you request the url of logic app, it will response with 202 accepted immediately. And in header of the response, we can find a "Location".
4. Request the url in "Location", it will response you with the result of the logic app(if the workflow is completed). If the workflow hasn't been completed, it will still response 202.
I have pretty simple LA that contains just 3 actions. It has HTTP trigger, then it gets some data from SQL server and returns http response with SQL data.
Sometimes, it takes 30-50 seconds to get data from SQL but Logic App in the meantime responses with Timeout error to caller.
The execution of template action 'Response_2' is failed: the client application timed out waiting for a response from service. This means that workflow took longer to respond than the alloted timeout value. The connection maintained between the client application and service will be closed and client application will get an HTTP status code 504 Gateway Timeout.
Any idea how to increase allowed time for response?
You can turn on the Asynchronous Response in the Settings of the Response action:
When you run your logic app longer than its time limit, you will accept 202 HTTP Code first:
It will return a response contains location header:
You can request the location URL, if the status of your logic app still is running, it will return 202.
If the status of your logic app is Succeeded, then it will return the results you want.
You can refer this official document.
I am currently working on a web application that uses a JWT for authentication on all REST calls. The problem we are having is when a customer is performing an action needing a REST call and the JWT is expired, for whatever reason. At that point it kicks back a 401 response, as it should.
The functionality I am looking for is some way to intercept the 401 error, refresh the JWT, and retry the request without sending errors to the user.
The application runs on AngularJS and uses Restangular to handle all of the rest calls. So far I have been looking closely at the setErrorInterceptor as outlined here and at the Restangular documentation. Using a version of that posted code, I could successfully resend the request and in the .then() portion I had a successful response. However, it seems from the documentation that this method can only return false or true, so I couldn't get at it.
At the moment we have a timer that basically compares the jwt expiration time to the current system time and refreshes the jwt when there are 60 seconds left before expiration. This caused issues when the user's system clock was off compared to the server clock, resulting in a token not being refreshed because the application thought there was more time.
A solution to that specifically would be something like getting the system time, comparing to server time and creating an offset variable, but that doesn't cover all the bases for things that could go wrong. On another topic, checking the JWT before every request is not feasible.
Is there a way within Restangular's interceptors to accomplish the error interception and resending? If not, are there ways outside of it?
I created an App Engine backend to serve http requests for a long running process. The backend process works as expected when the query references an input of small size, but times out when the input size is large. The query parameter is the url of an App Engine BlobStore blob, which is the input data for the backend process. I thought the whole point of using App Engine backends was to avoid the timeout restricts that App Engine frontends possess. How can I avoid getting a timeout?
I call the backend like this, setting the connection timeout length to infinite:
HttpURLConnection connection = (HttpURLConnection)(new URL(url + "?" + query).openConnection());
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestMethod("GET");
connection.setConnectTimeout(0);
connection.connect();
InputStream in = connection.getInputStream();
int ch;
while ((ch = in.read()) != -1)
json = json + String.valueOf((char) ch);
System.out.println("Response Message is: " + json);
connection.disconnect();
The traceback (edited for anonymity) is:
Uncaught exception from servlet
java.net.SocketTimeoutException: Timeout while fetching URL: http://my-backend.myapp.appspot.com/somemethod?someparameter=AMIfv97IBE43y1pFaLNSKO1hAH1U4cpB45dc756FzVAyifPner8_TCJbg1pPMwMulsGnObJTgiC2I6G6CdWpSrH8TrRBO9x8BG_No26AM9LmGSkcbQZiilhC_-KGLx17mrS6QOLsUm3JFY88h8TnFNer5N6-cl0iKA
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:142)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296)
at org.someorg.server.HUDXML3UploadService.doPost(SomeService.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
As you can see, I'm not getting the DeadlineExceededException, so I think something other than Google's limits is causing the timeout, and also making this a different issue from similar stackoverflow posts on the topic.
I humbly thank you for any insights.
Update 2/19/2012: I see what's going on, I think. I should be able to have the client wait indefinitely using GWT [or any other type of client-side async framework] async handler for an any client request to complete, so I don't think that is the problem. The problem is that the file upload is calling the _ah/upload App Engine system endpoint which then, once the blob is stored in the Blobstore) calls the upload service's doPost backend to process the blob. The client request to _ah/upload is what is timing out, because the backend doesn't return in a timely fashion. To make this timeout problem go away, I attempted to make the _ah_upload service itself a public backend accessible via http://backend_name.project_name.appspot.com/_ah/upload, but I don't think that google allows a system service (like _ah/upload) to be run as a backend. Now my next approach is to just have ah_upload immediately return after triggering the backend processing, and then call another service to get the original response I wanted, after processing is finished.
The solution was to start a backend process as a tasks and add that to the task queue, then returning a response to client before it waits to process the backend task (which can take a long time). If I could have assigned ah_upload to a backend, this would have also solved the problem, since the clien't async handler could wait forever for the backend to finish, but I do not think Google permits assigning System Servlets to backends. The client will now have to poll persisted backend process response data, as Paul C mentioned, since tasks can not respond like a normal servlet.