While working with snowflake external functions we found a issue, and we believe Snowflake is not respecting Retry-After response header.
The external function invokes the AWS lambda function via AWS API gateway which in-turn invokes a 3rd party API (does some data enrichment), and return the response back to the snowflake client with status 200 (for happy case scenario)
For a batch request of 50, if we try to process a 1 million records, then after certain time, our 3rd party API starts sending HTTP 429 and we send the same to the snowflake client, along with Retry-After header in response of 10 seconds. (As the third party API tends to get overloaded with requests after a certain time). This is an indicator which tells Snowflake to scale back the rate at which it sends rows, and retries sending batches of rows that were not processed successfully.
To our surprise Snowflake is not actually respecting the parameter and retrying the request again within 1-3 seconds. Please find attached the cloud watch logs for the lambda function:
How can we slow down the request rate, so snowflake starts invoking after 10 seconds thereby giving some time for the 3rd party API to recover from 429?
Related
I have a very simple Azure Logic App that makes a REST call to an SAP web server and translates the response JSON before sending a response back to the caller of the Logic App. What is baffling me is that when the SAP call takes just over 1 minute, the Response action throws this error:
ActionResponseTimedOut. The execution of template action 'Response' is
failed: the client application timed out waiting for a response from
service. This means that workflow took longer to respond than the
alloted timeout value. The connection maintained between the client
application and service will be closed and client application will get
an HTTP status code 504 Gateway Timeout.
According to Microsoft documentation, the time-out HTTP calls is supposed to be 120 seconds (2 minutes). Unless the Logic App history display is completely wrong, the entire Logic App never takes any where near 120 seconds to complete, it keeps failing at just over 60 seconds.
The SAP GET CustomerCredit action shown in the sample below is a Logic Apps Customer Connector, not the built-in SAP action. The Logic App is the current production version, not a preview version.
Am I doing something wrong? I'd be fine if the Logic App actually timed-out after 2 minutes, but a 1 minute time-out is a bit extreme.
I don't know why your logic app shows ActionResponseTimedOut error even if it doesn't execute more than 120 seconds. I test it in my side and it works fine it the execution time less than 120 seconds. Here I can provide a workaround which may help with your problem.
1. Click "..." --> "Setting" of your "Response" action.
2. Enable "Asynchronous Response"
3. Then when you request the url of logic app, it will response with 202 accepted immediately. And in header of the response, we can find a "Location".
4. Request the url in "Location", it will response you with the result of the logic app(if the workflow is completed). If the workflow hasn't been completed, it will still response 202.
I have pretty simple LA that contains just 3 actions. It has HTTP trigger, then it gets some data from SQL server and returns http response with SQL data.
Sometimes, it takes 30-50 seconds to get data from SQL but Logic App in the meantime responses with Timeout error to caller.
The execution of template action 'Response_2' is failed: the client application timed out waiting for a response from service. This means that workflow took longer to respond than the alloted timeout value. The connection maintained between the client application and service will be closed and client application will get an HTTP status code 504 Gateway Timeout.
Any idea how to increase allowed time for response?
You can turn on the Asynchronous Response in the Settings of the Response action:
When you run your logic app longer than its time limit, you will accept 202 HTTP Code first:
It will return a response contains location header:
You can request the location URL, if the status of your logic app still is running, it will return 202.
If the status of your logic app is Succeeded, then it will return the results you want.
You can refer this official document.
We have a Azure Function that is called from the Azure Scheduler. It is a long running function. We are using the Durable Functions Framework. It is returning to the Azure Scheduler 202 Accepted, the Location for the callback to check status of the function. For some reason it seems the Azure Scheduler is not calling back. Based on the documentation of "Limits" you are to implement the "HTTP asynchronous protocols" if you have an action that takes longer than 1 minute. This is what we have done. We have verified the response headers have Location and Retry-After with values.
From Docs:
There’s a static (not configurable) request timeout of 60 seconds for HTTP actions. For longer running operations, follow HTTP asynchronous protocols; for example, return a 202 immediately but continue working in the background.
Any one else encounter this behavior?
We have a web application (UI wrapped in CEF + WPF) where we make a JavaScript POST request to the server. The POST request only has query string parameters and no request body. The server takes up to 5 minutes to process and send a small response back. Even when the server sends the response back (lets say 4 minutes), the request is still shown as pending in the Inspector (and success callback is not executed). Only after 8-10 minutes the response is shown as complete (and success callback is executed).
When we check the time distribution of the finished request call, it says that everything finished in 4 minutes which is very strange. This behavior is not happening in Google chrome browser. Any advise on what could be going wrong.
In my shopify store I have setup an order creation webhook. The webhook points to a Cakephp action URL which receives the data from webhook as following:-
$content = file_get_contents ( "php://input" );
After that it is saving this order data to the app database as:-
$orderData =array('order'=>$data['order_number'],'details'=>$content);
$orders = new Order ();
$orders->saveall($orderData);
Now the issue is that for each single order created the webhook is getting invoked multiple times. Although it performs the necessary action in the first attempt, yet Shopify is not able to identify the call success and is getting it invoked again and again until the limit reaches. After the limit is reached the webhook is getting deleted from the store.
My question is that do we need to send any type of status or response to the webhook call after it performs the necessary action. Because it is not very clear from shopify webhook documentation. They state that webhook call success is determined from HTTP status 200. How can I check what is the status returned by a webhook call? How can I make sure that Shopify is informed of webhook success through my app code and it does not invokes further calls to the webhook?
Yes, you need to send a 200 response to Shopify within a short time 5s. Otherwise, Shopify will send a request in a short time.
The official guide suggests that you store the webhook data and process it with a queue, thread, or whatever ways you preferred. After that, you return a 200 response to Shopify immediately.
IMO, if there are many webhook requests sending to you, it's better to separate the webhook receiver from your app server. You can do it with AWS Lambda or docker swarm so that the webhook requests won't break your app server.
Source:
Time limit: enter link description here
Webhooks with AWS Lambda: enter link description here
Just to clarify for others, you have to explicitly return a 2XX HTTP code or it'll retry 19 times over 48 hours, then delete your webhook if it exceeds that.