I configured a simple cron job available at secure(admin only) path /cron?method=sendMail to send an email once daily. The servlet at the endpoint of the url is a jsp file which has the code to send the email.
This is tested through the frontend with the full url and it works.
I do not have any backend servers configured. Only frontend instances with 1 resident and dynamics.
The issue is that cron triggers successfully but the request fails with the following message.
0.1.0.1 - - [17/Nov/2013:01:07:44 -0800] "GET /cron?method=sendMail HTTP/1.1" 503 364 - "AppEngine-Google; (+http://code.google.com/appengine)"
This request is getting retried continuously and keeps failing.
The complete url however works. What is wrong with the setup? what am I missing?
Also, how do I stop the retries?
The cron entry I use goes like
<cron>
<url>/cron?method=sendMail</url>
<description>Send mail</description>
<schedule>every mon,tue,wed,thu,fri,sat,sun 11:30</schedule>
<timezone>Asia/Calcutta</timezone>
</cron>
The issue was resolved using the following tip from the documentation to secure the url
Requests from the Cron Service will also contain a HTTP header: X-AppEngine-Cron: true
The X-AppEngine-Cron header is set internally by Google App Engine. If your request handler finds this header it can trust that the request is a cron request. If the header is present in an external user request to your app, it is stripped.
Related
I have 2 service. One is hosted in Google App Engine and one is hosted in Cloud Run.
I use urlfetch (Python 2) imported from google.appengine.api in GAE to call APIs provided by the Cloud Run.
Occasionally there are a few (like <10 per week) DeadlineExceededError shown up like this:
Deadline exceeded while waiting for HTTP response from URL
But these few days such error suddenly occurs frequently (like ~40 per day). Not sure if it is due to Christmas peak hour or what.
I've checked Load Balancer logs of Cloud Run and turned out the request has never reached the Load Balancer.
Has anyone encountered similar issue before? Is anything wrong with GAE urlfetch?
I found a conversion which is similar but the suggestion was to handle the error...
Wonder what can I do to mitigate the issue. Many thanks.
Update 1
Checked again, found some requests from App Engine did show up in Cloud Run Load Balancer logs but the time is weird:
e.g.
Logs from GAE project
10:36:24.706 send request
10:36:29.648 deadline exceeded
Logs from Cloud Run project
10:36:35.742 reached load balancer
10:36:49.289 finished processing
Not sure why it took so long for the request to reach the Load Balancer...
Update 2
I am using GAE Standard located in US with the following settings:
runtime: python27
api_version: 1
threadsafe: true
automatic_scaling:
max_pending_latency: 5s
inbound_services:
- warmup
- channel_presence
builtins:
- appstats: on
- remote_api: on
- deferred: on
...
The Cloud Run hosted API gateway I was trying to call is located in Asia. In front of it there is a Google Load Balancer whose type is HTTP(S) (classic).
Update 3
I wrote a simple script to directly call Cloud Run endpoint using axios (whose timeout is set to 5s) periodically. After a while some requests were timed out. I checked the logs in my Cloud Run project, 2 different phenomena were found:
For request A, pretty much like what I mentioned in Update 1, logs were found for both Load Balancer and Cloud Run revision.
Time of CR revision log - Time of LB log > 5s so I think this is an acceptable time out.
But for request B, no logs were found at all.
So I guess the problem is not about urlfetch nor GAE?
Deadline exceeded while waiting for HTTP response from URL is actually a DeadlineExceededError. The URL was not fetched because the deadline was exceeded. This can occur with either the client-supplied deadline (which you would need to change), or the system default if the client does not supply a deadline parameter.
When you are making a HTTP request, App Engine maps this request to URLFetch. URLFetch has its own deadline that is configurable. See the URLFetch documentation.
You can set a deadline for each URLFetch request. By default, the deadline for a fetch is 5 seconds. You can change this default by:
Including the following appengine.api.urlfetch.defaultDeadline setting in your appengine-web.xml configuration file. Specify the timeout in seconds:
<system-properties>:
<property name="appengine.api.urlfetch.defaultDeadline" value="10"/>
</system-properties>
You can also adjust the default deadline by using the urlfetch.set_default_fetch_deadline() function. This function stores the new default deadline on a thread-local variable, so it must be set for each request, for example, in a custom middleware.
from google.appengine.api import urlfetch
urlfetch.set_default_fetch_deadline(45)
If your Cloud Run service is processing long requests, you can increase the request timeout. If your service doesn't return a response within the time specified, the request ends and the service returns an HTTP 504 error.
Update the timeoutSeconds attribute in YAML file as :
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: SERVICE
spec:
template:
spec:
containers:
- image: IMAGE
timeoutSeconds: VALUE
OR
You can update the request timeout for a given revision at any time by using the following command:
gcloud run services update [SERVICE] --timeout=[TIMEOUT]
If requests are terminating earlier with error code 503, you might need to update the request timeout setting for your language framework:
Node.js developers might need to update the [server.timeout property via server.setTimeout][6] (use server.setTimeout(0) to achieve an unlimited timeout) depending on the version you are using.
Python developers need to update Gunicorn's default timeout.
We are using AngularJs(1.5.8) and Spring-boot in our application. we build these as a single war file and deployed in tomcat server.
we are consuming the spring-boot API from Angularjs through HTTP Post request. The issue is when the response takes more than 120 seconds or 200 seconds (this time is not consistent), automatically a new request is triggered from the browser which causes some duplicate issue in Server. The browser is considering only the latest request even if we got the response for the previous request.
I am trying to set up a CRON job in a Google Cloud Platform. The job is showing up in the GCP console, although it is failing. There are no logs that reveal why it is failing. The schedule seems to be working ok, and I am able to manually run the job, although it also fails when initiated manually.
If I go to http://...../api/status/addit in the url bar, the job runs as expected.
There is a link to "View Logs" in on the task queues page where it shows my CRON job, but when I go to those logs they are completely empty.
Looking at the nginx request logs does not show any requests made to that url (or any requests for that matter). If I go to the url for the job manually, I can see those requests show up in the logs and everything that is supposed to happen happens so I know that endpoint is good.
Google App Engine Flexible environment, Python 3
Flask API
What other info can I provide? There are so many moving parts that I don't want to flood the question with irrelevant info.
cron.yaml:
cron:
- description: 'test cron job'
url: /api/status/addit
schedule: every 1 minutes
endpoint:
< some Flask Blueprint stuff initiates the "status" blueprint so that this url will resolve to /api/status/addit >
...
#status.route('/addit')
def add_to_file():
print('made it into the request')
from flask import Response
res = Response("{'foo':'bar'}", status=202, mimetype='application/json')
return res
I experienced the same issue with the same kind of configuration (GAE flexible, Python 3):
Cron fails with no logging.
Turns out it is a firewall issue: my default action was set to DENY.
According to the docs:
Note: If you define a firewall in the flexible environment, you must set firewall rules for both the 10.0.0.1 and 0.1.0.1 IP addresses to allow your app to receive requests from the Cron service.
Whitelisting 10.0.0.1 and 0.1.0.1 solved my issue.
Your urls don't match. Try:
cron:
- description: 'test cron job'
url: /addit
schedule: every 1 minutes
I ran into a similar problem when I was using SSL and a script to redirect users from http to https URLs (in my case SSLify). Google app cron seems to use the http version (at least for my flex app), so when my app was called by cron, it returned a 302 redirect to the https version which was interpreted as an error.
"A cron job will invoke a URL, using an HTTP GET" https://cloud.google.com/appengine/docs/flexible/nodejs/scheduling-jobs-with-cron-yaml
Thanks to https://stackoverflow.com/a/53018498/4288232 and comments that lead me to the solution.
I'm starting to use cloud endpoints in my GAE project but have been running into issues with the api not updating on the server.
localhost:8888/_ah/api/explorer is ok.
But when I deploy, nothing changes.
myapp.appspot.com:8888/_ah/api/explorer is bad
Further investigation shows the url end points update
example: https://myapp.appspot.com/_ah/api/myapp/v1/foo/list
But the loaded client api is still incorrect.
example: gapi.client.load('myapp', 'v1', callback, url);
gapi.client.myapp.foo.list();
If I changed the call from foo/list to foo/list2, the rest url would update, the api package would not.
I'll try to cover the two cases people could run into:
Client Side:
The Google APIs Explorer web app aggressively caches, so you'll need to clear your cache or force a refresh when you update your API server side to see the changes in the client.
Server Side (In Deployed Production App Engine App):
If you're having deployment issues, there are two places to look when debugging:
Check your Admin Logs (https://appengine.google.com/adminlogs?&app_id=s~YOUR-APP-ID) after deployment. After a successful deployment of your application code, you should see the message:
Completed update of a new default version
and shortly after that you should see:
Successfully updated API configuration
If you this message indicates the API configuration update failed, you should deploy again. If said error is persistent, you should notify us of a bug. If you don't see any message about your API configuration, you should check that the path /_ah/spi/.* is explicitly named in your routing config (app.yaml for Python, web.xml for Java).
Check your Application Logs (https://appengine.google.com/logs?&app_id=s~YOUR-APP-ID) after deployment. After the deployment finishes, Google's API infrastructure makes a request to /_ah/spi/BackendService.getApiConfigs in your application so that your API configuration (as JSON) can be registered with Google's API infrastructure and all the discovery-related configs can be created. If this request does not complete with a 200, then your API changes will not show up since Google's API infrastructure will have nothing to register.
If you are consistently getting a 302 redirect for requests to /_ah/spi/BackendService.getApiConfigs, it is because you (or your generated API config) have specified a "bns adapter" that uses http: as the protocol in your API root, but your web.xml (Java) or app.yaml (Python) is required that paths through /_ah/spi are secure. This will make requests using http: as the protocol be redirected (using 302) to the same page with https: as the protocol. This was discussed on the Trusted Tester forum before going to Experimental.
This is what happened to me.
I tested my endpoint on localhost and it worked fine.
I deployed my endpoint on appspot and when I made requests to it I received in the browser the message 'Not found'.
So I looked in the logs and when I made requests to the endpoint I saw a 404 http error code on favicon file. And in effects I forgot to put that file in my deploy.
So I redeployed my war with the favicon file, the 404 http code disappeared and the endpoint worked fine on appspot too!
I realize that this may sound silly, but it is what I experienced. (I apologize for my poor english)
I noticed that if you upload your app for the first time without the following in your web.xml:
<security-constraint>
<web-resource-collection>
<url-pattern>/_ah/spi/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
Then your bns adapter will be set as http going forward. When I add the above afterwards, I get 302 http code on /_ah/spi/BackendService.getApiConfigs and the endpoints never update.
So now I have reverted to not use https on /_ah/spi and my endpoints are updating. I guess for those that see their endpoints not being updated revert back to the first configuration they had for ssl on /_ah/spi/.
Yaw.
I had the same error Not Found (the 404 error code) when I was calling my API using this URL
https: // MY_APP_ID.appspot.com / _ah / api / MY_SERVICE / v1 / user
I tried everything and finally fixed it by removing the discovery files from WEB-INF and kept only MY_SERVICE-v1.api and then redeployed the API. It works fine now.
I was also getting stale API discovery doc after deploying new version, it took a couple of minutes for GAE to start serving the new one to me.
I had the same problem, and I checked the admin logs, other logs etc... but still my API wasn't updating to the latest version.
So I decided to check in the API code for the last method I had written (I am writing in Java 7). And I found out that GAE doesn't like statements like:
if (!blocked){ .... }
I switched that to:
if (blocked == false) { ... }
And it worked like a charm. So by the looks of it, GAE scans the new API methods and doesn't accept some shortcuts.
Hoping to get some help or direction with the following issue I am facing.
I am getting a 405 (method not allowed) error when using GAE backends with a cron job on the live system. The cron job started on the backend I defined, but it is throwing the 405 error after delegating the call to the target Restlet. The url route is as follows as per the logs.
http://backendname.appid.appspot.com/cronURL
My current configuration is as follows:
GAE/J: 1.6.1
RESTLET: 2.1 RC5
I've have done the following:
Defined the attachment of the cron url route to my java class in my Restlet application
I have backends.xml defined
Set my backend as public to see if that resolves the issue, but it didn't
I have cron.xml defined
Set url for cron job
Set the target as my backend instance name
Please let me know if you would like more information.
Thanks!
Have you marked up your RESTlet with the #Get annotation?
A cron job will invoke the specified URL using an HTTP GET. Other verbs (e.g., PUT, POST, DELETE) are not supported by cron jobs.