We are experimenting with Flexible Environment and we want to migrate an existing application. Everything seems to go well except from cron entries.
We have a bunch of cron and every url is secured as admin" (as stated in the documentation), but every time the url is hit by the cron service we get a 403 status, if the url is hit by the user via browser everything works well.
On standard environment everything is good.
Is there anything we can check about?
I assume you add login: admin in app.yaml handler section to secure cron service on Google App Engine standard environment.
But on App Engine flex, it changed how to secure your cron handlers like this (PHP example):
Check $_SERVER['HTTP_X_APPENGINE_CRON'] and if it's true, the requests are coming from App Engine cron service.
Related
I have a Python app on Google App Engine Standard which is secured using Google Cloud Identity Aware Proxy (IAP).
I would like to trigger a part of my app every day by using Cloud Scheduler. (It is calling an API, doing calculations, and stores results in Google Cloud Storage.)
To do so, I tried triggering a POST request on an "App Engine HTTP". example URL: "/refresh_data"
When running the job, I get the following error:
jsonPayload: {
status: "FAILED_PRECONDITION"
relativeUrl: "/refresh_data"
jobName: "..."
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
targetType: "APP_ENGINE_HTTP"
}
I can't find any documentation relative to the "FAILED_PRECONDITION" error so I'm feeling kind of stuck here.
As an alternative, I tried to trigger a POST request to a simple "HTTP" and by granting the Owner access level in IAP to the service account I am using for Cloud Scheduler. This time, the error message I get is the following :
jsonPayload: {
status: "UNAUTHENTICATED"
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
jobName: "..."
targetType: "HTTP"
url: "https:.../refresh_data"
}
I really don't understand how to make this work... The Cloud Scheduler documentation barely documents the "App Engine HTTP" option, and nothing is documented relative to te use of IAP...
Any help would be much appreciated :)
It's the dark side of IAP. I sent this feedback to Google months ago. Same thing with Pubsub, you can't trigger and App Engine IAP protected from these serverless product, even if you use a service account with the correct authorization.
You can check this relevant documentation from IAP on how to authenticate with a Service Account.
Whenever you are using Cloud Scheduler, the requests will be done with its Service Account, therefore the guide to follow should be the one linked above as you are basically trying to authenticate programmatically rather than with the Google sign-in.
That being said, you would need to generate the OIDC token prior to making a request to an IAP-secured endpoint. Once you have the OIDC token, it needs to be included in the Authorization: Bearer header.
I had a similar problem when upgrading my GAE app from python 2.7 to python3 (standard).
I got the same error message as you (status: "FAILED_PRECONDITION") when running my previous cron.yaml jobs set up as cloud scheduler jobs.
And the upload of previous cron.yaml files did not work to run either.
Then I found out that just adding an ending '/' on the url fixed it.
So a cron like:
cron:
- description: competition participants pilot list update
url: /daily1/
schedule: every 8 hours from 05:00 to 21:00
worked after being uploaded with cloud SDK:
gcloud app deploy cron.yaml --project my-gae-project
(and I forgot that) I also had to:
gcloud services enable cloudscheduler.googleapis.com --project my-gae-project
I got this working recently for HTTP endpoints behind an IAP (we use cloud run, but it should be similar for any IAP protected HTTP endpoint). This is the Cloud Scheduler configuration that worked for me:
Auth Header: "Add OIDC Token"
Service account: Select a service account with the role "IAP-secured Web App User" (if you recently added this permission, it can take a few minutes to propagate)
Audience: Your IAP's Client ID (from here: https://console.cloud.google.com/apis/credentials)
I have been using the Google App Engine Launcher's "Deploy" button. I realize that the Launcher is going away at the end of July 2020, so I'm trying to learn to use gcloud. My web site has a custom url, www.xxx.org, but "gcloud app deploy" uploads to xxx.appspot.com instead of to www.xxx.org, presumably because "target url" is set to xxx.appspot.com. Can someone please tell me what gcloud command I need to use to change "target url" to www.xxx.org, or some other scheme that would work? I've wandered through lots of gcloud and GAE documentation and web searches without being able to figure out what I need to do.
There is a very good and complete guide about mapping a custom domain to your application in the official documentation. As #DanCornilescu said,it needs to be configured, but it is not necessary to be done before deployment.
Basically, you need to follow the steps from the guide:
1) Verify that you are the owner of your domain through Webmaster
Central
2)Ensure that your domain has been verified, otherwise you will not be able to proceed with the following steps. Note that only verified domains will be displayed.
3)If you need to delegate the ownership of your domain to other users or service accounts, you can add permission through the Webmaster Central page
4)After you verify ownership of your domain, you can map that domain to your App Engine app
5)Add the resource records that you receive to the DNS configuration of your domain registrar
6)Test for success by browsing to your app via its new domain URL, for example www.example.com
In case you need to do more complex mappings for different urls and subdomains you can always write down the routing rules in the dispatch.yaml file and they will be mapped successfully.
Also, to add a bit more of information about the dipatch.yaml file, it will handle the routing in the App as opposed with the app.yaml which will take care of the App's settings. Both .yamls can exist in the same application and can be deployed at the same time by using the following gcloud command:
gcloud app deploy app.yaml dispatch.yaml
The deployment is always done to .appspot.com, the custom domain is automatically updated if it is already configured. If it is not then it needs to be configured (it can be done after the deployment as well).
In Google App Engine, I have 3 services, 1 for front end, 2 for back end.
Is there a way to block http calls to my backend services for accounts not from my company's domain (and the service account of the front end), but allow everyone http access to my front end service?
I know there is the firewall option, but this is restricted to IP addresses, I would prefer user based
If it matters all services are python3
There's currently no option to filter traffic to specific App Engine services within a single application/project:
App Engine Firewall filters by source IP ranges but can only be set for the whole app, not per service.
Identity-Aware Proxy can filter access by user account as you'd prefer but also applies to the whole app. Also, it only supports user account and can't be used with service accounts.
One option you may have would be to split your app in 2 different projects. Keep the front-end in one project open to the world and restrict access to the backend services in your other project via firewall rules.
I have seen the following being used in task queues in GAE. Maybe it would help.
If u were using python 2, in standard environment, i think u could have used login handler element in app.yaml file.
You could have added following lines to your app.yaml file:
handlers:
- url: /.*
script: worker.app
login: admin
This prevents other users from accessing this service.
But the same login handler is not available for python3, according to Google Docs.
Just found following in Google Docs:
If a task performs sensitive operations (such as modifying data), you might want to secure the handler URL to prevent a malicious external user from calling it directly. You can prevent users from accessing task URLs by restricting access to App Engine administrators. Task requests themselves are issued by App Engine and can always target a restricted URL.
You can restrict a URL by adding the login: admin element to the handler configuration in your app.yaml file.
You can also call your backend services through cloud tasks or task queues (both are almost the same i guess), in case this only work for cloud tasks.
Find the code usage here:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/6f5f3bcb81779679a24e0964a6c57c0c7deabfac/appengine/standard/taskqueue/counter
Find details about handler here.
https://cloud.google.com/appengine/docs/standard/python/config/appref#handlers_element
Find details about Cloud task and queue here:
https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-handlers
Firstly, just letting you know I have searched a fair bit here and I am aware of some of the other questions on this topic but none answer my question.
The authentication of the Local GAE differs from the appspot deploy and I need it not to with minimal work-around code.
I'm writing an HTML5 app and I can do the google authentication via a button and it updates all the correct tokens so I can access the profile in either GAE Launched apps or appspot deployed ones.
I need the google account details of the logged in user within the app
I am writing (for API calls to calendar and contacts for example)
, and I'd rather not have to write a login handler only for my local development platform - automated for simplicity or otherwise.
I've read that adding login:required forces a login, and on appspot this works perfectly. Locally it does nothing useful.
I've read that you can write a Python decorator to use #login_required - but I'm not writing in Python (It's php generating an HTML5 page). I could write a bit of a PHP wrapper to handle it, or automate a call in Javascript on page load - but this is the workaround I don't want to write because it's handled in the production environment for me.
I want the login:required option as everything is handled for me in
production
. I have googled the options for the login tag and nothing there suggests I can force a google login in the locally launched app. I have googled the launcher and settings, but nothing seems apparent.
I suppose I could live with the dev workaround, and the app could assume I'm authenticated and the JSON request handlers in my app would just use the login:required with the correct google tokens being passed once I am authenticated.
Do I have any other options?
This sounds like it could be a PHP runtime bug. login:required works fine on the python local dev server. Have you checked the issues page to see if it's been reported?
https://code.google.com/p/googleappengine/issues/list
Otherwise it's possible that it's bypassing the login on your dev server because you have some cookie in your browser indicating that you're already logged in. You might try clearing cookies
Alternatively (at least on the python devserver), you can go to your login page via http://localhost:8000/_ah/login to force a logout (obviously fix the hostname and port number)
There appears to be no way round this other than to write the whole OAuth handler yourself (or get one elsewhere) - significant overkill for a development environment only 'issue'.
I have written the app to handle the getting of the google profile details as it starts and force an authenticate if they are not present.
This means that the login:required will work as expected in the production world and force you to authenticate to google before you even get to the application... then the app just gets the profile details because the tokens are already present.
login:required in the dev environment just puts up a screen which you just 'ok', then the app attempts to gets the profile details but forces the authentication itself because there are no authentication tokens present.
It's unfortunate, but it's a single step in a development that users will not have to use, but it works.
I have a Dynamic Backend setup on GAE which I want to run using cron every 15mins. The problem is cron requires a url that begins with "/". While a backend URL uses the following format: http://backendname.yourapp.appspot.com.
I've read in other forums that you could use fetchurl to call your backend but I don't think that's the ideal way. Because that would require you to make your backend publicly accessible.
According to google's documentation:
http://code.google.com/appengine/docs/java/backends/overview.html#Public_and_Private_Backends
"Private backends can be accessed by application administrators, instances of the application, and by App Engine APIs and services (such as Task Queue tasks and Cron jobs) without any special configuration."
Has anybody got backends called by declaring it in cron.xml?
Use target tag to specify a backend in your cron.xml.
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
<cron>
<url>/long-task</url>
<description></description>
<schedule>every 30 minutes</schedule>
<target>name-of-the-backend</target>
</cron>
</cronentries>
urlfetch can be used to access internal-only URLs such as private backends - if that weren't possible, there'd be no way to communicate within your app! A better idea, though, might be to use the Task Queue, which can be configured to run tasks against a backend.
I haven't try google app engine's Backends functions, because its pricing model.
However, according to the documents:
http://code.google.com/appengine/docs/python/backends/overview.html#Public_and_Private_Backends
Backends are private by default, since they typically function as a component inside an application, rather than acting as its public face.
I think GAE will automatically hide the backend url from public http visiting (even without the needs to configure the app.yaml). You may test its behavior by logout and access the backend url directly.