I'm trying to implement a simple design in google cloud using app engine standard and flexible with datastore. App1 lives in GAE standard environment. When a user interacts with this app, it writes some data to datastore and queues a task. The target of the queued task is App2 that lives in app engine flexible environment (the task can take a longer time to complete than standard environment allows). The idea is for App2 to read the data from datastore, perform the task using the data, once complete it should write a report entity to datastore. I've attached a simple diagram.
In App1 I've set up a Service Account named flexKey with Owner permissions, downloaded the json file.
when I run App2 locally I first export the path to the credentials json file as an environment variable:
GOOGLE_APPLICATION_CREDENTIALS: "path/to/flexKey.json"
then launch the app with mvn jetty:run-exploded and everything works fine, App2 is able to authenticate with live datastore (not local emulation), and read the data written by App1. When I unset the environment variable, I get an 'Unauthenticated' error (expected)
To use the same service account when App2 is deployed I've added the following in app.yaml for App2 to set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the service account json file flexKey.json (this is the path to the file on the deployed instance):
env: flex
env_variables:
GOOGLE_APPLICATION_CREDENTIALS: "/var/lib/jetty/webapps/root/WEB-INF/classes/flexKey.json"
runtime: java
However, when I deploy App2 to app engine flexible environment, there is an error authenticating with datastore when trying to do the read query (this works fine when querying datastore with the same credentials from a locally running instance of App2):
com.google.cloud.datastore.DatastoreException: Missing or insufficient permissions.
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:114)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:182)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:178)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:174)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:173)
.....
the code is PERMISSION_DENIED
If I leave out the environment variable GOOGLE_APPLICATION_CREDENTIALS from the app.yaml file for App2, then I get an 'Unauthenticated' error, so I think it's reading the file so I'm not sure what the issue is
I'm using Objectify v6
I'm not able to see why the same credentials (created with Owner role) work fine when querying the datastore from a locally running instance of the app but don't work when datastore is queried from the deployed version of the same app (deployed to flexible environment). setting the path to the credentials file via an environment variable in app.yaml is the method recommended in the documentation unless I am mistaken.
Is the GOOGLE_APPLICATION_CREDENTIALS environment variable not properly set in app.yaml?
Is there something conceptually problematic about my design?
All help appreciated.
You have a blocking issue in your design: it is not possible for one application to enqueue tasks into a push queue targeted at a service from another application. From the <target> (push queues) rows in the Syntax tables for both queue.yaml and queue.xml references:
The string is prepended to the domain name of your app when
constructing the HTTP request for a task. For example, if your app ID
is my-app and you set the target to my-version.my-service, the
URL hostname will be set to
my-version.my-service.my-app.appspot.com.
If you want to use the task queue then you have to make the 2 services part of the same application. As a (positive) side effect you don't have to worry about setting up the authentication for datastore access anymore - both services can directly access the app's datastore.
Related
For using the google pub sub we need key.json file which serves as service account.
For deployed code we can pass the service_account: name#service-name.iam.gserviceaccount.com in app.yaml which works if everything is in google cloud.
As it is not recommend to download the service account file.
Is there a way where we can just pass the key name in local environment file along with individual credential who has access to service account in run time or any other way where we can run google cloud services in local enviornment without downloading the service key file?
Google makes this really hard to find, but gcloud auth application-default login might do what you need.
From that page:
Obtains user access credentials via a web flow and puts them in the
well-known location for Application Default Credentials (ADC).
This command is useful when you are developing code that would
normally use a service account but need to run the code in a local
development environment where it's easier to provide user credentials.
The credentials will apply to all API calls that make use of the
Application Default Credentials client library. Do not set the
GOOGLE_APPLICATION_CREDENTIALS environment variable if you want to use
the credentials generated by this command in your local development.
This command tries to find a quota project from gcloud's context and
write it to ADC so that Google client libraries can use it for billing
and quota. Alternatively, you can use the --client-id-file flag. In
this case, the project owning the client ID will be used for billing
and quota. You can create the client ID file at
https://console.cloud.google.com/apis/credentials.
I have been using the Google App Engine Launcher's "Deploy" button. I realize that the Launcher is going away at the end of July 2020, so I'm trying to learn to use gcloud. My web site has a custom url, www.xxx.org, but "gcloud app deploy" uploads to xxx.appspot.com instead of to www.xxx.org, presumably because "target url" is set to xxx.appspot.com. Can someone please tell me what gcloud command I need to use to change "target url" to www.xxx.org, or some other scheme that would work? I've wandered through lots of gcloud and GAE documentation and web searches without being able to figure out what I need to do.
There is a very good and complete guide about mapping a custom domain to your application in the official documentation. As #DanCornilescu said,it needs to be configured, but it is not necessary to be done before deployment.
Basically, you need to follow the steps from the guide:
1) Verify that you are the owner of your domain through Webmaster
Central
2)Ensure that your domain has been verified, otherwise you will not be able to proceed with the following steps. Note that only verified domains will be displayed.
3)If you need to delegate the ownership of your domain to other users or service accounts, you can add permission through the Webmaster Central page
4)After you verify ownership of your domain, you can map that domain to your App Engine app
5)Add the resource records that you receive to the DNS configuration of your domain registrar
6)Test for success by browsing to your app via its new domain URL, for example www.example.com
In case you need to do more complex mappings for different urls and subdomains you can always write down the routing rules in the dispatch.yaml file and they will be mapped successfully.
Also, to add a bit more of information about the dipatch.yaml file, it will handle the routing in the App as opposed with the app.yaml which will take care of the App's settings. Both .yamls can exist in the same application and can be deployed at the same time by using the following gcloud command:
gcloud app deploy app.yaml dispatch.yaml
The deployment is always done to .appspot.com, the custom domain is automatically updated if it is already configured. If it is not then it needs to be configured (it can be done after the deployment as well).
In Google App Engine, I have 3 services, 1 for front end, 2 for back end.
Is there a way to block http calls to my backend services for accounts not from my company's domain (and the service account of the front end), but allow everyone http access to my front end service?
I know there is the firewall option, but this is restricted to IP addresses, I would prefer user based
If it matters all services are python3
There's currently no option to filter traffic to specific App Engine services within a single application/project:
App Engine Firewall filters by source IP ranges but can only be set for the whole app, not per service.
Identity-Aware Proxy can filter access by user account as you'd prefer but also applies to the whole app. Also, it only supports user account and can't be used with service accounts.
One option you may have would be to split your app in 2 different projects. Keep the front-end in one project open to the world and restrict access to the backend services in your other project via firewall rules.
I have seen the following being used in task queues in GAE. Maybe it would help.
If u were using python 2, in standard environment, i think u could have used login handler element in app.yaml file.
You could have added following lines to your app.yaml file:
handlers:
- url: /.*
script: worker.app
login: admin
This prevents other users from accessing this service.
But the same login handler is not available for python3, according to Google Docs.
Just found following in Google Docs:
If a task performs sensitive operations (such as modifying data), you might want to secure the handler URL to prevent a malicious external user from calling it directly. You can prevent users from accessing task URLs by restricting access to App Engine administrators. Task requests themselves are issued by App Engine and can always target a restricted URL.
You can restrict a URL by adding the login: admin element to the handler configuration in your app.yaml file.
You can also call your backend services through cloud tasks or task queues (both are almost the same i guess), in case this only work for cloud tasks.
Find the code usage here:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/6f5f3bcb81779679a24e0964a6c57c0c7deabfac/appengine/standard/taskqueue/counter
Find details about handler here.
https://cloud.google.com/appengine/docs/standard/python/config/appref#handlers_element
Find details about Cloud task and queue here:
https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-handlers
I have multiple small to medium sized projects all hosted under my current Rackspace server at apps.foo.com. I would like to move these to an Google App Engine Instance & though I have managed to move a few over, I tend to keep hitting the 10,000 file limit.
Hence I've decided to go with 2 separate App Engine instances though I would like the same domain name to point to them with a setup like apps.foo.com/m1 and apps.foo.com/m2, How can I do this?
I've already migrated the domain name to one instance though I can't figure out how to add another. Please help!
You can't map the same domain to 2 different GAE apps - GAE wouldn't know to which one of the 2 apps hypothetically mapped to the same domain to route an incoming request for the domain.
The request path following the domain is not part of the domain, it is only parsed (following the destination app's parsing rules) after GAE has already selected the destination app based on the request domain.
You might be interested in my recent reply to this Q&A related to reaching deployment quota: Getting error on GAE: Max number of files and blobs is 10000
You can use dispatch to reroute requests to the relevant service.
Deploy your API & WebApp to the same project but as separate services (using the service attribute in the app.yaml file).
Deploy the dispatch
dispatch.yaml
- url: "project-name.appspot.com/api/*"
service: api-service
- url: "project-name.appspot.com/*"
service: web-client-service
For my WebApp's index.html I added also:
<base href="https://project-name.appspot.com/">
I am developing an app locally in Google App Engine. I have built a small datastore for development purposes. Rebuilding it after every power cycle on my Mac got tedious so I made it permanent. Now I run my app locally with the following command:
/usr/local/bin/dev_appserver.py "--datastore_path=./permanent.datastore" appengine_prototype
Life is good. I have decided to deploy my app so I can test http post commands from a different machine. When I tried to register my current application id (example), I found that it was unavailable (shocker!). So I registered a different application id and planned to change my local application id to match. However, when I changed the
application: *app-id*
line in my app.yaml file, my app stopped recognizing my permanent datastore.
So, how can I change my application id to the one I registered, maintain the connection to the permanent datastore and then push the whole shebang online? I tried running the app twice locally, first with the permanent datastore referenced in the command and then without, hoping that the default temporary datastore would inherit from the previous permanent datastore. That didn't work. Do I need to start by copying the permanent datastore to the default temporary datastore? How would I do that? Any help would be much appreciated.
Thanks,
Dessie
If your intention is to eventually push your local data to your live environment anyway, then your best bet is to:
use bulkloader.py to backup your local data (while using oldid in your config)
then change your config to your newid
then use bulkloader.py to push your data to your new development server (ran with --datastore_path=./permanent.datastore2 or something)
then use bulkloader.py to push your data to the GAE production server
Details of bulkloader.py can be found in the docs and an example here