I have a new Python 3 / Flask app for the GAE App Engine Standard Environment. It works fine in the local test env and also when deployed to the cloud. As long as I do not use Datastore.
My first attempt for local test failed at line
datastore_client = datastore.Client()
when the following was written to the terminal console upon startup of
dev_server.py:
"google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application."
The datastore emulator is running locally on localhost:8081.
These are set:
export DATASTORE_DATASET=xxxxxx
export DATASTORE_EMULATOR_HOST=localhost:8081
export DATASTORE_EMULATOR_HOST_PATH=localhost:8081/datastore
export DATASTORE_HOST=http://localhost:8081
export DATASTORE_PROJECT_ID=xxxxxx
What am I missing? What kind of credentials do I need for a local (my laptop) datastore emulator and how do I set them?
You need to set GOOGLE_APPLICATION_CREDENTIALS per:
https://cloud.google.com/docs/authentication/getting-started
Related
The Go package, google.golang.org/appengine, provides IsDevAppServer which reports whether an App Engine app is running in the development App Server (e.g. localhost:8080). However, this does not work unless the (deprecated) standalone SDK is used. See appengine.go#L57 for the implementation.
New GAE apps written in Go are basically a regular web server that can be compiled and started locally like any go program;
old; dev_appserver.py
new; go run main.go
Detecting a development server can be useful for to prevent CORS issues when running locally:
func setDevHeaders(w http.ResponseWriter) {
w.Header().Set("Access-Control-Allow-Origin", "http://localhost:4200")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
w.Header().Set("Access-Control-Request-Method", "POST, GET")
w.Header().Set("Access-Control-Expose-Headers", "Content-Disposition")
}
When needed I can then branch:
if appengine.IsDevAppServer() {
setDevHeaders(w)
}
What is the recommended way to achieve this in a standalone Go server running on App Engine?
The App Engine Standard Go environnement sets a number of environment variables automatically. You can have a look at the list here.
You can check if they are set and if they aren't, then your code is running locally (or at least not deployed). Or you can set the NODE_ENV environment variable to development on your machine (in your shell where you run your app locally, not in the app.yaml file) and check for its value. It'll be set to production when running on App Engine.
It sounds tricky but I found that metadata server also respond to app engine standard.
Furthur, I used cloud.google.com/go/compute/metadata pacakge,
metadata.OnGCE()
as a quick check whether it is in my local machine or on google's machine (yeah, it returns true on app engine)
I'm running a local instance of a PHP app-engine project, I have got some buckets setup in GCP specifically for the local dev version, however instead of the data that I write to the bucket appearing online, they are being saved locally into the dev_appserver Datastore. I can see the files in the local admin interface (localhost:8000) under Datastore.
This is an issue as the application I'm developing has a companion app which needs to also access the bucket files.
The
--support_datastore_emulator=[true|false]
flag is documented under
dev_appserver.py -h
But it doesn't seem to have any effect when using =false.
So my question is: How do I stop the dev_appserver from using the local Datastore and make it use the 'real' buckets on the web?
Try setting the --default_gcs_bucket_name flag documented here to establish the default GCS bucket to use:
dev_appserver.py app.yaml --default_gcs_bucket_name gs://BUCKET-NAME
I'm trying to implement a simple design in google cloud using app engine standard and flexible with datastore. App1 lives in GAE standard environment. When a user interacts with this app, it writes some data to datastore and queues a task. The target of the queued task is App2 that lives in app engine flexible environment (the task can take a longer time to complete than standard environment allows). The idea is for App2 to read the data from datastore, perform the task using the data, once complete it should write a report entity to datastore. I've attached a simple diagram.
In App1 I've set up a Service Account named flexKey with Owner permissions, downloaded the json file.
when I run App2 locally I first export the path to the credentials json file as an environment variable:
GOOGLE_APPLICATION_CREDENTIALS: "path/to/flexKey.json"
then launch the app with mvn jetty:run-exploded and everything works fine, App2 is able to authenticate with live datastore (not local emulation), and read the data written by App1. When I unset the environment variable, I get an 'Unauthenticated' error (expected)
To use the same service account when App2 is deployed I've added the following in app.yaml for App2 to set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the service account json file flexKey.json (this is the path to the file on the deployed instance):
env: flex
env_variables:
GOOGLE_APPLICATION_CREDENTIALS: "/var/lib/jetty/webapps/root/WEB-INF/classes/flexKey.json"
runtime: java
However, when I deploy App2 to app engine flexible environment, there is an error authenticating with datastore when trying to do the read query (this works fine when querying datastore with the same credentials from a locally running instance of App2):
com.google.cloud.datastore.DatastoreException: Missing or insufficient permissions.
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:114)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:182)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:178)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:174)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:173)
.....
the code is PERMISSION_DENIED
If I leave out the environment variable GOOGLE_APPLICATION_CREDENTIALS from the app.yaml file for App2, then I get an 'Unauthenticated' error, so I think it's reading the file so I'm not sure what the issue is
I'm using Objectify v6
I'm not able to see why the same credentials (created with Owner role) work fine when querying the datastore from a locally running instance of the app but don't work when datastore is queried from the deployed version of the same app (deployed to flexible environment). setting the path to the credentials file via an environment variable in app.yaml is the method recommended in the documentation unless I am mistaken.
Is the GOOGLE_APPLICATION_CREDENTIALS environment variable not properly set in app.yaml?
Is there something conceptually problematic about my design?
All help appreciated.
You have a blocking issue in your design: it is not possible for one application to enqueue tasks into a push queue targeted at a service from another application. From the <target> (push queues) rows in the Syntax tables for both queue.yaml and queue.xml references:
The string is prepended to the domain name of your app when
constructing the HTTP request for a task. For example, if your app ID
is my-app and you set the target to my-version.my-service, the
URL hostname will be set to
my-version.my-service.my-app.appspot.com.
If you want to use the task queue then you have to make the 2 services part of the same application. As a (positive) side effect you don't have to worry about setting up the authentication for datastore access anymore - both services can directly access the app's datastore.
I have a local Datastore emulator instance running on my local machine (Mac OSX 10.11.6).
I have a bunch of data in there already, but I want to remove all data from the local database to start with a clean slate.
However I have already tried deleting the directory (where I thought) the data was being stored.
This message appears when starting the emulator.
WARNING: Reusing existing data in [/Users/myusername/.config/gcloud/emulators/datastore].
I deleted the entire datastore directory and it seems to be re-generated properly when the emulator is restarted. However, the old data is still there.
rm -rf /Users/myusername/.config/gcloud/emulators/datastore
I also confirmed that my Node App is connecting to the local emulator.
myusername$ $(gcloud beta emulators datastore env-init)
myusername$ gcloud beta emulators datastore env-init
export DATASTORE_DATASET=projectname
export DATASTORE_EMULATOR_HOST=localhost:8081
export DATASTORE_EMULATOR_HOST_PATH=localhost:8081/datastore
export DATASTORE_HOST=http://localhost:8081
export DATASTORE_PROJECT_ID=projectname
Datastore emulator supports resetting datastore when its running. This can be used to reset datastore when running tests.
Make POST request to http://localhost:8081/reset to clear the internal state of the emulator i.e
curl -X POST http://<emulator-host>:<port>/reset
see https://github.com/GoogleCloudPlatform/google-cloud-java/issues/1292 discussion
Stop the local dev server from running
Find its dev_appserver.datastore file and drag it to the trash
Restart dev server
A new, empty local datastore will be created.
I have a Google App engine application that I want to work differently depending upon if it is running in my local dev environment (i.e. with dev_appserver.py) as against running in actual GAE cloud.
Currently I use a flag variable that I manually toggle to achieve that. But I am sure one day I will forget to change it and will lead to problem. So I would like to know if there is an API or some other way to figure out where the GAE app is actually running?
Thanks.
You could check the SERVER_SOFTWARE environment variable to see if it is Development/X or Google App Engine/X:
http://code.google.com/appengine/docs/python/runtime.html#The_Environment
I know the original question was for python, but to do this for Java use:
if (SystemProperty.environment.value() == SystemProperty.Environment.Value.Production) {
// running on prod
} else {
// running on dev
}
I use the datastore.get() to pull values.
On the dev_appserver, the datastore contains an entity "Environment" set to "local".
On the GAE dev app, "Environment" is "dev"
On the GAE prod app, "Environment" is "prod"
You can extend to UAT, etc.