Having a tough time getting the Default Application Credentials to load in the dataflow SDK when running locally in a java app engine project developing on OS X. Runs fine when deployed.
According to this the dev app server doesn't support them, and you're meant to use the gcloud command line tool's command: gcloud preview app run - but according to the official Google Group for the SDK here that command was deprecated in Jan 2016.
So I seem to be stuck between a rock and a hard place... Does anyone know how to get the Application Default Credentials to work locally with an App Engine app?
I'm trying to use the Dataflow API and it just throws up when it starts making use of the cloud storage api which is the first thing the Pipeline does because it can't seem to load the correct credentials from the environment variables ( that are definitely set on the ENV and in the appengine-web.xml <env-variables> element ) or from the ~/.config/cloud/default_application_credentials.json file.
Cheers!
Can you try running the following command and see if it solves it?
gcloud auth application-default login
This is fully supported (but poorly documented) within the dev appserver. There is a very well answered question that gives you step by step instructions here: Unable to access BigQuery from local App Engine development server
Related
I have deployed source code via gcloud command line with no issue. However, I am currently away from my desktop and see a critical change to my app.yaml file that I would like to make.
Is this possible to do via my Google Cloud account?
You can use the App Engine Admin API to patch the specific version of your service and update the instance type since your app is using App Engine Standard. You can use the "Try this API" feature to update it right from your browser.
We are planning the migration of an internal app running on Google App Engine Standard Environment for Java 8 from the now superseded App Engine APIs to the recommended client library for Cloud Firestore in Datastore Mode, also in order to enable porting to other execution environments.
Besides a host of issues with missing IN/OR query operators, we are also struggling with the setup for local testing: according to Using the Java 8 Local Development Server
The development web server simulates Datastore using a local
file-backed Datastore on your computer. The Datastore is named
local_db.bin, and it is created in your application's WAR directory,
in the WEB-INF /appengine-generated/ directory.
but we have no clue about how to connect Google Cloud Client Library for Datastore to the local emulator.
Defining default credentials with
gcloud auth application-default login
or setting the GOOGLE_APPLICATION_CREDENTIALS environment variable after obtaining the credentials for the service account with something like
gcloud iam service-accounts keys create key.json \
--iam-account=project-id#appspot.gserviceaccount.com
as suggested elsewhere, just results in the client library connecting to the actual cloud server, rather than to the local emulator, as per the ADC policy.
I'd expect the development server to automatically provide connection hints to the client library, but that's apparently not the case.
Any suggestion for setting up a local testing environment, taking into account that we can't just migrate to the standalone Datastore Emulator, as we need other services currently provided only by the App Engine development server (e.g. email submission)?
Edit / After further tinkering we are working around the issue by using both the Local Development Server and the standalone Datastore Emulator as:
gcloud beta emulators datastore start \
—project=project-id \
--host-port=localhost:8081 \
--data-dir=target/war
DATASTORE_EMULATOR_HOST=localhost:8081 java_dev_appserver.sh \
--port=8080 \
target/war
However, the process is quite cumbersome and difficult to automate: what we are looking for is a way to automatically connect Google Client Library to the Datastore Emulator managed by the Java 8 Local Development Server when launching the app with something like the App Engine Maven plugin, e.g. mvn appengine:run.
This GitHub issue was closed with the confirmation that the Datastore Client Library is not compatible with the local Web Server Datastore Emulator.
I actually tried it, to see if was possible to force a connection to the local Web Server. The code below sets a custom builder with the desired host configuration:
DatastoreOptions.Builder builder = DatastoreOptions.newBuilder();
builder.setHost("http://localhost:8080");
builder.setProjectId("<PROJECT_ID>");
Datastore ds = builder.build().getService();
Key key = ds.newKeyFactory().setKind("MyEntity").newKey("mykey");
Entity entity = Entity.newBuilder(key).set("p1", "Hello World!").build();
entity = ds.put(entity);
System.out.println(entity);
entity = ds.get(key);
System.out.println(entity);
After running the local Web Server I noticed a connection was indeed possible, however the Datastore Client Library returned the following error when trying to store new entities:
[INFO] GCLOUD: com.google.cloud.datastore.DatastoreException: Non-protobuf error: <html><head><title>Error 404</title></head>|<body><h2>Error 404</h2></body>|</html>. HTTP status code was 404.
With the following output by the Web Server:
Oct 02, 2019 3:05:59 PM com.google.appengine.tools.development.jetty9.LocalResourceFileServlet doGet
WARNING: No file found for: /v1/projects/<PROJECT_ID>:commit
I believe this adds to the confirmation that the new library is just not compatible to the old emulator.
The workaround you found is probably the best solution while you work/wait on the full migration to the Datastore mode Emulator.
I am currently using Google's 'legacy standalone SDK' for management of my google app engine python projects. This provides a nice GUI where I can select and run a project locally and then deploy it when I'm happy with testing.
I recently received the following message from Google '... App Engine app that has used the legacy standalone SDK in the past 30 days, and need to be migrated to Cloud SDK' so I installed the Cloud SDK.
The provided Google Cloud SDK Shell a) runs as a dos window, and b) appears to duplicate the on-line (live) Dashboard. Given that I can't change (local) code and deploy without testing, my questions are:-
1) how do I run a project locally?
2) how do I view local logs?
3) when happy, how do I deploy?
or... is there GUI for the Local Environment?
or... is the above documented somewhere that I have not found?
Many thanks and apologies if this is a dumb question.
David
On July 30, 2019, the standalone App Engine SDK is deprecated. Here you can check documentation how to migrate from AppCft to gcloud command line
How to test your application locally?
For Python you can use Local Development Server that includes Cloud Datastore Emulator. (There is also solution for Java Java 8 Local Development Server).
How to view logs locally?
In python to debug your code from the command line use interactive console. You have to import to your code PDB debbuger
How to deploy application?
You deploy your app with gcloud command gcloud app deploy. Cloud SDK does not include GUI for deployment.
In my previous App Engine projects I used the Cloud Datastore, and during development I could debug my app on the local server and it would use a local database, stored in a file I could wipe out if I wanted to start from scratch.
With Cloud Firestore, even when I'm running locally it's talking to my real cloud database. Is there still a local option? Note that I'm not talking about client-side persistence, I'm talking about a mock development DB.
Google recommends setting up multiple projects if you want dev/staging/production, and I'm guessing that's the answer, but I'd like to know before adjusting my workflow.
I think (now only a few months later) that this is supported. When I run my app, using dev_appserver.py, I see a message
INFO 2019-02-14 00:08:56,030 admin_server.py:150] Starting admin server at: http://localhost:8000
Going to that URL shows me all the instances I have been seeing. These seem to persist even when the dev_appserver is restarted. Reading this and other posts I was convinced that my development was using my actual cloud database, but going to https://console.firebase.google.com/project/myproject was showing completely different content.
Just to be sure (because google is google and everything is named the same) I'm using an appengine app and a gcloud project, storing things to Firestore using ndb.Models...
Oh, but careful. My app I was also using the cloudstorage (blobstore?) and even though the localhost:8000 showed these, THESE WERE THE REMOTE INSTANCES.
There is a local emulator for Firestore when using the Firebase CLI:
https://firebase.google.com/docs/rules/emulator-setup
My team and I are working on the Trendy Lights Tutorial.
We have set up all the files and also have converted the .p12 key to .pem key but failed to run the app on the Google Cloud Platform and constantly got the error message saying:
You do not have permission to access project [...] and service
"cloudbuilt.googleapis.com" is not for consumer..."
We have already whitelisted our service account for the use of Earth Engine. Does anyone know what the problem might be?
Thank you so much!
I see that you are attempting to deploy your application to the App Engine Flexible Environment.
The Flexible environment differs from the Standard environment in that it gives you more control over the individual instances that are running your deployed application. It does this by hosting your application within Docker on Compute Engine virtual machines.
Therefore, you will need to enable the Compute Engine API for your project so that the GCloud tool can start new Compute Engine virtual machines when you deploy your application.
Note: Since the Flexible environment uses Compute Engine resources, you will also need to enable billing for your project.
If after performing the above you still experience the 'cloudbuild.googleapis.com' error, I ask that you run gcloud components update, then ensure that the value of account seen in the output of the command gcloud info has Owner or Editor permissions in your project.
If it still persists after all of the above, you should then open a Public Issue Tracker to inform our backend team of the issue.