I'm trying to access Google Datastore from a standalone Java app (not a web application or servlet), using the low-level Java API, running on my local system.
remoteapi appears the simplest way do this. However, I need to specify a server per
RemoteApiOptions options = new RemoteApiOptions()
.server("your_app_id.appspot.com", 443)
.useApplicationDefaultCredential();
How do I specify the server(...) invocation to access Datastore? I have run
gcloud auth login
and I have a project configured for Datastore (with a project ID, etc.) but no 'app_id' since there's no app involved.
Is there a standard server for accessing Datastore? Or is there a better way to access Datastore from a standalone java app?
To access Cloud Datastore outside of App Engine, follow the docs, which do not involve the Remote API. The complete example given on that page uses Maven (for the Java version), but, if that's not how you prefer to develop, you can look at the sources just as a usage example, and separately install the Google Cloud Client Library for Java; for running standalone (not in App Engine or Compute Engine), you'll explicitly authorize, as per the example on that page...:
import com.google.gcloud.AuthCredentials;
import com.google.gcloud.datastore.Datastore;
import com.google.gcloud.datastore.DatastoreOptions;
import com.google.gcloud.datastore.Entity;
import com.google.gcloud.datastore.Key;
import com.google.gcloud.datastore.KeyFactory;
DatastoreOptions options = DatastoreOptions.builder()
.projectId(PROJECT_ID)
.authCredentials(AuthCredentials.createForJson(
new FileInputStream(PATH_TO_JSON_KEY))).build();
Datastore datastore = options.service();
KeyFactory keyFactory = datastore.newKeyFactory().kind(KIND);
Key key = keyFactory.newKey(keyName);
Entity entity = datastore.get(key);
Related
As Python 2 came to an end and App Engine 1 is no longer supported, we are migrating to App Engine 2. I have a sizable Datastore. Given that the two App Engines need to be on two different projects, is it possible to connect to the old App Engine's datastore as the new App Engines's datastore? Ideally using NDB.
The datastore of a GAE/GCP project can be accessed from an app in another project or even from outside Google Cloud, see How do I use Google datastore for my web app which is NOT hosted in google app engine?.
I didn't yet play with the python 3 ndb library (aka Cloud NDB), I can't exactly say if/how it can use another project's credentials. I'm unsure if you want to try to use it, though: from Migrating to Cloud NDB:
Cloud NDB is intended to replace the features in App Engine NDB, so it
will not support new features of Firestore in Datastore mode. New
Python 3 apps should use the Datastore mode client library
instead of Cloud NDB.
In the worst case the Datastore mode client library (actually the recommended one for python 3 GAE apps) should be able to access your python 2 app's datastore - it is generic, it can be used for any app, not just GAE. According to the docs its Client() method supports specifying a project and credentials (where you'd be using the service account for the python 2 app's project). Potentially of interest: GCP-The App Engine APIs are not available, with py 3
I have a situation where an existing GAE App (let's call it app A) is running, but for non-technical reasons can't be modified. As users migrates to a new client version, we need to migrate their data from app A to a new GAE app (which I'll call app B).
Is there a way that I can grant app B access to app A's live datastore without modifying app A? My not modifying I mean not having to deploy new code. Changing setting or permissions in the Cloud Console is fine.
In case it matters, both apps that I'm referring to are written in Go.
It might not be possible to share the datastore across multiple GAE apps using the Google App Engine Standard Environment Client Libraries. At least for python it's not possible, donno about go.
But the Cloud Datastore Client Libraries can be used to share a datastore across many apps, even from outside Google Cloud.
Regardless of the particular way the old app accesses the datastore (language/library/etc.) it can be configured from the Cloud Console to allow access to a remote app. The exact procedure steps are captured in How do I use Google datastore for my web app which is NOT hosted in google app engine?
The new app would be using the above-metioned client library with the old app's service account credentials (obtained in the above paragraph procedure) to access the old app's datastore.
ndb: (from google.appengine.ext import ndb)
datastore: (from gcloud import datastore)
What's the difference? I've seen both of them used, and hints they both save data to google datastore. Why are there two different implementations?
The Python NDB Client is specific to Python Applications running on Google App Engine. The datastore client removes that restriction and you can run your Python application anywhere, including Google App Engine, Google Compute Engine or anywhere else.
Exceprt form - https://cloud.google.com/appengine/docs/python/ndb/
The Google Datastore NDB Client Library allows App Engine Python
apps to connect to Cloud Datastore.
In addition, the NDB client has certain features (e.g. caching) while the other does not appeat to support.
The reason for the two implementations is that originally, the Datastore (called App Engine Datastore) was only available from inside App Engine (through a private RPC API). On Python, the only way to access this API was through an ORM-like library (NDB). As you can see on the import, it is part of the App Engine API.
Now Google has made the Datastore available outside of App Engine through a restful API called Cloud Datastore API. gcloud library is a client library that allows access to different rest APIs from Google Cloud, including the Cloud Datastore API.
My app is running on App Engine, but I would like to access its NDB DataStore entities from my Compute Engine VM to do some processing and write the results back to the App Engine DataStore. How can I do that?
Also, are the Google Cloud DataStore and App Engine DataStore the same thing?
https://developers.google.com/datastore/
https://developers.google.com/appengine/docs/python/ndb/
David's solution requires you to use App Engine instance time to make requests, but you can bypass it and make requests directly to Datastore from Compute Engine instance. There is a pretty good tutorial about how to do this. But its not so pretty like ndb.
>>> import googledatastore as datastore
>>> datastore.set_options(dataset='project-id')
>>> req = datastore.BeginTransactionRequest()
>>> datastore.begin_transaction(req)
<datastore.datastore_v1_pb2.BeginTransactionResponse object at ...>
Google Cloud Datastore is kind of a standalone version of App Engine's datastore.
Back to your problem, you have two options :
Write a web-service to expose your entities from the App Engine app to the Compute Engine VMs. One option is Cloud Endpoints. Cloud Endpoints uses OAuth2 authentication and the Compute Engine VMs are shipped with OAuth2 service accounts that you can use to authenticate to the service.
Use the App Engine remote API to access the Datastore. The remote API provides access to the Datastore as if you were in an App Engine instance. But by default the API requires you to provide the password of an App Engine amdin, so you might have to store your password in the Compute Engine VMs, which is not safe.
Aside from options that #David explained, you can also look into Managed VMs: they are a cross between Compute Engine and App Engine - basically a managed Compute Engine instance that has access to App Engine services.
The officially recommended way is to use the datastore client libraries; see https://cloud.google.com/datastore/docs/reference/libraries
You need to create a service account, or use the standard compute engine service account, give permission to either all APIs, or to datastore to that service account, and create the compute engine instance to be part of that service account. See here for more information.
Then you can do something like:
from google.auth import compute_engine
from google.cloud import datastore
datastore_client = datastore.Client(project='yourproject',
credentials=compute_engine.Credentials())
q = datastore_client.query(kind='YourEntity')
q.add_filter('field_name', '=', 'HelloThere')
print list(q.fetch(1))
I have seen many references to using GQL in the "GAE Datastore viewer" but I just cant find a place to do that...
I am using the GAE JAVA SDK v1.8.8 and accessing the Data Store viewer via http://localhost:8899/_ah/admin/datastore.
Was this option removed? or should I somehow enable it?
Using GQL is not part of the local development server. It is available in the live App Engine console for your application.
The development web server includes a Datastore Viewer (/_ah/admin/datastore) via which you can only browse the local datastore.