We have a large dataset from an appengine app in our datastore. Now I want to do some ETL on them to push them to bigquery, and I thought of using a Dataflow batch job.
All examples I find are using this class to query the Datastore:
import com.google.api.services.datastore.DatastoreV1.Query;
And that does work. However, I'm not familiar wit this DatastoreV1 API and would like to use the API provided with the appengine SDK, like this:
import com.google.appengine.api.datastore.Query;
The problem is that the DatastoreIO doesn't accept these queries:
PCollection<Entity> projects = p.apply(Read.from(DatastoreIO.source().withQuery(q).withDataset(DATASET_ID)));
It will only take DatastoreV1.Query objects. Is there any way to use the app engine provided API's? I'm much more familiar with those calls. Better yet, if we could use Objectify, that would be awesome :)
Thanks!
This isn't possible with the current implementation of the API. We can look at adding as a feature, and would gladly accept a pull request to expand the current functionality. The AppEngine team is also actively working on increasing interoperability between their SDK and the Datastore API.
Related
I am deploying a Plotly Dash app on Google App Engine but meet some difficulties. The data source to be queried in the dashboard is a Bigquery table, whose content is changing. I hope that the data in the App can always be the latest.
What I tried is at the beginning of the main.py code, I read in the table from Bigquery by Bigquery Python API, but after the App being deployed onto GAE, I found the data was fixed; even I deleted the Bigquery table, the App was not affected. May I know what is the correct way to get data from BigQuery to App Engine? Thanks.
How to connect to BigQuery from GAE using Python might be a bit more of a task than a single question can answer, but here are some hints:
Everything Google Cloud can (in my opinion) be best understood through the repositories on Github. For instance, the python docs samples contain several examples, out of which I think the client example is probably the easiest and most basic. Bigquery Python Samples are here. That will basically answer your question, except for a few gotchas I will mention.
You will of course need to download the client library to do development on a local environment. That is straightforward, but if something seems not to be working make sure you have enabled the API service account for your project--that can be a little confusing.
Something that is critically important to remember is that your GAE app will not be able to easily communicate with BigQuery if it is in a different region, and, in fact, once you set up a GAE app you cannot move or delete it! So, do pay attention to what you are doing as you set up, and if you have a locations mismatch you will need to migrate your BQ instance to the matching location.
I am somewhat new to Web development - specifically Google App Engine and JavaScript/HTML development, but I have an app deployed and working on Google App Engine and it is working ok.
I would like a user of my App to be able to store and retrieve a serialization of the app state in JSON using the GAE Datastore. (Note - This is only a user-initiated action - so channels seem to be overkill)
The examples provided by Google demonstrates one approach that allows the server-side Python implementation to do this. Specifically https://developers.google.com/appengine/docs/python/gettingstartedpython27/usingdatastore. I have this working ok.
But this approach seems rather inelegant especially if as an "app" I want to store and retrieve serialized chunks of data somewhat asynchronously without reloading the page/app each time (again, this is only ever user-initiated).
I have not been able to find any high-level guidance on an approach to do that (assuming it is possible).
Any suggestions/links/examples would be greatly appreciated.
Thank you!
Jeff
As with many things, this depends on your specific needs. If you just want direct access to datastore storage, the datastore is exposed as an independent service with an API.
If you instead want to assert logic over the usage and interact with your app in some fashion, you may also want to look at Google Cloud Endpoints. With an endpoints API, you gain a more structured API you can call directly from javascript, or generate client libraries to be consumed by other languages/platforms.
I am just getting started with Google Web Toolkit and Google App Engine and have a quick question. I think I understand how to use the datastore now but I was wondering if there is a way that I can quickly create a "database" with static data from an excel sheet? I just need to add some data for a proof of concept later this week.
I am picturing something similar to a SQL database browser where I can just import the data?
I developing in Eclipse with appropriate plugins.
Thanks,
Rob
The easiest way to do this would be to save your spreadsheet as a CSV file, then use the bulkloader to load it into the datastore.
Your best bet is probably to write something to handle uploading it, or to handle processing it on the server.
However, you should also look at the bulk loader. It might be able to save you a little bit of time.
Here is the API (Google Documents List API) that "allows client applications to programmatically access and manipulate user data stored with Google Documents".
Is it possible to make invitations to share/view data similar to Google Docs?
Definitely: GAE is a platform, a canvas for what you want to build. Document access management is entirely up to you, but of course you'll have access to some APIs to facilitate your life (e.g. Google Accounts integration).
Of course it is. You can create virtually any kind of web application that you want with AppEngine. It doesn't come with anything built-in to do this other than an API for sending emails, but you can certainly code up this functionality.
I would like to write a client application for Android that uses the Google App Engine as a database backend. My Android client would connect to the App Engine to save information, then it would connect later for reports. Is it possible to use the App Engine as a backend like this?
If you're looking for something like the remote api that the App Engine has in python, then you'll be disappointed to find it missing in Java.
That said, absolutely nothing stops your from hitting your app and posting data either through POST / JSON / XML / any other format you can think of. The same thing goes for getting your reports back.
If security is a concern, the OAuth protocol allows you to authenticate to app engine from your android device.
This is an aside, but as far as reporting is concerned, you might not find the app engine a very suitable platform for reporting type apps. Just make sure you understand its limitations - the lack of joins, 1000 object limit, no sum / average, necessary indexes, etc. It's certainly not impossible, but do think carefully about how you're going to model your data.
Yes, it is possible.
Without more details in your question, any more details in the answer would be speculation.
Yes, its very much possible. It's something I am also currently working on.
My code uses HTTP GET and HTTP POST and I am using a RESTful service on the GAE.
I'm sorry I can't provide any code because I am still learning however the library I'm using is called RESTLET. They have libraries for both GAE and Android however I'm only using RESTLET on the GAE and I'm just using the HTTP library in the Android SDK for the client.
http://www.restlet.org/
The version you require is 2.0 M6 and not the stable release.
No.
In your response to Laurence, you said you want a direct DB connetion. A client cannot connect directly to the GAE datastore. You must write web handlers to interface between the client and your data. It doesn't have to be much, but it must be something.
Yes, it is very possible. You would not connect directly to the GAE database though. A better architecture would be to make your app hit a URL that writes to the DB. For example, you could set up a Struts 2 action that takes the values of your query parameters and then mutates and validates them as necessary before persisting them.