I want to ingest logs from an appengine app to bigquery without using appengine mapreduce ?
We've open sourced a Java implementation of migrating appengine logs to bigquery here: http://blog.streak.com/2012/07/export-your-google-app-engine-logs-to.html
See the bigquery docs here. You can post a multi-part http request that has the data you want to add to the table. If you are doing an append you won't need to provide the schema.
There's also a Python runtime implementation called "log2bq" that demonstrates how to ingest App Engine logs into BigQuery: http://code.google.com/p/log2bq/
Google has recently released a (BETA) feature called "Google Cloud Logging: Logs Export"
https://cloud.google.com/logging/docs/install/logs_export
They summarize it as:
Export your Google Compute Engine logs and your Google App Engine logs to a Google Cloud Storage bucket, a Google BigQuery dataset, a Google Cloud Pub/Sub topic, or any combination of the three.
I haven't tried out all of the functionality provided by this new service but...
We have recently started using the "Stream App Engine Logs to BigQuery" feature in our Python GAE project. This sends our app's logs directly to BigQuery as they are occurring to provide near real-time log records in a BigQuery dataset.
Related
I'm trying to connect and get data from Google Cloud Datastore API.
After disabling my App Engine Application (which have no use for me at the moment), I cannot access any GC Datastore API methods.
When I send my request, I always get the following response:
{ [Error: The project [my-project] does not exist.] code: 404, metadata: Metadata { _internal_repr: {} } }
I've already double-checked this and my project name is correct.
Accessing the google cloud console, I noticed that the datastore dashboard also show the same message:
Then, after enabling the app engine application again, everything works.
I know that GC Datastore was initially created to work with App Engine, but since last year it was supposed to be available everywhere (according to this).
Someone have a tip about this issue? Any help would be appreciated.
Thanks.
The Google Cloud Datastore API currently requires that the associated App Engine application not be disabled.
I've filed https://github.com/GoogleCloudPlatform/google-cloud-datastore/issues/107 to track this. It's something we may be able to fix in the future.
We got a couple millions data in the current GAE project using Google Cloud store. Mostly GPS point information. We want to be able to use all these GPS points in another demo instance, which is hosted in another GAE instance. Anyway we can do it?
Using Golang + Google App Engine
There is a Google Cloud Datastore API that you can use to access your Datastore data from any other deployment, including a different App Engine app. It's not available in Go, so you will have to mix in some Python or Java.
I would like to import data from flat files stored in Google Drive into DataStore. Then use the full-text search and other query options to analyze the data using apps-script.
The script API doc shows how we can access Google Drive data from the apps-script.
Now, is there any API in apps-script to access DataStore from the scripts?
Google provides a (beta) REST API to access your data. Steps to enable are here.
However, BigQuery is usually better for the type of analysis you describe. See:
https://developers.google.com/apps-script/advanced/bigquery
At this point in time, you should consider writing your own Web Service to get you access to the DataStore. You can then access that Web Service hosted in your App Engine application from your App Scripts. A detailed example is provided over here.
Additionally, the Cloud DataStore is now provided under the Google Cloud Platform and while it is still in preview, there is an API available to interact with the Datastore. This API is exhaustive and allows for both read and write operations. But keep in mind that it is currently under Preview.
I was having a look at the new Google Cloud Datastore and looks great. But there is something I could not understand... is it supposed to substitute Google App Engine Datastore? How can I use it inside GAE? What are the differences between both of them?
I have a GAE app in Java that uses 3 entities with thousands of rows each one, and I need to do joins quite often...
The cloud datastore is the App Engine datastore, for use outside App Engine. You won't get any benefit trying to use it with your App Engine app, unless you need other external apps to also have access to the data.
You certainly won't get more efficient joins. If you really need that, perhaps you should look into Cloud SQL, which is basically a version of MySQL you can use from App Engine.
They both are the same, in fact Google App Engine can use Google Cloud DataStore as one of its way to store data, the other options include Google Cloud SQL, Google Cloud Storage. You may select among these three according to the type of data you want to store and the way you want to access them.
From your question its clear that Google Cloud SQL would be right choice as no other options provide joins for retrieving results.
Cloud SQL is nothing but MySQL(the popular open source database) running on google platform. So you can perform your regular SQL-like queries to get your results.
I just want to know if it is possible to download and upload the whole database from Google App Engine to a SQLite db, in case I want to transfer my app from Google App Engine to another server?
What is the chance?
Thank you!
There is a project called approcket that synchronizes Google App Engine datastore and a local MySQL database. Maybe you should have a look.
In a related answer, I linked to the documentation for the bulk loader tool.
However you will be more interested in the rest of the document, especially the section on creating an exporter class for exportation of the data.