In http://blog.notdot.net/2010/09/Under-the-hood-with-App-Engine-APIs, it explains how you can perform an asynchronous datastore get request. I want to perform an asynchronous put request.
How do I do that?
Thanks!
As of GAE 1.5.0 there is 'put_async'.
Your best option for doing asynchronous calls to the datastore at the moment is to use Guido's experimental NDB project, which is a reworking of the App Engine datastore API to support asynchronicity.
My blog post was intended to be instrucitonal, but not as a template for something to do directly - reaching down to that level of the code to do asynchronous requests is likely to be very involved and awkward, and you're much better using a library that does it for you, like NDB.
Related
The documentation I have read has only covered asynchronous pull - I'd like to verify that that is the only option for the Python API.
The Cloud Pub/Sub client library only support asynchronous subscribing, which is the recommended way to run a subscriber. For specific use cases where a synchronous pull is needed, use the REST/HTTP pull method or the gRPC pull method, which requires generating the service code.
I am trying to figure out how to access Google App Engine Memcache service from outside Google App Engine. Any help on how this can be done would be greatly appreciated.
Thanks in advance!
I don't think this is currently possible. I don't know if there is any technical argument for this or if this decision has been made simply for billing purposes. But it seems like memcache is intended to be an integral part of App Engine. The only relevant discussion I could find is this feature request. It calls for possibility of accesing memcached data of one App Engine project by another App Engine project. It seems to me that Google didn't consider such functionality to be beneficial. You could try filing your own feature request to make memcache a standalone service. In case you do not succeed (and I am afraid you won't), here is a simple workaround.
A simple workaround:
Create a simple App Engine project which would serve as a facade over memcache service. This dummy App Engine project would simply translate your HTTP requests to memcache API calls and return the obtained data in the body of a HTTP response. For example, to retrieve a memcache record you could send a GET request such as:
https://<your-poject-id>.appspot.com/get?key=<some-particular-key>
This call would get "translated" into:
memcache.get(<some-particular-key>);
And the obtained data appended to the HTTP response.
Since accessing memcache is free, you would only have to pay for instance time. I don't know what through-put are you expecting, but I can imagine scenarios where you could even fit into the free daily quota (currently 28 hours/day). All in all, the intermediate App Engine project should not come with significant cost in neither performance nor price.
Before using this workaround:
The above snippet of code is intended for illustration purposes only. There still remain some issues to be dealt with before using this approach in production. For example, as pointed out by Suken, anyone would be able to access your memcache if they knew what requests to send. Here are four additional things I would personally do:
Address the security issues by sending some authentication token with each request. An obvious necessity would be to make the calls over HTTPS to prevent man-in-the-middle attackers from obtaining this token. Note that App Engine's appspot.com subdomains are accessible via HTTPS by default.
Prefer batch API calls such as getAll() over their single record alternatives such as get(). Retrieving multiple records in one batch call is much faster than making multiple separate API calls.
Use POST requests (instead of GET) to access the facade application. You won't have to worry about your batch requests being to large. I only used GET request in the example above because it was easier to write.
Check if such usage of App Engine doesn't violate the Terms of Service. Personally, I don't believe it does. And I don't see why Google should mind. After all, you will be paying for instance hours.
EDIT: After giving this some more thought, I believe that the suggested workaround is actually what Google presumes you to do. Given that the Goolge's objective is to earn money, it would be unreasonable to provide a free service unless it was a part of a paid one. Of course, another billing schemes could be created. For example, allowing direct access only for developers who are willing to pay for dedicated memcache. The question is whether your use case is broad enough to convince Google to take some action.
No, AFAIK the Memcache service is not available outside GAE. To be even more specific it is only available inside the GAE standard environment, it is unavailable in the GAE flexible environment.
But some of the alternate solutions suggested for GAE flexible users might be useable for you as well. From Memcache:
The Memcache service is currently not available for the App Engine
flexible environment. An alpha version of the memcache service will be
available shortly. If you would like to be notified when the service
is available, fill out this early access form.
If you need access to a memcache service immediately, you can use the
third party memcache service from Redis Labs. To access this service,
see Caching Application Data Using Redis Labs Memcache.
You can also use Redis Labs Redis Cloud, a third party fully-managed
service. To access this service, see Caching Application Data Using
Redis Labs Redis.
As stated by other users the Memcache is not offered as a service outside GAE (Google App Engine). I would like to point out that implementing GAE facade over Memcache service has security ramifications. Please note that facade GAE Memcache app will be exposed on the public internet like any other GAE service. I am assuming that you want to use Memcache for internal use only. Another aspect to think about is writing into memcache. If you intend to write to memcache from outside GAE, then definitely avoid facade implementation. If comprised anyone will be able to use you facade implementation as their own cache without paying for it ;)
My suggestion is to spin up a stack using GCP Cloud Launcher. There are various stack templates available for both Redis and Memcache stacks. Further you can configure the template to use preemptible burstable instances to reduce the cost of your Memcache.
We have a large dataset from an appengine app in our datastore. Now I want to do some ETL on them to push them to bigquery, and I thought of using a Dataflow batch job.
All examples I find are using this class to query the Datastore:
import com.google.api.services.datastore.DatastoreV1.Query;
And that does work. However, I'm not familiar wit this DatastoreV1 API and would like to use the API provided with the appengine SDK, like this:
import com.google.appengine.api.datastore.Query;
The problem is that the DatastoreIO doesn't accept these queries:
PCollection<Entity> projects = p.apply(Read.from(DatastoreIO.source().withQuery(q).withDataset(DATASET_ID)));
It will only take DatastoreV1.Query objects. Is there any way to use the app engine provided API's? I'm much more familiar with those calls. Better yet, if we could use Objectify, that would be awesome :)
Thanks!
This isn't possible with the current implementation of the API. We can look at adding as a feature, and would gladly accept a pull request to expand the current functionality. The AppEngine team is also actively working on increasing interoperability between their SDK and the Datastore API.
I am somewhat new to Web development - specifically Google App Engine and JavaScript/HTML development, but I have an app deployed and working on Google App Engine and it is working ok.
I would like a user of my App to be able to store and retrieve a serialization of the app state in JSON using the GAE Datastore. (Note - This is only a user-initiated action - so channels seem to be overkill)
The examples provided by Google demonstrates one approach that allows the server-side Python implementation to do this. Specifically https://developers.google.com/appengine/docs/python/gettingstartedpython27/usingdatastore. I have this working ok.
But this approach seems rather inelegant especially if as an "app" I want to store and retrieve serialized chunks of data somewhat asynchronously without reloading the page/app each time (again, this is only ever user-initiated).
I have not been able to find any high-level guidance on an approach to do that (assuming it is possible).
Any suggestions/links/examples would be greatly appreciated.
Thank you!
Jeff
As with many things, this depends on your specific needs. If you just want direct access to datastore storage, the datastore is exposed as an independent service with an API.
If you instead want to assert logic over the usage and interact with your app in some fashion, you may also want to look at Google Cloud Endpoints. With an endpoints API, you gain a more structured API you can call directly from javascript, or generate client libraries to be consumed by other languages/platforms.
I do not think there's a way to do this but I thought I'd ask to see what workarounds people are using to avoid polling on Heroku and/or GAE.
How are you sending out notifications in somewhat real-time from these apps?
GAE can use one of the following to notify another system:
xmpp
urlfetch to a callback url
like to pubsubhubbub
send email
With Heroku's new Cedar stack you can do long polling.
There's a link to some long polling example code here.
As #dar mentions, you can use PubSubHubbub to do realtime notification without polling. Nick Johnson has an excellent tutorial on using an existing PubSubHubbub server, or even including a server with your application.
This will let you publish feeds and subscribe to them in realtime without polling.