My application checks user authentication and authorization on every request and I am thinking of moving the authentication and authorization data from datastore to memcache.
Is memcache suitable for this usecase? Is it secure or I should not do it?
Yes, Memcache is secure. If you save important content to Memcache you have to back it up on datastore because it can be evicted at any moment. There are datastore APIs on Python (NDB) and Java (Objectify) that manage the cache for you automatically, but I'm not sure about Go.
If your users log in with their Google Account you can use the Users API1.
Yes, memcache is secure as the marked answer says (even shared memcache).
However as one comment says, you should really not reinvent appengine sessions, and instead use them. Google for "appengine go sessions" and you will find resources about it. The first link in that search:
https://godoc.org/code.google.com/p/sadbox/appengine/sessions
shows you how to use them in Go.
Besides handling the memcache and datastore implementation, it has additional features like configurable expiration.
Related
I am trying to figure out how to access Google App Engine Memcache service from outside Google App Engine. Any help on how this can be done would be greatly appreciated.
Thanks in advance!
I don't think this is currently possible. I don't know if there is any technical argument for this or if this decision has been made simply for billing purposes. But it seems like memcache is intended to be an integral part of App Engine. The only relevant discussion I could find is this feature request. It calls for possibility of accesing memcached data of one App Engine project by another App Engine project. It seems to me that Google didn't consider such functionality to be beneficial. You could try filing your own feature request to make memcache a standalone service. In case you do not succeed (and I am afraid you won't), here is a simple workaround.
A simple workaround:
Create a simple App Engine project which would serve as a facade over memcache service. This dummy App Engine project would simply translate your HTTP requests to memcache API calls and return the obtained data in the body of a HTTP response. For example, to retrieve a memcache record you could send a GET request such as:
https://<your-poject-id>.appspot.com/get?key=<some-particular-key>
This call would get "translated" into:
memcache.get(<some-particular-key>);
And the obtained data appended to the HTTP response.
Since accessing memcache is free, you would only have to pay for instance time. I don't know what through-put are you expecting, but I can imagine scenarios where you could even fit into the free daily quota (currently 28 hours/day). All in all, the intermediate App Engine project should not come with significant cost in neither performance nor price.
Before using this workaround:
The above snippet of code is intended for illustration purposes only. There still remain some issues to be dealt with before using this approach in production. For example, as pointed out by Suken, anyone would be able to access your memcache if they knew what requests to send. Here are four additional things I would personally do:
Address the security issues by sending some authentication token with each request. An obvious necessity would be to make the calls over HTTPS to prevent man-in-the-middle attackers from obtaining this token. Note that App Engine's appspot.com subdomains are accessible via HTTPS by default.
Prefer batch API calls such as getAll() over their single record alternatives such as get(). Retrieving multiple records in one batch call is much faster than making multiple separate API calls.
Use POST requests (instead of GET) to access the facade application. You won't have to worry about your batch requests being to large. I only used GET request in the example above because it was easier to write.
Check if such usage of App Engine doesn't violate the Terms of Service. Personally, I don't believe it does. And I don't see why Google should mind. After all, you will be paying for instance hours.
EDIT: After giving this some more thought, I believe that the suggested workaround is actually what Google presumes you to do. Given that the Goolge's objective is to earn money, it would be unreasonable to provide a free service unless it was a part of a paid one. Of course, another billing schemes could be created. For example, allowing direct access only for developers who are willing to pay for dedicated memcache. The question is whether your use case is broad enough to convince Google to take some action.
No, AFAIK the Memcache service is not available outside GAE. To be even more specific it is only available inside the GAE standard environment, it is unavailable in the GAE flexible environment.
But some of the alternate solutions suggested for GAE flexible users might be useable for you as well. From Memcache:
The Memcache service is currently not available for the App Engine
flexible environment. An alpha version of the memcache service will be
available shortly. If you would like to be notified when the service
is available, fill out this early access form.
If you need access to a memcache service immediately, you can use the
third party memcache service from Redis Labs. To access this service,
see Caching Application Data Using Redis Labs Memcache.
You can also use Redis Labs Redis Cloud, a third party fully-managed
service. To access this service, see Caching Application Data Using
Redis Labs Redis.
As stated by other users the Memcache is not offered as a service outside GAE (Google App Engine). I would like to point out that implementing GAE facade over Memcache service has security ramifications. Please note that facade GAE Memcache app will be exposed on the public internet like any other GAE service. I am assuming that you want to use Memcache for internal use only. Another aspect to think about is writing into memcache. If you intend to write to memcache from outside GAE, then definitely avoid facade implementation. If comprised anyone will be able to use you facade implementation as their own cache without paying for it ;)
My suggestion is to spin up a stack using GCP Cloud Launcher. There are various stack templates available for both Redis and Memcache stacks. Further you can configure the template to use preemptible burstable instances to reduce the cost of your Memcache.
I have a small question with possibly a complex answer. I have tried to research around, but I think I may not know the keywords.
I want to build a web service that will send a JSON response, which would be used for another application. My goal is having the App Engine server crawl a set of webpages and store the relevant values so the second application (client) would not need to query everything. It will only go to my server with the already condensed information.
I know, it's pretty common, but how can I defend from attackers who wish to exhaust my App Engine resources/quota?
I have been thinking on limiting the amount of requests by IP (say.. 200 requests / 5 minutes), but is that feasible? Or is there a better, and more clever way of doing it?
First, you need to cache the JSON. don't hit the datastore for every request. use memcache or possibly, depending on your requirements, you can cache the JSON in a static file in Cloud Storage. This simple is the best defender against DDOS, since every request adds minimal overhead.
Also, take a look in the DDOS protection service offered by app engine:
https://developers.google.com/appengine/docs/java/config/dos
You could require users to log-in then generate and send an auth key to the client app that must accompany any requests to the app engine service.
I would like to know a few things about how google handles the data withing google app engine:
Where is the data located - is it possible to find out?
How long will the data be stored? And what happens the data gets deleted?
Can you backup lost data or broken data?
What kind of security is added to protect the data?
You can find most of the answers here. More on ACID
Where? Google data centers across the globe
For how long? Forever is too much to say, so let's say for as long as you need the data to be there.
If the data gets deleted?
If you delete it yourself it'll be propagated within a sensible timeframe infrastructurally speaking
If there are accidental errors on Google's side, the data is replicated and backed.
To protect against accidental errors on your side you can use DataStore Admin Console to back up your data
Data is secured on Google's side. If you need to transfer sensitive information, App Engine enabled SSL for custom domains recently. If you don't want to go through the hassle of buying and registering certificates you can always use SSL through custom App Engine domain for free. More here
I am working on a project on Google AppEngine, where I am using the datastore and memcache. For testing purposes, I go to the control panel, change and save of some values of some saved entities in the datastore. However, since they're already in memcache, I assume, the changes don't reflect on the application.
Using Admin/control panel, is there a way to enforce reloading these entities into memcache from the datastore, or at least to clear memcache?
Thank you,
Hazem
With the release of SDK 1.6.4 today a memcache page has been added to the production admin dashboard. It has a button that lets you flush memcache (as well as other features).
That does not solve the problem of automatically managing the memcache version of your entities. For that you could look into the NDB library (new standard library in SDK 1.6.4) which automatically caches your entities as you use them and invalidates the cache as necessary. It has lots of other features as well. I have not used it yet but it certainly sounds good: http://code.google.com/appengine/docs/python/ndb/
UPDATED March 27 now that SDK 1.6.4 has gone final and I can confirm it contains the memcache control panel.
As an alternative: I believe you could use Remote API, fire up a shell and execute e.g. memcache.flush_all()
no there isnt.
make a handler that does that for you and when you need to flush memcache you can simply hit that url
I'm playing around with the Google App Engine and the Datastore.
Really amazing stuff going on over there.
But I couldn't help and wonder what Google is allowed to do with the data my application is storing in there Datastore.
Can someone explain it in simple worlds?
Thanks
Google App Engine is governed by Google's general Privacy Policy. They promise not to share information outside of Google except in certain circumstances (court order, etc.), and they restrict access to only employees who need it. However, because they can use your data to "provide, maintain, protect and improve our services," Google may be using your data for their own purposes (probably not, but I see nothing that prevents them).
Disclaimer: I am not a lawyer. Also, this policy doesn't provide GAE-specific information.
Worth noting that Google Cloud has separate Terms of Service that includes this line:
5.2 Use of Customer Data. Google will not access or use Customer Data, except as necessary to provide the Services to Customer.