Memcache is one of those things where the solution could be absolutely anything, and no one ever really gives a decent answer, maybe because there is none. So I'm not looking for a direct answer, but maybe just something to get me going in the right direction.
For a typical request, here is my AppStats info:
So, out of a total 440 ms request, I spend 342 ms in memcache. And here I figured memcache was supposed to be lightning fast. I must be doing something wrong.
Looking at my memcache statistics in my admin console, I have this:
Hit count: 3848
Miss count: 21382
Hit ratio: 15%
I'm no expert on this stuff, but I'm pretty sure 15% is terrible.
The typical request above is a bit too detailed to explain, but basically, I create and put a new entity, which also updates and puts a parent entity, which also updates and puts any users associated with the parent entity.
Throughout all this, I always get by key, never query. Oh and I'm using NDB, so all the basic memcache stuff is handled automatically. So I never actually touch memcache manually on my own in my code.
Any ideas?
Edit: Here is the breakdown of my request
So I only have 2 datastore gets and 2 puts. The rest is automatically handled memcache stuff. Why is it doing so much work? Would I be better off handling this stuff manually?
Let's take a closer look at your data. Seven memcache writes took as much time as two datastore writes. This actually proves that memcache is, like, 3.5 times faster than Datastore.
If a typical request to your application requires updates of at least three database entities--followed by an update of more entities (the users associated), you can't make this operation "lightning fast." Memcache helps when you read entries much more frequently than you write them. If the amount of reads and writes to a User's record are on par, you should consider turning cache off for this model.
You can also try asynchronous operations and task queues. From your description, it looks like you try to first update the entity, and update its parent only after the update completes because it's natural. You may run these concurrently; this probably will require some refactoring, but it's worth it.
Second, updating "all the associated users" may be, perhaps. deferred to a task spawned in background; Task Queues have a very convenient interface for this. The "associated users" won't be updated immediately, but they probably don't need to! However, the latency of your request will be less then.
Related
I have launched a new web app this month. I'm trying to understand why I'm getting such a high datastore read value, even though all my entities are cached.
So, my main point of misunderstanding is this. in the total quota overview for this month I have 1.12 M read operations in the datastore.
But when I go to the memcache section in the console, it tells me the hit ratio is 96.35% and the numbers are: 1,457,499 hit / 55,177 miss
First of all, is it true that these numbers are per month or are they per day?
Second, how is this possible?
I know that reads in transactions don't user the cache. But I do not make heavy use of transactions. Is there anything other than transactions that can cause this?
If you want more insight into your Objectify memcache hit rates, mount the MemcacheStatsServlet (or look at its code and do something similar). This will provide your cache hit ratio broken down by Kind.
Keep in mind that since it is reporting for just one instance (whichever you happen to hit with your request for stats), this is only a representative sample of what is going on in your cluster.
According to the Google App Engine documentation, writing to one entity group is limited to one write per second when using High Replication Datastore. So...
What happens if I exceed this limit? Some kind of exception? And what should I do?
How do I know that I'm close to exceeding this limit? I can design the application in a way that particular actions (adding an entity...) are not likely to happen often but naturally I can't guarantee that.
Based on the GAE documentation and from my limited experience:
Expect something like 1 QPS rate and tune your app accordingly.
Sharding is a common pattern to handle datastore contention.
Always add defensive code to handle every possible exception (Application Error 5, The datastore operation timed out, Transaction collision for entity group, ..)
In case of error, retry the write moving the task in a proper taskqueue or, if you can, just alert the user to try again.
Retrying a write, usually works.
When possible, use a write-behind cache mechanism moving the writes operation that can lead to contention to Memcache and a Taskqueue slowing down the datastore hit rate.
A good way to avoid contention is to keep the entity groups small, but don't count on it too much.
You can have contention even on single entity.
one write per second is a little low. it actually supports more than that. i would even say between 5 and 10 writes per second but i can't guarantee that of course.
if you hit that limit you will get an exception yes. the exception message will be:
Too much contention on these datastore entities. please try again.
but i don't know/remember the exact Exception that will raise.
you have the choice to retry, continue or whatever else you think is right to to at that point.
you can't tell if you are close to a 1 write per second limit. it happens and you deal with it.
Making my way through the GAE documents.
I have a question I can't find an obvious answer to. Given that transaction to an entity group is limited to 1/sec, how can you scale a request where say, 10,000 users all want to access a particular user's page, at the same time?
Wouldn't this give you 10,000 reads on the particular user's entity group in 1/sec, thereby causing catastrophic system failure and unhappy users?
Or am I confused, and only writes get contentious.
AppEngine uses for transactions a optimistic concurrency control, meaning that they do not lock the data, but throw an exception when they detect that data is "dirty". So, first transaction to change data is ok, the second gets the exception and must retry.
Given this, I assume that reads do not block if they are not part of transaction, even if some other transaction is in progress.
Also, to make transactions less of a bottleneck, one should carefully organize entity groups and make them as small as possible and also have them organized in such a way that there is as few contention (parallel requests) as possible. Meaning:
Have small entity graphs - do not put a lot of entities under common parent.
Try having user entity as a root parent. Users usually do not create parallel transactions (e.g. make multiple money transfers at the same time, etc..)
Right. I wasn't thinking. The answer is memcache. At least partially. That, and an efficient data model/ schema.
I'm building an application for people attending events. I need to create a Ticket entity for a subset of Person entities, for a specific Event. The amount of people may exceed 50,000 entities.
Obviously I can't just do a for-loop where I iterate over a Person query and generate these Tickets.
How do I structure this on App Engine, is there a way to take advantage of MapReduce?
You may want to take a look at the Deferred library. You can spool up a bunch of task queues in parrallel to do the work that you want. You may want to look at the Mapper example class in the google docs which may help push you in the right direction.
You can iterate in a single for loop if you the Backend where a request can last for long. But such long running processes are not a good thing in my opinion. I guess proper use of task queues is more than enough.
I read about the Deferred library. Sometimes it behaves strangely and pickling your data can introduce some headaches. I could for the TaskQueue API
I do not suggest Deferred Library, although it's very easy to write code, the disadvantage is it will pickle your data, put it into one entity, and load and unpickle it later that cost a lot of overhead. Puts 30K entities costs me about 3 CPU Hours!
The cheapest way is just use the Task Queue that split the Person and enqueue with keys or other positional information. Insert same 30K entities used less than 1 CPU Hour.
In your question, fetch 1 million entities and run over is very fast depends on GAE's design, just do it. The slowest part is store the new Ticket entity.
BTW,
why not just Person.all().filter("something like attending events").
I have a very small amount of data (~200 bytes) that I retrieve from the database very often. The write rate is insignificant.
I would like to get away from all the unnecessary database calls to fetch this almost static data. Is memcached a good use for this? Something else?
If it's of any relevance I'm running this on GAE using python. The data in question could be easily (de)serialized as json.
Memcache is well-suited for this - reading from the datastore is much more expensive than reading from memcache. This is especially true for small amounts of data for which the cost to retrieve is dominated by latency to the datastore.
If your app receives enough requests that instances typically stay alive for a little while, then you could go one step further and use App Caching to largely avoid memcache too. (Basically, cache the value in a global variable, and also app-cache the time the value was last updated. Provide an accessor for the value which retrieves the latest from memcache/db if it hasn't been updated in X minutes). Memcache is pretty cheap though, so this extra work might only make sense if you access this variable rather frequently.
If it changes less often than once per day, you could just hardcode it in webapp code, and reupload the file each time it changes.