If you have a application that exceeds that Datastore quota of 1 Gig (Master/Slave configuration) How do you clear the condition?
I have an application called "parking-helper" and it has a Total Stored Data 100% used message.
But I've deleted 99% of the data (a few days ago) cleared the indexes,
vacuumed the indexes and cleared the memcache and also waited for at
least 2 reset cycles and yet the datastore still says 100% used.
The datastore admin shows that the datastore does not add up to more than a
meg.
I know that memcache and indexes count also. It would be nice to see the sizes of these in the Datastore Admin view (but currently we cannot see these values). But I believe I cleared them also.
How can I reset this application without the drastic measure of deleting and re-creating which means I'd have to pick another application id and re-create all the other data.
Thanks,
Ralph
It's usually just a matter of waiting. And it looks like your quota is under 100% now.
Related
After updating a record in the datastore, a query to that record returns sometimes the correct data and sometimes stale data. It takes up to 1 hour for the data to replicate.
Is that regular behavior of the datastore eventual consistency? I'd expect that changes in the datastore take up to 2 minutes to propagate, but not an hour.
I found out what the problem was. After all it wasn't the fault of the datastore, but rather the cache of objectify. Disabling it's cache fixed the issue.
I have launched a new web app this month. I'm trying to understand why I'm getting such a high datastore read value, even though all my entities are cached.
So, my main point of misunderstanding is this. in the total quota overview for this month I have 1.12 M read operations in the datastore.
But when I go to the memcache section in the console, it tells me the hit ratio is 96.35% and the numbers are: 1,457,499 hit / 55,177 miss
First of all, is it true that these numbers are per month or are they per day?
Second, how is this possible?
I know that reads in transactions don't user the cache. But I do not make heavy use of transactions. Is there anything other than transactions that can cause this?
If you want more insight into your Objectify memcache hit rates, mount the MemcacheStatsServlet (or look at its code and do something similar). This will provide your cache hit ratio broken down by Kind.
Keep in mind that since it is reporting for just one instance (whichever you happen to hit with your request for stats), this is only a representative sample of what is going on in your cluster.
We currently have our application hosted in Google App Engine. Billing is enabled to that application. This application is still in beta that we are using for testing purpose. We have a logic of serving data from the Memcache if present, if not then we get the data from the datastore and update the memcache and serve the data. We are encountering strange behaviour related to Memcache. The data related to some keys in Memcache is getting dropped after few minutes after being set. We tried setting expiration time for the keys in the memcache, even that does not seem to work. Since the data is getting dropped from the memcache the data is again from the datastore which is increasing the billing for our application.
Currently nearly 80% of the billing is related to datastore read. The datastore read is high as the memcache is not working efficiently as it should be. Any insight why we are facing this issue would be really helpful.
Just an FYI, we are having around 75000 keys in the memcache with total size of 100 MB data. Our structure demands keeping such large number of keys in memcache, which I think should not be an issue.
Our application is being by 10 users and the billing amount per day is coming to around $40.
Thanks,
Krish
Unfortunately memcache will evict keys as and when it requires. Setting the time to expire only means the item will be in memcache for up to the expiry time.
Take a look at the docs regarding eviction.
Also, take a look at this for some more insight into ways around memcache issues.
Regarding your data structure, perhaps you could post a new question and we can see if others have advice for you.
I ran a function that loads a lot of data to GAE using db.put(). However, it raised over quota exception when I hit my write quota. When I rechecked the data by running the app, the data returned was indeed incomplete. So when the quota is available again, I ran the data loader again from some index (so I don't write the same data again and again).
Here is the problem: after I ran the data loader manually (again and again), it seems all the data that I need for the app to work is already there, although the first time I load the data there was over quota exception.
So, my question specifically is: does function that ran over quota in GAE being queued until the quota is available again or does it being terminated?
Background of project: my friend and I are building a search system. We need the database of the search system, thus we load the database to GAE.
If you hit write quota while adding many values to the datastore, the remaining values will not be saved anywhere and you will have to try again. Datastore admin shows the number of entities based on datastore statistics, but this will have a delay in being updated. Though officially it is mentioned as upto 24 hours, it can be even more as mentioned in this previous post. So for finding if recently uploaded entities are present in the datastore, we cannot rely on datastore admin and need to query and find if a particular entity you added recently is present. Or else you can read the entity key value that is returned for each db.put() and use the last returned value to see which is the last successfully stored entity.
Memcache in general and on AppEngine in specific is unreliable in the sense that my data may be deleted from the cache for whatever reason at any point in time. However, in some cases there might be cases where a small risk may be worth the added performance using memcache could give, such as updating some data in memcache that gets saved periodically to some other, more reliable storage. Are there any numbers from Google that could give me an indication of the actual probability that a memcache entry would be lost from the cache before its expiration time, given that I keep within my quotas?
Are there any reasons other than hardware failure and administrative operations such as machines at the data centers being upgraded/moved/replaced that would cause entries to be removed from memcache prematurely?
Memcache, like any cache, should be used as... a cache. If you can't find something in the cache, there must be a strategy to find it in permanent storage.
In addition to the reasons you mention, Memcache and other caching approaches have limits to the amount of items they will hold (discarding usually the least recently used ones when the cache is full), and often also set other cache invalidation policies (e.g. flush everything unused for one hour).
If you don't configure and operate the cache yourself, you have NO guarantee of when and how items might be removed from the cache intentionally / by design.
Any concrete answer you get to this question is 100% subject to change.
That said, I've used memcache under light loads to accumulate data for 15 minutes or so before writing it all to the Datastore. This was for totally non-critical analytic data though. Do not depend on it.
It's not that data can be lost, but that if it is lost, it can be easily regained.
For example, using it to store data from the datastore is ideal, in that if a piece of data is not in the cache, it can be easily fetched.
If you're storing data such as a hit counter in the cache, it can't be regained if the cache is cleared, so you'll lose data.
If you're concerned about load for a common job, how about setting a job to update the counter later, using the task queue?
I have implemented a shared-memcache based stats counter that collects hourly to DB and can identify cache loss (log it). So far I see constantly <10% cache-losses total each day after at most 1h (average 30 mins) cache time with about 60 active counters. Counter losses appear to be random single counters. I suspect, that counters that are incremented only once (occurs quite often in my case) could have higher probability of being dropped.
My App uses <1MB total memcache in the shared memcache system. Unfortunately using dedicated memcache with 1GB minimum and substantial costs per year is out of the question. Stats counter used.
I have created a stackdriver counter that records memcache losses for a counter that is saved every full hour. The graph shows successful saves in red and memcache fails in blue. The counter saves every full hour and has a few counts in the hour.