I'm having a consistency problem on App Engine I'm not sure how to address. I've performed a query, selected an entity from the result, edited it and saved. All, using the low level Java API.
Performing an Entity Key query on the edited item, returns the updated version. However performing an indexed query by a certain property of this entity, returns the old version.
I've originally considered this as an occasionally expected "eventually consistent" HRD issue. However, I'm now hours after the original save, and this is still the case.
Ridiculously enough (and perhaps not so surprising), this occurs even when using the Datastore Viewer in the App Engine console: A GQL query returns a row with stale data, and selecting the entity opens the updated data.
What can I do? This sounds like an index bug. Is there a way to "refresh" the Index?
Related
When i was merging 2 gae instances, i took an export of the data in datastore from the first and imported it in the second.
All predefined indexes worked fine and i found the imported entities, but when i searched using auto built indexes it didn't return any values.
first time i did the import using Admin UI, then i tried using "gcloud datastore import", yet i got the same result.
reading imported entity by key and writing it again did the refresh for that entity indexes, but if do this for all imported entities it will cost a lot $$$.
Any advice how to refresh auto built indexes as it should be automatically refreshed as mentioned in documentation.
I have tried to replicate your scenario and the indexes are building automatically. If you are still observing that built-in indexes are not automatically built after an import, you might want to file a bug at Issue Tracker so this can be further investigated.
You will be charged the same amount for your built-in indexes, irrespective of the index rebuilds being triggered manually or automatically as they will have the same reads and writes and take up the same space.
I am writing a Google App Engine Go application and I'm having problem in testing some functionality. Here is some sample code. The problem is as follows:
I create a new item and save it to datastore
I make a search query for that item immediately after (for example, getting all items in the namespace)
The item is not there
If I query for the item later (say, in subsequent pages), the item is found normally. I understand this behaviour might be intentional for apps deployed on GAE, but I would expect the local datastore to be instantly good for queries.
Is there any way I can force the datastore to consolidate and be good for such queries?
This is called eventual consistency, and it's a feature of App Engine's Datastore.
You can use a get method instead of a query to test if an entity has been saved.
In Java we can set the desired behavior of a local datastore by changing a run parameter:
By default, the local datastore is configured to simulate the
consistency model of the High Replication Datastore, with the
percentage of datastore writes that are not immediately visible in
global queries set to 10%.
To adjust this level of consistency, set the
datastore.default_high_rep_job_policy_unapplied_job_pct system
property with a value corresponding to the amount of eventual
consistency you want your application to see.
I could not find something similar for Go.
I ran a function that loads a lot of data to GAE using db.put(). However, it raised over quota exception when I hit my write quota. When I rechecked the data by running the app, the data returned was indeed incomplete. So when the quota is available again, I ran the data loader again from some index (so I don't write the same data again and again).
Here is the problem: after I ran the data loader manually (again and again), it seems all the data that I need for the app to work is already there, although the first time I load the data there was over quota exception.
So, my question specifically is: does function that ran over quota in GAE being queued until the quota is available again or does it being terminated?
Background of project: my friend and I are building a search system. We need the database of the search system, thus we load the database to GAE.
If you hit write quota while adding many values to the datastore, the remaining values will not be saved anywhere and you will have to try again. Datastore admin shows the number of entities based on datastore statistics, but this will have a delay in being updated. Though officially it is mentioned as upto 24 hours, it can be even more as mentioned in this previous post. So for finding if recently uploaded entities are present in the datastore, we cannot rely on datastore admin and need to query and find if a particular entity you added recently is present. Or else you can read the entity key value that is returned for each db.put() and use the last returned value to see which is the last successfully stored entity.
I am having the following problem. I am now using the low-level
google datastore API rather than JDO, that way I should be in a
better position to see exactly what is happening in my code. I am
writing an entity to the datastore and shortly thereafter reading it
from the datastore using Jetty and eclipse. Sometimes the written
entity is not being read. This would be a real problem if it were to
happen in production code. I am using the 2.0 RC2 API.
I have tried this several times, sometimes the entity is retrieved
from the datastore and sometimes it is not. I am doing a simple
query on the datastore just after committing a write transaction.
(If I run the code through the debugger things run slow enough
that the entity has a chance of being read back on the second pass).
Any help with this issue would be greatly appreciated,
Regards,
The development server has the same consistency guarantees as the High Replication datastore on the live server. A "global" query uses an index that is only guaranteed to be eventually consistent with writes. To perform a query with strongly consistent guarantees, the query must be limited to an entity group, using an "ancestor" key.
A typical technique is to group data specific to a single user in a group, so the user can see changes to queries limited to the user's group with strong consistency guarantees. Another technique is to use fancier client logic to update the client's local view as soon as the change is submitted, so the user sees the change in the UI immediately while the update to the global index is in progress.
See the docs on queries and transactions.
I 've noticed this behavior which results in consecutive gets to succeed.
Has anybody else seen this?
I've found a way to remove one single entity from memcache, painful but it works.
Now, I use Java and Objectify but I hope you'll find this useful, whatever environment and language you use.
Go to the page https://console.cloud.google.com/appengine/memcache for your project.
Enter under Namespace the value "ObjectifyCache", or whatever namespace you use.
Under Key Type, select Java String
This is the tricky bit. Under Key you've got to enter the "URL-safe key" you'll find from the Datastore Edit page for your entity (https://console.cloud.google.com/datastore/entities/edit)
Click on Find and hopefully en entity will appear.
Check the box, and click on DELETE
Now you click on Find again, nothing will come up.
If you're using the high-replication datastore, gets immediately after deletes may succeed and pull up the stale results. It takes up to a few seconds for the results of each operation to appear in the results of other operations.
Memcache is operates independently of the datastore. Some libraries like Objectify connect them,. If you're using Objectify to cache entities and you delete something from outside of Objectify (e.g. the data viewer) you'll have to update your cache yourself. This happens to me occasionally and I just wipe the whole memcache.
You have to find a way to work with this behavior. The simplest (expensive and really slow) method, for example, would just be to wait ten seconds after you do every datastore operation. Better methods might use a cache to return freshly stored or deleted entities.