I would like to implement datetime ordered entity in appengine, pretty much like Appengine's own logs. So I probably will need some kind of unique ordered id generation algorithm.
Has anyone got any suggestion on this?
Having a similar need I passed a long integer time stamp as identifier to the Entity constructor. The identifier can be only a string or a long integer according to Java Datastore Entities, Properties, and Keys. In order to see the actual dates and times in the Datastore Viewer I put the same value converted to a java.util.Date into an unindexed property as well. Admittedly some denormalized redundancy but convenient in practice.
Use the date that you are appending. One way is to convert it to unix time (ms since 1970) so its numeric.
A better way but more code is to not use the datastore and use bigquery instead. Probably cheaper.
We need more informations about what you want to do.
If you want to make some logs, you can use timestamp indeed.
With python and ndb it's easy :
class Log(ndb.Model):
date = ndb.DateTimeProperty(auto_now_add=True)
message = ndb.StringProperty()
Then you order your logs by the date field.
If you want to make like AppEngine, you can link your log with a parent key and order by date and parent key.
AppEngine Python ndb
I hope it helped you.
Related
When would using a ComputedProperty (ndb) in google app engine give you a distinct advantage over just computing when needed, on the backend (such as in a handler), without the datastore being involved?
Everything I'm reading seems to indicate that it's mostly useless, and would just slow queries down (at least the put operation if nothing else).
Thoughts?
I did see this:
"Note: Use ComputedProperty if the application queries for the computed value. If you just want to use the derived version in Python code, define a regular method or use Python's #property built-in."
but that doesn't really explain any advantage (why query if you can derive?)
The documentation is quite clear on that regard, and i'll cite it again for reference, the Computed Properties section:
Note: Use ComputedProperty if the application queries for the computed value. If you just want to use the derived version in Python code, define a regular method or use Python's #property built-in.
When to use it? When you need to query some derived data, it needs to be written to the datastore so it gets indexed.
First example that came to mind: You're already storing the birthday of a user, but also need to filter by actual age, adding a property to derive that value might be the easiest and most efficient solution:
age = ndb.ComputedProperty(lambda self: calc_age(self.birthday))
Of course you could just have a function that returns the age, but that's only useful after you get the entity, can't use it for queries.
In a appengine search document, can I set a field to not be indexed?
I looked at the documentation and I think this is not possible, but I don't find anywhere that concretely says so, so trying to make sure.
I wonder why this is... perhaps they are trying to prevent "abusive" use of the search by storing "too much" non-indexed information in the document... but it is pretty convenient in some cases vs having to go to another data store, especially when the total data sizes are fairly small.
Thanks
No, the full text search service is a search index, not a data repository, so by definition everything is indexed.
You would probably struggle with this approach because the data types you put in are not necessarily what are stored, for example dates put in only have precision of a day (the time component is dropped), and numbers have truncated precision.
I am developing a RESTfull webservice with GAE. My technology stack is focused around Jersey, Spring, and Objectify.
If you don't know Objectify is ...
“Objectify is a Java data access API specifically designed for the Google App Engine datastore. It occupies a "middle ground"; easier to use and more transparent than JDO or JPA, but significantly more convenient than the Low-Level API. Objectify is designed to make novices immediately productive yet also expose the full power of the GAE datastore.”
https://code.google.com/p/objectify-appengine/
As of now I have used Objectify Keys to store relationships in my models. Like this ...
public class MyModel {
#Id private Long id;
private Key<MyOtherModel>> myOtherModel;
...
Objectify keys provide additional power as compared to Long IDs, but they can be created from a Long ID and a MyOtherModel.class with a static method Key.create(...),
Key.create(MyOtherModel.class, id)
so I don't exactly have to store relationships as Objectify keys at the model level, I just thought it be more consistent.
The problem is I need to write a lot of additional code to create XML adapters to convert the Objectify keys to Long IDs when I serialize my model objects to JSON, and deserialize them from JSON to a Java object.
I was thinking about using Long IDs instead and creating an Objectify Key in the DAO, when I need it. Also, this would remove any Objectify specific code from anything that wasn't a DAO.
I would like some perspective from a more experienced programmer. I have never created a software of this size, several thousand lines of code that is.
Thanks a lot everyone.
I am an in-experienced datastore/objectify developer too, so I'm just musing here.
I see your point, that replacing the Key<> type in MyModel with a Long id would simplify things for you. I would note though, that the Key<> object can contain a path (as well as a kind and an id). So, if your data model becomes more complex and MyOtherModel is no longer a root kind then your ability to generate a Key<> from a Long id breaks down.
If you know that won't happen, or don't mind changing MyModel later then I guess that isn't a problem.
For your serializing format I would suggest you use a String to hold your key or id. Your Long id can be converted to a string, and would have to be anyway for JSON (so there is no loss in efficiency), but that same string could later be used to hold the full Key too.
You can also store them as long (or Long or String) and have a method of getMyOtherModelKey() and that can return a key after calling the static method. You can also have getMyOtherModelID() to just return the ID. This really works both ways since you can have both methods if you store a key or just the ID.
The trick comes in if you use parents in any of your models. If you do the ID alone is not enough to get the other model, you need the ID and the IDs of all the parents (and grand parents if needed). This is where Keys are nice.
I have an entity with 2 properties: UserId(String) and RSSSubscriptions(String). Instances of this class will be storing in App Engine Datastore.
Where RSSSubscriptions should be a key value pair like "Site1: Feed1", "Site2: Feed2".
Since datatypes like Hashmaps are not persistable I am forced to keep this data in a String format. Currently I have stored it as a string type with JSONArray format. Say, "[{"Site1: Feed1"}, {"Site2: Feed2"}]".
My client will be an Android app. So Iam supposed to parse this string as JSON Array at client side. But I think its a bad idea to create a String with JSON format and append it with existing string, each time when user is adding new subscription. Any better Ideas?
You can use JSONProperty which is supported by ndb for that particular reason. In my opinion its a "hairy" solution to store Json as string and parse it back and forth. You have to be very careful to guarantee validity.
Correct answer depends on several factors with expected number of pairs being the most important. Important to remember that there are significant costs associated with storing the pair in an entity accessed by query. There are numerous ops costs for doing a query, and there will be significant cpu time. Compare this to using a single record keyed by user id, and storing the JSON inside a TextProperty. That is one small op cost and cpu times which will likely be 10x less than a query.
Please consider these factors when deciding to go with the technically cleaner approach of querying entities. Myself, I would always use a serialized string inside a TextProperty for anything in the "thousands of pairs" volume unless there was a very high rate of deletions (and even this it likely the string approach could be better). Using a query is generally the last design choice for GAE given its high resource costs.
Using AppEngine datastore, but this might be agnostic, no idea.
Assume a database entity called Comment. Each Comment belongs to a User. Every Comment has a date property, pretty standard so far.
I want something that will let me: specify a User and get back a dictionary-ish (coming from a Python background, pardon. Hash table, map, however it should be called in this context) data structure where:
keys: every date appearing in the User's comment
values: Comments that were made on date.
I guess I could just iterate over a range of dates an build a map like this myself, but I seriously doubt I need to "invent" my own solution here.
Is there a way/tool/technique to do this?
Datastore supports both references and list properties. This let's you build one-to-many relationships in two ways:
Parent (User) has a list property containing keys of Child entities (Comment).
Child has a key property pointing to Parent.
Since you need to limit Comments by date, you'd best go with option two. Then you could query Comments which have date=somedate (or date range) and where user=someuserkey.
There is no native grouping functionality in Datastore, so to also "group" by date, you can add a sort on date to the query. Than when you iterate over the result, when the date changes you can use/store it as a grouping key.
Update
Designing no-sql databases should be access-oriented (versus datamodel oriented in sql): for often-used operations you should be getting data out as cheaply (= as few operations) as possible.
So, as a rule of thumb you should, in one operation, only get data that is needed at that moment (= shown on that page to user). I'm not sure about your app's design, but I doubt you need all user's full comments (with text and everything) at one time.
I'd start by saying you shouldn't apologize for having a Python background. App Engine started supporting only Python. Using the db module, you could have a User entity as the parent of several DailyCommentBatch entities each a parent of a couple Comment entities. IIRC, this will keep all related entities stored together (or close).
If you are using the NDB (I love it) you may have employ a StructuredProperty either at the User or DailyCommentBatch levels.