I need to save in my model a list of objects from a certain class on the datastore.
Is there any simple way to archive this with ListProperty and custom property's without going into pickled/simplejson blob data?
I just want something like this:
class Test:
pass
class model(db.Model):
list = db.ListProperty(Test)
Looking at GAE documentation I can't really tell if this is impossible with the current version or not.
I was trying to avoid pickling because it's slow and has size limits.
You can only store a limited set of types directly in the datastore. To store your own types, you need to convert them into one of the accepted types in some manner - pickling is one common approach, as is serializing it as JSON.
The size limit isn't unique to pickling - 1MB is the largest Entity you can insert regardless of the fields and types.
You could save your Test objects in the datastore directly, by making a Test model/entity type. Otherwise you will have to serialize them somehow (using something like pickle or json)
You could have a list of keys
or you could give 'Test' entities a parent that is a entity of your 'model' class
Related
I'm new to GAE and I'm still trying to figure things out. We're developing an Android app which uses Cloud Datastore to store images, videos, text, audios, etc. So we have now over 15 types of content objects.
I've been modelling each type of object as a distinct ndb Model class, but I'm wondering if this kind of design could affect performance.
Specifically, wouldn't it be better to write a simple class (e.g ContentObject) which simply had a content_type, and a few generic fields as string, number and blob?
I guess I'd go for the latter if I had to worry about creating/maintaining tables (or simply knowing that there are regular db tables behind).
I really like the first option, but I had to ask, just in case.
There are no performance differences to worry about between the 2 approaches.
With dedicated models you'll have to write a bit more code - each model needs to be handled separately. But it's simpler code, especially if eventually you will have some properties which only exist for some entities or are handled differently, which would require conditional logic with a generic model.
Building queries is also simpler with dedicated models if there are property differences, using a single model may require filling in unused properties (maybe by using default values) if they are used for sorting/filtering query results (entities with missing properties aren't indexed by the respective properties so they won't show up in the results).
On the other hand you'll need separate queries for each model, you can't obtain results for different kinds in the same query. And you'll need to maintain separate composite indexes for each kind (with a total limit of 200 such indexes per application).
If you're worrying about code duplication, which could also be a reason for which you'd consider a shared model, it's also possible to combine the common properties in a single ndb model class, with a single/common implementation for handling those common properties, and inherit that class in dedicated subclasses handling the differences. Something like this:
class Content(ndb.Model):
type = ndb.StringProperty() # not really needed, cls._get_kind() can be used instead
blob = ndb.StringProperty()
# other generic/common content properties and related methods
class Video(Content):
has_cc = ndb.BooleanProperty()
# other video-specific content properties and related methods
But this is just an implementation approach, from the datastore perspective you're still using dedicated models - in the above example a video entity will have a Video kind, not a Content kind.
There are no tables with the datastore, the only thing shared between entities of the same kind is their ndb model (which is specific just for the more performant ndb client library, other client libraries don't have one) and the search indexes definitions.
I'm trying to figure out how to get nested structs to work with GAE datastore using Go. I know the datastore doesn't specifically support nested structs. I need to find a simple way of getting user information to go with a post when it is sent out to a user as JSON.
One thing I thought of was to put two fields for the user. One for the ID/key referencing to user and another one for the user type struct which would be added there when the post is loaded from the datastore. Extra fields seem silly so I'm hoping there is a better solution for this.
There are two entity types or structs: POST and USER
Posts need to contain information about the user who made the post.
The structure for the JSON I'm going to output for users is as follows:
POST
field1
field2
USER
user_field1
user_Field2
Go's appengine datastore api provides the PropertyLoadSaver interface for this sort of thing: https://developers.google.com/appengine/docs/go/datastore/reference#PropertyLoadSaver
You structure your struct however you want and then implement the Load and Save methods of that interface to populate it correctly. It means you write the serialization code yourself but it gives you full freedom in how you structure your data.
This will allow you still filter over the fields and have a nested struct.
The python runtime has the ndb library which supports nested structures like this. Go does not, so I can think of two solutions:
In the POST kind, have a user field that is a key, referencing a USER kind with the necessary fields. Requires two fetches and roundtrips.
Make a user field in the POST kind that is a blob. The blob is a string that is [de]serialized in go. This means you can't search or filter on any of the user data, but it also allows you to store everything in one entity.
You should use these based on the needs of your app. If you need users to be a real thing, use 1. If users aren't objects you need to work with (i.e., just data to display), you can use 2.
This is something I have been struggling about for some time now. The thing is: I have many (20 or so) static arrays of values. I say static because that is how I'm actually storing them, as static arrays inside some domains. For example, if I have a list of known websites, I do:
class Website {
...
static websites = ["web1", "web2" ...]
}
But I do this just while developing, because I can easily change the arrays if needed, but what I'm going to do when the application is ready for deployment? In my project it is very probable that, at some point, these arrays of values change. I've been researching on that matter, one can store application properties inside an external .properties file, but it will be impossible to store an array, even futile, because if some array gets an additional value, the application can't recognize it until the name of the new property is added where needed.
Another approach is to store this information in the database, but for some reason it seems like a waste to add 20 or more tables that will have just two rows, an id and a name.
And the last option, as far as I know, would be an XML, but I'm not very experienced with those. It seems groovy has a way of creating and reading XML files relatively easy, but I don't know how difficult would be to modify an XML whose layout is predefined in the application.
Needless to say that storing them in the config.groovy is not an option since any change will require to recompile.
I haven't come across some "standard" (maybe a best practice?) way of dealing with these.
So the questions is: Where to store these arrays?
Use Enum for a fixed set of attributes. Do this, if you rely at some places in your code on some concrete values.
If you do not rely on the attributes within your code (which you obviously don't), use a String-type. In this case, if you need to provide a select box, just do a distinct-query on this property in your database.
Default: Use a domain class per dynamic set of attributes. Another table is OK.
For something as simple as arrays you should use groovy own type of property files. They allow you too define properties as proper groovy variables (not just strings) and obviously loading them would be done dinamically in a simple way by using ConfigSlurper. For an example on how to use this kind of file you can have a look at the following ConfigSlurper:
For defining properties:
my.property.array=[1,2,3,4]
For loading property files:
def config = new ConfigSlurper().parse(new File('myconfig.groovy').toURL())
assert [1,2,3,4] == config.my.property.array
Tip: When you want to access the property files you want to do it in a way that can work for any environment. To make paths environment-independent use the following path as the root path:
import org.codehaus.groovy.grails.commons.ApplicationHolder
def ctx = ApplicationHolder.application.mainContext.servletContext
def rootPath = ctx.contextPath
If you use an object database, what happens when you need to change the structure of your object model?
For instance, I'm playing around with the Google App Engine. While I'm developing my app, I've realized that in some cases, I mis-named a class, and I want to change the name. And I have two classes that I think I need to consolidate.
However,I don't think I can, because the name of the class in intuitively tied into the datastore, and there is actual data stored under those class names.
I suppose the good thing about the "old way" of abstracting the object model from the data storage is that the data storage doesn't know anything about the object model --it's just data. So, you can change your object model and just load the data out of the datastore differently.
So, in general, when using a datastore which is intimate with your data model...how do you change things around?
If it's just class naming you're concerned about, you can change the class name without changing the kind (the identifier that is used in the datastore):
class Foo(db.Model):
#classmethod
def kind(cls):
return 'Bar'
If you want to rename your class, just implement the kind() method as above, and have it return the old kind name.
If you need to make changes to the actual representation of data in the datastore, you'll have to run a mapreduce to update the old data.
The same way you do it in relational databases, except without a nice simple SQL script: http://code.google.com/appengine/articles/update_schema.html
Also, just like the old days, objects without properties don't automatically get defaults and properties that don't exist in the schema still hang around as phantoms in the objects.
To rename a property, I expect you can remove the old property (the phantom hangs around) add the new name, populate the data with a copy from the old (phantom) property. The re-written object will only have the new property
You may be able to do it the way we are doing it in our project:
Before we update the object-model (schema), we export our data to a file or blob in json format using a custom export function and version tag on top. After the schema has been updated we import the json with another custom function which creates new entities and populates them with old data. Of course the import version needs to know the json format associated with each version number.
What is the best way to store lists (collections) of simple types in Google App Engine for Java?
I usually use something like that:
public class Person {
// ....
#Persistent(serialized = "true")
private List<Date> DatesList;
// ....
}
I am not sure how it is stored in DataStore, atomically or as a list of references? Hence wanted to get your advice on implementing these kinds of collections in a best way for performance of a query.
Like in the example above, when I load the object, will the DateList will be loaded with it. Or it will be loaded when accessed fro first time?
Does #defaultFetchGroup attribute affect this collection?
Thanks
The list will be returned/loaded by the datastore along with the rest of the object, not when it is first referenced.
GAE JDO Collections
If a field is declared as a List, objects returned by the datastore have an ArrayList value. If a field is declared as a Set, the datastore returns a HashSet. If a field is declared as a SortedSet, the datastore returns a TreeSet.