Currently, a lot of my code makes extensive use of ancestors to put and fetch objects. However, I'm looking to change some stuff around.
I initially thought that ancestors helped make querying faster if you knew who the ancestor of the entity you're looking for was. But I think it turns out that ancestors are mostly useful for transaction support. I don't make use of transactions, so I'm wondering if ancestors are more of a burden on the system here than a help.
What I have is a User entity, and a lot of other entities such as say Comments, Tags, Friends. A User can create many Comments, Tags, and Friends, and so whenever a user does so, I set the ancestor for all these newly created objects as the User.
So when I create a Comment, I set the ancestor as the user:
comment = Comment(aUser, key_name = commentId)
Now the only reason I'm doing this is strictly for querying purposes. I thought it would be faster when I wanted to get all comments by a certain user to just get all comments with a common ancestor rather than querying for all comments where authorEmail = userEmail.
So when I want to get all comments by a certain user, I do:
commentQuery = db.GqlQuery('SELECT * FROM Comment WHERE ANCESTOR IS :1', userKey)
So my question is, is this a good use of ancestors? Should each Comment instead have a ReferenceProperty that references the User object that created the comment, and filter by that?
(Also, my thinking was that using ancestors instead of an indexed ReferenceProperty would save on write costs. Am I mistaken here?)
You are right about the writing cost, an ancestor is part of the key which comes "free". using a reference property will increase your writing cost if the reference property is indexed.
Since you query on that reference property if will need to be indexed.
Ancestor is not only important for transactions, in the HRD (the default datastore implementation) if you don't create each comment with the same ancestor, the quires will not be strongly consistent.
-- Adding Nick's comment ---
Every entity with the same parent will be in the same entity group, and writes to entity groups are serialized, so using ancestors here will slow things down iff you're writing multiple entities concurrently. Since all the entities in a group are 'owned' by the user that forms the root of the group in your instance, though, this shouldn't be a problem - and in fact, what you're doing is actually a recommended design pattern.
Related
Lets say I have a number of (less than fifty) entities in my datamodel and I need to store who owns this entity for security reasons. I need to be able to decide on each request if the user doing a specific action on a resource is allowed to do that (who is doing what on which resource). And for this the resource ownership is needed.
I can think of several different ways to do this. One is that in each table I can have a foreing key pointing to the owner. One downside with this solution is that in code I need to look at each individual table to find out the ownership. Each time there is a new table added I would need to update the code to look in that new table.
Another solution could be to treat every specific entity as a generic resource, a resource that has an ownership. And store that ownership in one single table. I could even do that without any foreign key relationship and deal with it in code to keep the resource table in sync and e.g. make sure that each new entry in any table has a corresponding record in the "resource" table. One obvious downside would be that there will be a lot of records in this table. The benefit would be that there is one single place to go to to find the ownership.
So what would be the preferred way? Would there be a performance problem with storing the ownership in one table, given there might be in the range of hundreds of thousands of records (possibly even millions) in it eventually? What about the cost of preserving lots of foreign key constraints? Is there a better way of solving this?
Thanks
You're working in an object oriented language. Inheritance is perfect to solve this problem.
Depending on if you use Code First, or DB First, your approach will be slightly different, but boils down to this:
Make and abstract class, you can call it something like 'OwnableEntity'. In essence, you put your foreign key and navigation property in there.
Inherit all your entities from this 'OwnableEntity'
Make sure that your inheritance mapping in EF is correct (in this case, TPC inheritance mapping is what you will probably want to use)
From now on, you can write your 'check ownership' logic against 'OwnableEntity', and it will be OK for every entity you implement later on.
My app should contain several users, each of them having a list of objects ( only one user own the object ).
My question is : Would it be better to put an entity User that references the Ids of its objects, or should I put the user as the ancestor of the objects ? Please be kind, I am just beginning with nosql and datastore !
What approach you take will depend heavily on your access patterns, what make sense for easy retrieval, frequency of writes etc. You start your design process by building a basic entity relationship model, then start elaborating on what information you need to get to, and how frequently it is required what security restrictions are required. Then look at how you need to adjust the real model to reflect these access use cases taking into account performance, ease of use, security requirements.
Which approach you should choose depends mainly on the consistency model (strong vs eventual) you require for your entities. In Google Cloud Datastore, an entity group (an entity and its descendants) is a unit with strong consistency, transactionality, and locality.
You can read more on the topic here and here.
And there is one more important thing that is needed to take into account. If you model a parent-child relationship between a user and an object, the parent will be part of the object's key hence if you will change the object's owner later, you will end up with different object in terms of its key.
We are working on a mapping application that uses Google Maps API to display points on a map. All points are currently fetched from a MySQL database (holding some 5M + records). Currently all entities are stored in separate tables with attributes representing individual properties.
This presents following problems:
Every time there's a new property we have to make changes in the database, application code and the front-end. This is all fine but some properties have to be added for all entities so that's when it becomes a nightmare to go through 50+ different tables and add new properties.
There's no way to find all entities which share any given property e.g. no way to find all schools/colleges or universities that have a geography dept (without querying schools,uni's and colleges separately).
Removing a property is equally painful.
No standards for defining properties in individual tables. Same property can exist with different name or data type in another table.
No way to link or group points based on their properties (somehow related to point 2).
We are thinking to redesign the whole database but without DBA's help and lack of professional DB design experience we are really struggling.
Another problem we're facing with the new design is that there are lot of shared attributes/properties between entities.
For example:
An entity called "university" has 100+ attributes. Other entities (e.g. hospitals,banks,etc) share quite a few attributes with universities for example atm machines, parking, cafeteria etc etc.
We dont really want to have properties in separate table [and then linking them back to entities w/ foreign keys] as it will require us adding/removing manually. Also generalizing properties will results in groups containing 50+ attributes. Not all records (i.e. entities) require those properties.
So with keeping that in mind here's what we are thinking about the new design:
Have separate tables for each entity containing some basic info e.g. id,name,etc etc.
Have 2 tables attribute type and attribute to store properties information.
Link each entity (or a table if you like) to attribute using a many-to-many relation.
Store addresses in different table called addresses link entities via foreign keys.
We think this will allow us to be more flexible when adding, removing or querying on attributes.
This design, however, will result in increased number of joins when fetching data e.g.to display all "attributes" for a given university we might have a query with 20+ joins to fetch all related attributes in a single row.
We desperately need to know some opinions or possible flaws in this design approach.
Thanks for your time.
In trying to generalize your question without more specific examples, it's hard to truly critique your approach. If you'd like some more in depth analysis, try whipping up an ER diagram.
If your data model is changing so much that you're constantly adding/removing properties and many of these properties overlap, you might be better off using EAV.
Otherwise, if you want to maintain a relational approach but are finding a lot of overlap with properties, you can analyze the entities and look for abstractions that link to them.
Ex) My Db has Puppies, Kittens, and Walruses all with a hasFur and furColor attribute. Remove those attributes from the 3 tables and create a FurryAnimal table that links to each of those 3.
Of course, the simplest answer is to not touch the data model. Instead, create Views on the underlying tables that you can use to address (5), (4) and (2)
1 cannot be an issue. There is one place where your objects are defined. Everything else is generated/derived from that. Just refactor your code until this is the case.
2 is solved by having a metamodel, where you describe which properties are where. This is probably needed for 1 too.
You might want to totally avoid the problem by programming this in Smalltalk with Seaside on a Gemstone object oriented database. Then you can just have objects with collections and don't need so many joins.
I am trying to wrap my head around Entity Groups in Google AppEngine. I understand them in general, but since it sounds like you can not change the relationships once the object is created AND I have a big data migration to do, I want to try to get it right the first time.
I am making an Art site where members can sign up as regular a regular Member or as one of a handful of non-polymorphic Entity "types" (Artist, Venue, Organization, ArtistRepresentative, etc). Artists, for example can have Artwork, which can in turn have other Relationships (Gallery, Media, etc). All these things are connected via References and I understand that you don't need Entity Groups to merely do References. However, some of the References NEED to exist, which is why I am looking at Entity Groups.
From the docs:
"A good rule of thumb for entity groups is that they should be about the size of a single user's worth of data or smaller."
That said, I have a couple hopefully yes/no questions.
Question 0: I gather you don't need Entity Groups just to do transactions. However, since Entity Groups are stored in the same region of Big Table, this helps cut down on consistency issues and race conditions. Is this a fair look at Entity Groups and Transactions together?
Question 1: When a child Entity is saved, do any parent objects get implicitly accessed/saved? i.e. If I set up an Entity Group with path Member/Artist/Artwork, if I save an Artwork object, do the Member and Artist objects get updated/accessed? I would think not, but I am just making sure.
Question 2: If the answer to Question 1 is yes, does the accessing/updating only travel up the path and not affect other children. i.e. If I update Artwork, no other Artwork child of Member is updated.
Question 3: Assuming it is very important that the Member and its associated account type entity exist when a user signs up and that only the user will be updating its Member and associated account type Entity, does it make sense to put these in Entity Groups together?
i.e. Member/Artist, Member/Organization, Member/Venue.
Similarly, assuming only the user will be able to update the Artwork entities, does it make sense to include those as well? Note: Media/Gallery/etc which are references to Artwork may be related to lots of Artwork, not just those owned by the user (i.e. many to many relations).
It makes sense to have all the user's bits in an entity group if it works the way I suspect (i.e. Q1/Q2 are "no"), since they will all be in the same region of BigTable. However, adding the Artwork to the entity group seems like it might violate the "keep it small" principal and honestly, may not need to be in Transactions aside from saving bandwidth/retrys when users are uploading artwork images.
Any thoughts? Am I approaching Entity Groups wrong?
0: You do need entity groups for transactions among multiple entities
1: Modifying/accessing children does not modify/access a parent
2: N/A
3: Sounds reasonable. My feeling is, entity groups should not be used unless you need transactions among them.
It is not necessary to have the the Artwork as a child for permission purposes. But if you need transactional modification to them (including e.g. creation and deletion) it might be better. For example: if you delete an account, you delete the user entity but before you delete the child, you get DeadlineExceeded or the server crashes. Now you have an orphaned Artwork. If you have more than 1,000 Artworks for an Artist, you must delete in batches.
Good luck!
Okay, I'll be straight with you guys: I'm not sure exactly how Domain Driven my Design is, but I did start by building Model objects and ignoring the persistence layer altogether. Now I'm having difficulty deciding the best way to build my tables in SQL Server to match the models.
I'm building a web application in ASP.NET MVC, although I don't think the platform matters that much. I have the following object model hierarchy:
Property - has properties such as Address and Postcode
which have one or more
Case - inherits from PropertyObject
Quote - inherits from PropertyObject
which have one or more
Message - simple class that has properties Reference, Text and SentDate
Case and Quote have a lot of similar properties, so I also have a PropertyObject abstract base class that they inherit from. So Property has an Items property of type List which can contain both Case and Quote objects.
So essentially, I can have a Property that has a few Quotes and Cases and a load of Messages that can belong to either of those.
A PropertyObject has a Reference property (and therefore so do Quote and Case) so any Message object can be related back to a Quote OR Case by it's Reference property.
I'm thinking of using the Entity Framework to get my Models in and out of the database.
My initial thoughts were to have four tables: Property, Case, Quote and Message.
They'd all have their own sequential IDs, and the Case and Quote would be related back to Property by a PropertyID field.
The only way I can think of to relate a Message table back to the Case and Quote tables is to have both a RelationID and RelationType field, but there's no obvious way to tell SQL server how that relationship works, so I won't have any referential integrity.
Any ideas, suggestions, help?
Thanks,
Anthony
I am assuming Property doesn't also inherit from PropertyObject.
Given that these tables, Property, Case, Quote and Message, leads to a Table per Concrete Class or TPC inheritance strategy, which I generally don't recommend.
My recommendation is that you use either:
Table per Hierarchy or TPH - Case and Quote are stored in the same table with one column used as a discriminator, with nullable columns for properties that are not shared.
Table per Type or TPT - add a PropertyObject table with the shared fields and Case and Quote tables with just the extra fields for those types
Both of these strategies will allow you to maintain referential integrity and are supported by most ORMs.
see this for more: Tip 12 - How to choose an inheritance strategy
Hope this helps
Alex
Ahhh... Abstraction.
The trick with DDD is to recognize that abstraction is not always your friend. In some cases, too much abstraction leads to a too-complex relational model.
You don't always need inheritance. Indeed, the major purpose of inheritance is to reuse code. Reusing a structure can be important, but less so.
You have a prominent is-a pair of relationships: Case IS-A Property and Quote IS-A Property.
You have several ways to implement class hierarchies and "is-a" relationships.
As you've suggested with type discriminators to show which subclass this really is. This works when you often have to produce a union of the various subclasses. If you need all properties -- a union of CaseProperty and QuoteProperty, then this can work out.
You do not have to rely on inheritance; you can have disjoint tables for each set of relationships. CaseProperty and QuoteProperty. You'd have CaseMessage and QuoteMessage also, to follow the distinction forward.
You can have common features in a common table, and separate features in a separate table, and do a join to reconstruct a single object. So you might have a Property table with common features of all properties, plus CaseProperty and QuoteProperty with unique features of each subclass of Property. This is similar to what you're proposing with Case and Quote having foreign keys to Property.
You can flatten a polymorphic class hierarchy into a single table and use a type discriminator and NULL's. A master Property table has type discriminator for Case and Quote. Attributes of Case are nulled for rows that are supposed to be a Quote. Similarly, attributes of Quote are nulled for rows that are supposed to be a Case.
Your question "[how] to relate a Message table back to the Case and Quote tables" stems from a polymorphic set of subclases. In this case, the best solution might be this.
Message has an FK reference to Property.
Property has a type discriminator to separate Quote from Case. The Quote and Case class definitions both map to Property, but rely on a type discriminator, and (usually) different sets of columns.
The point is that the responsibility for Property, CaseProperty and QuoteProperty belongs to that class hierarchy, and not Message.
This is where the DDD concept of Services would come in. The Repository for each of your concrete classes only persist that entity, not the related objects.
So you have Property(), and is the base for your CaseProperty() : Property(). This special-entity is accessed via CasePropertyService(). Within here is where you would do your JOINs and such to the related tables in order to generate your CaseProperty() special entity (which is not really Case() and Property on its own, but a combination).
OT: Due to limitation of .net of where you can't inherit multiple classes, this is my work around. DDD is meant to be a guideline to the overall understanding of your domain. I often give my DDD outline to friends, and have them try to figure out what it does/represent. If it looks clean and they figure it out, it's clean. If your friends look at it and say, "I have no idea what you are trying to persist here." then go back to the drawing board.
But, there's a catch about using any ORM to persist storage of DDD objects (linq, EntityFramework, etc). Have a look at my answer over here:
Stackoverflow: Question about Repositories and their Save methods for domain objects
The catch is all objects must have an identity in the database for ORM. So, this helps you plan your DB structure.
I have recently moved away from using ORM to control direct access, and just have a clean DDD layer. I let my repositories and services control access to the DB layer, and use Velocity to entity-cache my objects. This actually works very well for: 1) DB performance, you design however is most efficient not being coupled to your DOmain objects with direct ORM representation, and 2) your domain model becomes much cleaner with no forced identies on Value Objects and such. Free!