Related
I am looking for advice on the best way to go about modeling my database
Lets say I have three entities: meetings, habits, and tasks. Each has its own unique schema, however, I would like all 3 to have several things in common.
They should all contain calendar information, such as a start_date, end_date, recurrence_pattern, etc...
There are a few ways I could go about this:
Add these fields to each of the entities
Create an Event entity and have a foreign_key field on each of the other entities, pointing to the related Event
Create an Event entity and have 3 foreign_key fields on the Event (one for each of the other entities). At any given time only 1 of those fields would have a value and the other 2 would be null
Create an Event entity with 2 fields related_type and related_id. the related_type value, for any given row, would be one of "meetings", "habits", or "tasks" and the related_id would be the actual id of that entity type.
I will have separate api endpoints that access meetings, habits, and tasks.
I will need to return the event data along with them.
I will also have an endpoint to return all events.
I will need to return the related entity data along with each event.
Option 4 seems to be the most flexible but eliminates working with foreign keys.
Im not sure if that is a problem or a hinders performance.
I say its flexible in the case that I add a new entity, lets call it "games", the event schema will already be able to handle this.
When creating a new game, I would create a new event, and set the related_type to "games".
Im thinking the events endpoint can join on the related_type and would also require little to no updating.
Additionally, this seems better than option 3 in the case that I add many new entities that have event data.
For each of these entities a new column would be added to the event.
Options 1 and 2 could work fine, however I cannot just query for all events, I would have to query for each of the other entities.
Is there any best practices around this scenario? Any other approaches?
In the end performance is more important then flexibility. I would rather update code than sacrifice on performance.
I am using django and maybe someone has some tips around this, however, I am really looking for best practices around the database itself and not the api implementation.
I would keep it simple and choose option 1. Splitting up data in more tables than necessary for proper normalization won't be a benefit.
Perhaps you will like the idea of using PostgreSQL table inheritance. You could have an empty table event, and your three tables inherit from that table. That way, they automatically have all column from event, but they are still three independent tables.
I've various models corresponding to network devices based on STI, with only few fields varying between them, that I store in a jsonb field.
Another jsonb field (network_data) stocks an array of network informations in a json format. Here's the content of this field:
[{"ip"=>"192.168.1.1","ip_type"=>"1", "services"=>{"ns"=>"0",..., "FIREWALL"=>"1", "DNSresolver"=>"1"},
"network_device_destination_id"=>"9"},{"ip"=>"192.168.1.2","ip_type"=>"0","services"=>{"ns"=>"0","BGP"=>"0",
...(other services)},"network_device_destination_id"=>"8"}]
The main reason of using STI is the polymorphic assocation between network devices (the key 'network_device_destination_id' in each json record).
I use activerecord stores with the gem for the specific fields directly stored in json and not in a array https://github.com/devmynd/jsonb_accessor
Then, for this storized attributes I can easily use simple form as for a normal attributes.
But for the elements in the array, it's really tricky to process validation, to write forms...
So I was asking myself if it would be possible to make a class for a network record, with its own rules, validations, and accessors, and using a king of nested_attributes array as we would do with a has_many relation.
Something like we would do with this gem:
https://github.com/lailsonbm/awesome_nested_fields
Thank you for your precious advices!
PH
I'm currently trying to do the same (and there will be more of us I think as time passes) and the only real reference I have found is by Seth Faxon on his blog.
on his blog
He's doing a variation on has_many by creating a subclass in the parent model, where every row of the json/jsonb is then accessed as an individual associated object Parent::Child.
I'm not sure how well it would go as a substitute for polymorphic association, but I'll give it a go.
In my model i've defined relationship, so it's property with foreign key is substituded by related model.
I had an idea to give away from database two same values, for example relatedId and related - if i define models relationships for field related, relatedId value will be left untouched - and i will be able to use it.
Is it possible somehow to use collection.where() method in backbone relational on model attributes, that represent related models (they have object data type)? if i define related id - like following - it does not work:
collection.where({
related : 14 // this property contains related model, but not id after backbone initializes, i've also tried to use relatedId key instead - this does not work
})
I need such method very much, because i have to find models by lot of attributes, and it is very hard to do it from scratch :/
Could you please advice a way?
I really like using database "views" to effectively flatten relationships for way easier querying. It works well.
I've been looking into Datomic, and it looks really interesting. But while there seems to be very good information on how Datomic works technically, I have not seen much on how one should think about data modeling.
What are some best practices for data modeling in Datomic? Are there any good resources on the subject?
Caveat Lector
As Datomic is new and my experience with it is limited, this answer shouldn't be considered best practices in any way. Take this instead as an intro to Datomic for those with a relational background and a hankering for a more productive data store.
Getting Started
In Datomic, you model your domain data as Entities that possess Values for Attributes. Because a reference to another Entity can be the Value of an Attribute, you can model Relationships between Entities simply.
At first look, this isn't all that different from the way data is modeled in a traditional relational database. In SQL, table rows are Entities and a table's columns name Attributes that have Values. A Relationship is represented by a foreign key Value in one table row referencing the primary key Value of another table row.
This similarity is nice because you can just sketch out your traditional ER diagrams when modeling your domain. You can rely on relationships just like you would in a SQL database, but don't need to mess around with foreign keys since that's handled for you. Writes in Datomic are transactional and your reads are consistent. So you can separate your data into entities at whatever granularity feels right, relying on joins to provide the bigger picture. That's a convenience you lose with many NoSQL stores, where it's common to have BIG, denormalized entities to achieve some useful level of atomicity during updates.
At this point, you're off to a good start. But Datomic is much more flexible than a SQL database.
Taking Advantage
Time is inherently part of all Datomic data, so there is no need to specifically include the history of your data as part of your data model. This is probably the most talked about aspect of Datomic.
In Datomic, your schema is not rigidly defined in the "rectangular shape" required by SQL. That is, an entity1 can have whatever attributes it needs to satisfy your model. An entity need not have NULL or default values for attributes that don't apply to it. And you can add attributes to a particular, individual entity as you see fit.
So you can change the shape of individual entities over the course of time to be responsive to changes in your domain (or changes to your understanding of the domain). So what? This is not unlike Document Stores like MongoDB and CouchDB.
The difference is that with Datomic you can enact schema changes atomically over all affected entities. Meaning that you can issue a transaction to update the shape of all entities, based upon arbitrary domain logic, written in your language[2], that will execute without affecting readers until committed. I'm not aware of anything close to this sort of power in either the relational or document store spaces.
Your entities are not rigidly defined as "living in a single table" either. You decide what defines the "type" of an entity in Datomic. You could choose to be explicit and mandate that every entity in your model will have a :table attribute that connotes what "type" it is. Or your entities can conform to any number of "types" simply by satisfying the attribute requirements of each type.
For example, your model could mandate that:
A Person requires attributes :name, :ssn, :dob
An Employee requires :name, :title, :salary
A Resident requires :name, :address
A Member requires :id, :plan, :expiration
Which means an entity like me:
{:name "Brian" :ssn 123-45-6789 :dob 1976-09-15
:address "400 South State St, Chicago, IL 60605"
:id 42 :plan "Basic" :expiration 2012-05-01}
can be inferred to be a Person, a Resident and a Member but NOT an Employee.
Datomic queries are expressed in Datalog and can incorporate rules expressed in your own language, referencing data and resources that are not stored in Datomic. You can store Database Functions as first-class values inside of Datomic. These resemble Stored Procedures in SQL, but can be manipulated as values inside of a transaction and are also written in your language. Both of these features let you express queries and updates in a more domain-centric way.
Finally, the impedance mismatch between the OO and relational worlds has always frustrated me. Using a functional, data-centric language (Clojure) helps with that, but Datomic looks to provide a durable data store that doesn't require mental gymnastics to bridge from code to storage.
As an example, an entity fetched from Datomic looks and acts like a Clojure (or Java) map. It can be passed up to higher levels of an application without translation into an object instance or general data structure. Traversing that entity's relationships will fetch the related entities from Datomic lazily. But with the guarantee that they will be consistant with the original query, even in the face of concurrent updates. And those entities will appear to be plain old maps nested inside the first entity.
This makes data modeling more natural and much, much less of a fight in my opinion.
Potential Pitfalls
Conflicting attributes
The example above illustrates a potential pitfall in your model. What if you later decide that :id is also an attribute of an Employee? The solution is to organize your attributes into namespaces. So you would have both :member/id and :employee/id. Doing this ahead of time helps avoid conflict later on.
An attribute's definition can't be changed (yet)
Once you've defined an attribute in your Datomic as a particular type, indexed or not, unique, etc. you can't change that later. We're talking ALTER TABLE ALTER COLUMN in SQL parlance here. For now, you could create a replacement attribute with the right definition and move your existing data.
This may sound terrible, but it's not. Because transactions are serialized, you can submit one that creates the new attribute, copies your data to it, resolves conflicts and removes the old attribute. It will run without interference from other transactions and can take advantage of domain-specific logic in your native language to do it's thing. It's essentially what an RDBMS is doing behind the scenes when you issue an ALTER TABLE, but you name the rules.
Don't be "a kid in a candy store"
Flexible schema doesn't mean no data model. I'd advise some upfront planning to model things in a sane way, same as you would for any other data store. Leverage Datomic's flexibility down the road when you have to, not just because you can.
Avoid storing large, constantly changing data
Datomic isn't a good data store for BLOBs or very large data that's constantly changing. Because it keeps a historical record of previous values and there isn't a method to purge older versions (yet). This kind of thing is almost always a better fit for an object store like S3. Update: There is a way to disable history on a per-attribute basis. Update: There is now also a way to excise data; however, storing references to external objects rather than the objects themselves may still stil be the best approach to handling BLOBs. Compare this strategy with using byte arrays.
Resources
Datomic mailing list
IRC channel #datomic on Freenode
Notes
I mean entity in the row sense, not the table sense which is more properly described as entity-type.
My understanding is that Java and Clojure are currently supported, but it is possible that other JVM languages could be supported in the future.
A very nice answer from bkirkbri. I want to make some additions:
If you store many entities of similar, but not equal "types" or schemas, use a type keyword in the schema, like
[:db/add #db/id[:db.part/user] :db/ident :article.type/animal]
[:db/add #db/id[:db.part/user] :db/ident :article.type/weapon]
[:db/add #db/id[:db.part/user] :db/ident :article.type/candy]
{:db/id #db/id[:db.part/db]
:db/ident :article/type
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The type of article"
:db.install/_attribute :db.part/db}
When you read them, get the entity-ids from a query and use datomic.api/entity and the eid and parse them by multimethods dispatching on type if nescessary, since it's hard to make a good query for all attributes in some more complex schema.
I have an abstract class Contact.
It leads to two subclasses:
Company (Title)
Person (FirstName, LastName)
I want to add a computed 'Title' col in the Person table, that return FirstName + ' ' + LastName, which will give me better search options.
So I want to create the Contact table have an abstract property Title, which each of these two implements, and so, I will be able to use:
Dim contacts = From c In context.Contacts
Where c.Title.Contains("Nash")
I am pretty sure this is impossible, the question is what is the efficient alternative way?
In my scenario I have a ListBox showing all the Contacts of both Company and Person types, I have a search TextBox and I want the Service query (GetContacts(searchQuery As String)) to query the filtered set against the DB.
Update
After Will's answer, I decided to create in the Person table a computed col as above.
The question is what what be the most efficient way to imlpement the WCF-RIA query method:
Public Function GetContacts(searchQuery As String) As IQueryable(Of Contact)
'Do here whatever it takes to retieve from Contacts + People
'and mix the results of both tables ordered by Title
End Function
Unfortunately, while there is a way to do this with partial classes, I am 99% sure you cannot mix linq queries that touch entity properties and "POCO" properties defined in a partial class.
The Linq to Entity context will actually convert these queries to sql, and it cannot handle situations where a particular method isn't directly supported by the context. A common example for L2E is the inability to use enums in your query. Like knowing how to handle enums, the context certainly doesn't know how to handle your POCO properties when converting to raw sql.
An option you might want to investigate is to create a the computed column within your database, or to run your queries, do the traditional ToArray() in order to trigger enumeration, and then examine the computed column in memory. This might not be a good solution, depending on the size of your table, however.
So, essentially, you wish to search two disparate types (backed by two different tables) and then combine the results for display to the user.
I would have to say that polymorphism is NOT the best solution. The desire to show them in the UI shouldn't force a design decision all the way down into your type definitions.
I have done something similar a few times before in WPF. I've done it two ways; by using polymorphism in the form of facade types which wrap the models and which can be treated by a common base type, and by treating all the different types in the collection as System.Object.
The first way is okay when you need type safety and the ability to treat different types the same way. The wrappers extend a common base class and are coded to "know" how to handle each of their wrapped types correctly.
The second way is okay when you don't need type safety, such as when exposing a collection to a WPF View where you are displaying them in an ItemsControl, which can figure out the correct DataTemplate to use by the type of each instance in the collection.
I'm not sure which way is best for you, but whichever it is, you should query both your Company and Person tables separately, Union the two result sets, then sort them appropriately.
Pseudocode:
//Wrapper version
var results = Company
.Where(x=>x.Title.Contains(searchTerm))
.Select(x=> new CompanyWrapper(x))
Cast<BaseWrapper>().Union(
Person
.Where(x=>x.ComputedTitle.Contains(searchTerm))
.Select(x=> new PersonWrapper(x))
.Cast<BaseWrapper>());
//System.Object version
var results = Company
.Where(x=>x.Title.Contains(searchTerm))
Cast<object>().Union(
Person
.Where(x=>x.ComputedTitle.Contains(searchTerm))
.Cast<object>());
In both cases, you may not have to downcast specifically. Again, the first gives you type safety if you need it in the UI, the second is simpler and requires less code on the backend but is only useful if you don't require type safety in the UI.
As for sorting, once you've searched and combined your result, you can OrderBy to sort the result, however you will have to provide a function which can perform the ordering. This function will differ depending on which version you choose.