I'm writing a financial application form. Angular Schema Forms is perfect for the kind of flexibility I need to make the forms.
I'm having a bit of trouble getting my head around the concept of a schema, in particular: should a schema relate to a single form, or all the forms in an app?
I have many types of Financial Products that a user can apply for.... Checking Account, Savings Account, Car Insurance, House, Pet, Gadget Insurance. Many of the fields in the various forms are common to all: Name, Date of Birth, Address. Some are specific to one form or a couple of forms (Car Registration Number).
Should I have one all-encompassing schema which contains definitions for all of the possible fields in all of my forms, and add/override them in the Form definition object? Or should each form contain its own Schema? Or even each page in each form?
And if its either of the latter two - how do I prevent duplication? DRY!
The Schema vs the UI Schema
The way I think of it is that the Schema represents a view of your data model and the UI Schema represents an editable version of the view. Can you write a view that includes all of your fields in the database, yes, but I'm more likely to look at the specifics of what data I need.
In my projects I generate a JSON Schema for each product/service I provide customers and I used the json-refs library to allow me to pre-process json schema references ($ref) so that I can re-use definitions.
In my case I have an order schema which includes only order related data, customer address, order related dates, etc... that every order will need. I also have a product schema for each product. I then include the order and product into a combined schema with $ref. Each product can have multiple sections and attributes which I also re-use across products.
Now in your case it would seem like my solution would fit as our data is similar, however I do not know what your back end library is. That said the new alpha versions of Angular Schema Form support $ref using json-ref on the client side, but I would still look for a back end solution if your stack has a supporting library or you are using NodeJS and can use json-ref.
Related
I have a system whereby you can create documents. You select the document type to create and a form is displayed. Data is then added to the form, and the document can be generated. In Laravel things are done via Models. I am creating a new Model for each document but I don't think this is the best way. An example of my database :
So at the heart of it are projects. I create a new project; I can now create documents for this project. When I select project brief from a select box, a form is displayed whereby I can input :
Project roles
Project Data
Deliverables
Budget
It's three text fields and a standard input field. If I select reporting doc from the select menu, I have to input the data for this document (which is a couple of normal inputs, a couple of text fields, and a date). Although they are both documents, they expect different data (which is why I have created a Model for each document).
The problems: As seen in the diagram, I want to allow supporting documents to be uploaded alongside a document which is generated. I have a doc_upload table for this. So a document can have one or more doc_uploads.
Going back to the MVC structure, in my DocUpload model I can't say that DocUpload belongs to both ProjectBriefDoc and ProjectReportingDoc because it can only belong to one Model. So not only am I going to create a new model for every single document, I will have to create a new Upload model for each document as well. As more documents are added, I can see this becoming a nightmare to manage.
I am after a more generic Model which can handle different types of documents. My question relates to the different types of data I need to capture for each document, and how I can fit this into my design.
I have a design that can work, but I think it is a bad idea. I am looking for advice to improve this design, taking into account that each document requires different input, and each document will need to allow for file uploads.
You don't need to have a table/Model for each document type you'll create.
A more flexible approach would be to have a project_documents table, where you'll have a project_id and some data related to it, and then a doc_uploads related to the project_documents table.
This way a project can have as many documents your business will ever need and each document can have as many files as it needs.
You could try something like that:
If you still want to keep both tables, your doc_upload table in your example can have two foreign keys and two belongsTo() Laravel Model declarations without conflicts (it's not a marriage, it's an open relationship).
Or you could use Polymorphic Relations to do the same thing, but it's an anti-pattern of Database Design (because it'll not ensure data integrity on the database level).
For a good reference about Database Design, google for "Bill Karwin" and "SQL Antipatterns".
This guy has a very good Slideshare presentation and a book written about this topic - he used to be an active SO user as well.
ok.
I have a suggestion..you don't have to have such a tight coupling on the doc_upload references. You can treat this actually as a stand alone table in your model that is not pegged to a single entity.. You can still use the ORM to CRUD your way through and manage this table..
What I would do is keep the doc_upload table and use it for all up_load references for all documents no matter what table model the document resides in and have the following fields in the doc_upload table
documenttype (which can be the object name the target document object)
documentid_fk (this is now the generic key to a single row in the appropriate document type table(s)
So given a document in a given table.. (you can derive the documenttype based on the model object) and you know the id of the document itself because you just pulled it from the db context.. should be able to pull all related documents in the doc_upload table that match those two values.
You may be able to use reflection in your model to know what Entity (doc type ) you are in.. and the key is just the key.. so you should be able.
You will still have to create a new model Entity for each flavor of project document you wish to have.. but that may not be too difficult if the rate of change is small..
You should be able to write a minimum amount of code to e pull all related uploaded documents into your app..
You may use inheritance by zero-or-one relation in data model design.
IMO having an abstract entity(table) called project-document containing shared properties of all documents, will serve you.
project-brief and project-report and other types of documents will be children of project-document table, having a zero-or-one relation. primary key of project-document will be foreign key and primary key of the children.
Now having one-to-many relation between project-document and doc-upload will solve the problem.
I also suggest adding a unique constraint {project_id, doc_type} inside project-document for cardinal check (if necessary)
As other answers are sort of alluding to, you probably don't want to have a different Model for different documents, but rather a single Model for "document" with different views on it for your different processes. Laravel seems to have a good "templating" system for implementing views:
http://laravel.com/docs/5.1/blade
http://daylerees.com/codebright-blade/
Currently scoping out a new system. Like many systems, it will be required to store documents and link them to other kinds of item. In this instance a Document object can belong to a Job or it can belong to an Item (which in turn belongs to a Job).
We could do this by having a JobId and an ItemId against a Document and leaving one or the other blank if necessary, but that's going to mean annoying conditional logic in the handling code. So, two link tables seems a better idea.
However, it is likely that we will need to link Documents to other items in the system at some point in the future. There are Company and User objects, for example, and we might want to record Documents against those. There may be more.
That would entail a proliferation of link tables which, while effective, is messy and hard to follow.
This solution is in SQL Server and will be handled in code via Entity Framework.
Are there any design principles that can allow us to hook up Document objects with a variety of other system objects as required in a neater and more flexible way?
You could store two values: the id, and the type of object to which the document is attached. It doesn't allow the use of foreign keys, but is compatible with many application development frameworks.
If you have the partitioning option then you could dedicate different partitions to different object types.
You could also have multiple tables, one for job documents, one for item documents, and get an overview of all of them with a view that UNION ALL's them together. If you need uniqueness in that result set then you could use UUIDs for the primary key, or add an extra column to the view to express from which table the row was read.
I would like some database/programming suggestion on a specific issue.
I have 5 different people (that live in different parts of the world) that provide me with data. This data is given to me in many variety of ways, following a standard structure layout. However it's not always harmonized, the data might have extra things that are not in the standard, so I'd like the structure to be as dynamic as possible to accommodate what the person wants to use.
These 5 data sources are then placed inside a central database I host. So basically I have 5 data sources that are formatted following a standard structure, and they are uploaded to my local database.
I want to automate the upload of this data as much as possible for the person providing the data, so I want them to upload new sets of data that are automatically inserted in my local db.
My questions are:
How should I keep the structure dynamic without having to revisit my standard layout to accommodate new fields of data, or different structure?
How do I make them upload data in a way that is incremental? For example they might be uploading an XML version of their data, my upload code should figure out what already exists.
My final and most important question. Are there better ways of going about this instead of having an upload infrastructure?
How should I keep the structure dynamic without having to revisit my standard layout to accommodate new fields of data, or different structure?
Basically, you pivot the normal database idea of columns and rows.
You have a data name table, which consists of the unique names of the fields of data, and an indicator to tell the import process what type of data is stored, like a date, timestamp, or integer.
You have a data table, which contains the data name id, a sequence number, the data field, and a foreign key to identifying information.
The sequence number is used to differentiate between different values of the same data name.
The data field holds every type of data possible. This would be a VARCHAR(MAX) in most databases. It's up to the upload process to convert dates and numbers to strings.
Your data table will have a foreign key to the rest of the information that identifies who the data field belongs to.
How do I make them upload data in a way that is incremental? For example they might be uploading an XML version of their data, my upload code should figure out what already exists.
The short answer is that you can't.
Your upload process has to identify duplicate data and not store it on the database.
My final and most important question. Are there better ways of going about this instead of having an upload infrastructure?
This is a hard question to answer without knowing more about the type of data you're receiving, but there is software that allows you to load databases without a lot of programming, by defining the input data structure and mapping that structure to your database tables.
This is a very general question, but I think I have a general answer. What I think solves your problem is to construct a new relational calculus where the properties attached to the master record are not pre-determined. Here is an example involving a phone book application.
Common method using a non-relational table:
Table PERSON has columns Name,
HomePhone, OfficePhone.
All well and good, but what do you do if the occasional person shows up with a mobile phone, more than one mobile phone, a fax phone, etc.
Instead what you do is:
Table Person has columns Person_ID,
Name.
Table Phones has columns Person_ID,
Phone_Type, PhoneNumber.
There is a one-to-many relationship between Person and Phones, and there can be any number of them from zero to a zillion. The tables are JOINed by Person_ID. You have to have business and presentation logic that enumerates the Phone_Type column (or just let it be free-form, which is not as useful but easier).
You can do that for any property, and is what relational data bases are all about. I hope this helps.
As others have said, EAV tables can handle dynamic structure. (be aware of performance issues on large tables)
But is it in your interest to have your database fields dictated by the client? You can't write business logic to act upon those new fields because they don't exist yet, they could be anything.
Can you force the client to conform to your model? This allows you to know the fields ahead of time and have business logic act upon the fields. It allows you to write meaningful reports as well, rather than just pivoted data dumps.
It came up again today how to handle allowing users to add custom fields to the standard entity schemas delivered in your product's database. I favour actually providing a function that gives the user limited DDL functions, so they can actually add a new, custom field to a table. Another approach is to have a separate table for custom fields, e.g. Customers, and CustomersEx, where only CustomersEx can change, but here updates become trickier than normal. The last and most badass option that we discussed was providing a EAV table, where rows are entity name, field name, field value.
Which approach is best?
EAV that is added to the existing relational structure. There is a whitepaer published by SQL Server CAT (Customer Advisor Team) on this topic. Although is vendor specific, the principels discussed and the solutions proposed apply to most RDBMS: Best Practices for Semantic Data Modeling for Performance and Scalability
I favor your second idea. We do something similar but call the tables Attr[ibute] tables, e.g. Company, CompanyAttr. The attributes are a collection in our business object (1:M) and NHibernate handles database operations. We explicitly display these fields as attributes in the UI and don't try to display them as if they were additional fields in the table.
I favor adding fields to a table via DDL, but that table should be separate from the main table. That way, you can script changes to your database schema without affecting your users' custom field additions. A right-join is easy enough to accomplish, and you won't need the record in the separate table if there are no custom fields.
If you just want to display data in a vertical fashion, EAV tables can be a good choice. You can also run a pivot query to display them horizontally.
I asked about Choosing a method to store user profiles the other day and received an interesting response from David Thomas Garcia suggesting I use the Table Module design pattern. It looks like this is probably the direction I want to take. Everything I've turned up with Google seems to be fairly high level discussion, so if anyone could point me in the direction of some examples or give me a better idea of the nuts and bolts involved that would be awesome.
The best reference is "Patterns of Enterprise Application Architecture" by Martin Fowler:
Here's an excerpt from the section on Table Module:
A Table Module organizes domain
logic with one class per table in the
database, and a single instance of a
class contains the various procedures
that will act on the data. The
primary distinction with Domain
Model is that, if you have many
orders, a Domain Model will have one
order object per order while a Table
Module will have one object to handle
all orders.
Table Module would be particularly useful in the flexible database architecture you have described for your user profile data, basically the Entity-Attribute-Value design.
Typically, if you use Domain Model, each row in the underlying table becomes one object instance. Since you are storing user profile information in multiple rows, then you end up having to create many Domain Model objects, whereas what you really want is one object that encapsulates all the user properties.
Instead, the Table Module makes it easier for you to code logic that applies to multiple rows in the underlying database table. If you create a profile for a given user, you'd specify all those properties, and the Table Module class would have the code to translate that into a series of INSERT statements, one row per property.
$table->setUserProfile( $userid, array('firstname'=>'Kevin', 'lastname'=>'Loney') );
Likewise, querying a given user's profile would use the Table Module to map the multiple rows of the query result set to object members.
$hashArray = $table->getUserProfile( $userid );