I'm a newbie and when messing around with creating database applications I always just created my forms and put all the code and bindings in there. Instead of having arrays and lists that held information I made changes to the database directly.
Now that I have evolved a little bit let's say that I sold widgets to customers and kept the sales information in a database. If I were writing a program to access the database wouldn't I want to create a class of type 'Customer' and 'Widget' to work with those entities?
If I'm mistaken then what is the appropriate approach to programming database applications?
Yes.
You want to look into n-tier programming.
Basically, you allow you front end (presentation layer) access only to your class library (business layer). Your class library then accesses you database.
This gives you a less tightly coupled solution, and allows for more maintainable code. Also by introducing tiers you allow for changes to your DB without the need to rewrite code in your frontend, as long as the interface with the Business Layer doesn't need to be changed.
As far as binding is concerned, if you are using Visual Studio Windows Forms (2005 onwards) you should be able to us an bindingSource that you can then use to bind your controls. If you are using ASP.NET then your control should bind to a list of objects without any problems.
For ASP.Net the ObjectDataSource might be worth looking into. I haven't used it myself, but there's lots of samples on the web. Try here or here.
Yes.
You want to look closely at Object-Relational Mapping.
Your real-world business entities are modeled by objects which map to relational tables.
You do not want your presentation layer to directly depend on your database structure; the problem with that is if your database structure changes at all, your presentation layer must change, and in the long term, that tends to cause problems. Moreover, there are security issues involved with having your presentation layer directly interact with your database.
The rough analogy here is to markets; when you go to the store to buy a loaf of bread, you don't need to know how to grow wheat; all you need to know is that you have money, and they have bread, and that they will exchange a certain amount of bread for a certain amount of money. You don't need to know what time of year to plant the wheat, or how to remove the chaff, or any of that, because the backing layer takes care of that for you. Similarly, the farmer doesn't need to know how to sell bread to a whole bunch of people, or even how to make bread; all he has to do is know how to grow wheat.
Modern design philosophy recommends that you use an intermediate layer to interact between your presentation layer and your database layer; this is where you put your business logic. So, for example, let's say that you're selling widgets on your site. Instead of having your presentation code query the database for widgets and display that, you have a Business Object which handles your widgets. This way, your business object needs to know what your database structure is, but your presentation layer only needs to know how to ask your business object for a list of widgets to display. More importantly, in your business object you can place the rules that are to be invoked when certain things happen. So instead of your presentation layer directly making changes to the database for inventory and orders when an order is made, your business object knows how to make the changes and what tables to modify when your presentation layer requests that a sale occurs.
In this way, you separate the display of your information from the persistence and the logic underlying the website. What is involved is good planning; specifically, you have to figure out what your web site will be doing at any given point, and what that means in terms of what interfaces your business objects will provide. Then you implement your business objects based off of those requirements; those business objects are where you put the knowledge of the database structure and your specific business logic ("when A happens, do B and then C", etc.).
This seems like a lot of extra work at the beginning, but it really is worth it.
Related
Since the native-cloud applications or microservices architecture requires decentralized data model (each microservices has its own database), and universal data model is centralized data model
So, how we have microservices architecture with universal data model patterns?
Is there any reference or implementation of universal data model and microservices?
In general the two concepts are not compatible. Using a universal data model for all of your services would clash with a couple of key ideas behind using Microservices, e.g. Polyglot Persistence, separate development & deployment of each service. Also, let's not forget that the "Data Model Resource Book" was last updated in 2009.
However, if you must combine the two approaches, e.g. because management insists on it, you can encapsulate all access to the universal data model by a dedicated service and make your other services dependent on it.
Some good thoughts on the subject can be found here: http://plainoldobjects.com/2015/09/02/does-each-microservice-really-need-its-own-database-2/
Yes to #Fritz's point -- universal data modeling and microservices are really two different concepts and are very difficult if not impossible to be used together. I would like to add that the reasoning for polyglot persistence is also because of how the data should be modeled. Microservices allow the use of different data stores that can best model the data according to their domain.
To elaborate more, I don't think it would do justice to mention microservices and data modeling but not domain driven design. From my experience, domain driven design really helps in thinking about services, their responsibilities, and their right to exist. For instance, I found it often to be the case that there are usually a collection of services that carries out a particular domain functionality. An example could be an e-commerce application that have payments, shopping carts, etc. These could be separated into different "bounded contexts" based on domain driven design terminology.
With the different bounded contexts, each microservice no longer sees the same concept in the system the same, so in effect, there is no real universal data model. The easiest example that I can think of to show this, is when you also want reporting on the metrics in the system. If the example was an ecommerce application, the notion of a transaction in the orders microservice are going to be different than transactions in a reporting service. The reporting service for instance may want to know about transactions at a sub-level such as the profit or revenue generated for a particular order instead of the particular line items in an order. However, in the perspective of the orders service, the order details such as the line items and the address of the individual that made the purchase are probably important and should be known. This should then require two different data models.
With respect to domain modeling, I may be a bit extreme but I would go as far as saying that if there are multiple services sharing the same data source, they should really be the same service; there should be only one service for a single data source. My arguments for that would be that the domain hasn't been properly modeled and that the coupling makes it different to evolve any one service if there are multiple services that relies on a single data source. The case could be that one service requires the schema of the data source to change while the other one does not but still is required to accommodate the schema change. Hope this helps!
If my question isn't clear, please help me improve it with comments.
I'm new to Delphi and dbExpress and I'm just getting acquaintance with TSQLDataset, TDataSetProvider, TClientDataSet and TDataSource classes.
The project that I'm working on uses this components in a way that feels strange to me. There is a huge data module unit that contains lots and lots of the quartet of classes previously described. I'm guessing that there are better (and more modularized) ways of doing this. DataSnap is used only to place this data module in a server application, so the clients access the data through it.
So, let me try to explain some of my doubts:
What is the role of each one of this classes? I read the documentation but I can't get a practical insight on this subject (specially about TDataSetProvider).
Which classes should be in the data module and which should be in my forms?
Is it possible to create an intermediate layer to abstract my business model from my database setup (maybe creating functions that return immutable datasets?)?
If so, is it wise to use DataSnap to do so?
I'm sorry if I'm not clear enough. Thanks in advance.
Components
TDataSource is the bridge between data-aware controls and the dataset (TDataSet descendant) from which they are to get their values.
TClientDataSet is one such dataset. TClientDataSet can be used in isolation, for example to access data contained in xml files, but can also be hooked up to a TDataSetProvider.
TDataSetProvider is the bridge between the in-memory TClientDataSet and the actual dataset that takes its data from a database through some kind of driver. In Client Server development you will usually see a TRemoteDataSetProvider (name may be different, I don't work with these components that often), which bridges the gap between the client and server.
TSQLDataSet is the actual dataset getting its data from some database.
It would feel strange to me to see this quartet all in one executable. I would expect the TSQLDataSet on the server side hooked up to the TRemoteDataSetProvider's counter part. However, I guess that with an embedded database it could be a way to support the briefcase model, which is where TClientDataSet is really helpful (TClientDataset is very powerful, that is just one of its strong points.)
Single datamodule
Ouch. A single huge datamodule is lazy programming or the result of misconceptions about how to use datamodules. It is perfectly normal to have a single datamodule that "hosts" the database connection which is then used by various other datamodules that are more tightly focused on aspects of the application.
Domain abstraction
With regard to abstracting your business model, dbexpress and datasnap should really not be anywhere in your business model. They should be part of your data layer.
TDataSource, TClientDataSet and custom TDataSetProvider descendant(s) can be used to leverage the power of the data-aware controls in the UI while still keeping the UI separate from the business model. In this case the custom TDataSetProvider would be the bridge between the client dataset and the collections and instances in the domain layer.
Even so, I would still expect to see a separate data layer, using TRemoteDataSetProviders or straight TDataSet descendants (like TSQLDataSet) to provide the domain layer with its data.
The single huge data module you mentioned could be part of that data layer, with the client datasets providing the business layer with its data. As you also mention TDataSource as part of a common quartet, the application was probably developed in a RAD-data-aware fashion where UI controls are basically hooked up straight to database columns/tables.
If you want to transform this application to have a more layered architecture, tread carefully and slowly. Get to know the current architecture first and get to know it well enough to see the impact that this kind of transformation would have. The links provided by Serg will certainly help you there. Pawel Glowacki has written extensively about DataSnap.
The project you are working on is a Delphi Multitier Database Application
Your question is too wide for SO and hardly can be answered in its current form - you should learn to understand the underlying architecture.
You can start from video tutorial by Pawel Glowacki:
http://www.youtube.com/watch?v=B4uxLLIUddg
http://cc.embarcadero.com/item/28188
I created for a project a single class, that contains all access code to the database.
Is this a good practice , under the assumption that this class doesn't contain any logic, or should i use several classes? If yes, how should i partition my code? I use C# .Net.
Actually Under the concept of MVC framework, it is a good practice to create a different class for database access, seperate class for logic and seperate class for your views.
You are doing good if you are writing a seperate class for database access under the assumption that it does not contain any logic.
In Agile Developement there is a term named as Database Encapsulation Layers.
A database encapsulation layer hides the implementation details of your database, including their physical schemas, from your business code. In effect this layer provides your business objects with persistence services – the ability to read data from, write data to, and delete data from – data sources. Ideally your business objects should know nothing about how they are persisted, it just happens. Database encapsulation layers aren’t magic and they aren’t academic theories; database encapsulation layers are commonly used practice by both large and small applications as well as in both simple and complex applications. Database encapsulation layers are an important technique that every agile software developer should be aware of and be prepared to use.
An effective database encapsulation layer will provide several benefits:
-> It reduces the coupling between your object schema and your data schema, increasing your ability to evolve either one.
-> It implements all database-related code in one place.
-> It simplifies the job of application programmers.
-> It allows application programmers to focus on the business problem and Agile DBA(s) can focus on the database.
-> It gives you a common place to implement data-oriented business rules and logic.
-> It takes advantage of specific database features, increasing application performance.
Hope this helps.
If your database is quite small, say, only a couple of tables, you could write all your queries in one class. otherwise I would suggest that per Entity/Table one class. for example, StudentDao.class will only focus on the queries to database table "STUDENT", and TeacherDao.class will only contain queries to table "TEACHER". if you are gonna implement a complex business logic, you may want to have a service class, to weave StudentDao and TeacherDao together.
Unless your data access is very simple, probably not.
you probably shouldn't need to write this code yourself. Take a look at some Object Relational Mapping tools. NHibernate is a popular .Net solution. http://en.wikipedia.org/wiki/NHibernate
If you really do want to write it yourself look up design patterns in this area, like the Data Transfer Object pattern. http://martinfowler.com/eaaCatalog/dataTransferObject.html
These are some of the suggestions while accessing database.
1.) Always keep your database access parameters in a properties file. Use a handler to get those data. Because when you change your database then you need not change your code just make a change in the properties file it's enough.
-- So here you need a handler class.
2.) Never create a single class (a god class) which performs all the actions. Disperse your behaviour in to different classes depending on the intent. For example Keep all read behavior in one class, Write behavior in another class ... so on.
3.) You can create a class which deals with connection creations and pooling stuff...
Hope this helps.
People seem to avoid building user interfaces that pull their information (names, field types, etc. as well as relationships) from a database; they instead hard-code forms (and tables, etc.) that have pretty much the same data names and types and things.
Am I making sense?
For instance, imagine an emumerated field in MySQL: why not just have the UI construct a drop-down list whenever it encounters an ENUM? Why put the same values in both the database and the code?
Perhaps I'm just missing something; perhaps there are projects out there that do this — sort of super-crud interfaces that can be pointed at any database and from it build a fully-functional relationally-aware user interface. Are there?
I'm possibly not quite conforming to the stackoverflow norms with this question; I shall summarise:
Can you please tell me of a project that constructs its user interface (solely) from analysis of the database schema?
Why is this not a common way to do it — surely it is good to only define data structure in one place (i.e. the database)?
Thank you, and may joyous code-love rain upon your IDE.
I'd like to point out that, last time I checked, .NET and Qt (and probably other environments) make it possible to use "database-aware widgets" (sometimes shortened to just data-aware widgets), which is probably the best pragmatic solution available. What I mean by data-aware widgets is that the widgets themselves know that they're linked directly to database fields, so you would have a combobox that knows that it's backed by an enum and fetches the possible values directly from the database at runtime, just like you suggested.
This is a really neat utility, and used well, it probably won't hurt anything. It still requires that you spend some time laying out widgets manually on a form, but then if you update the database to add a new value to that enum, you don't have to rebuild your app to see it show up in the UI.
But the reason most usability experts will cringe when they hear your question is because programmers tend to think that, well, why not just generate the entire UI, form layout and everything, from the database? And this is where it starts to get really nasty, really fast.
Let's say you have a simple Person table, with first_name, last_name, email_address, street_address, city, state, zip, and phone_number. You want to automatically generate a UI based on these fields. How do you sort the fields? I mean, ideally, first name and last name should be right next to one another. And it would look very silly if you had city and state before street address. So you have to add a new column to the table to specify sort order, if you go with the quickest method, or a new table to specify each field's order index to their field ID.
What if you want to group parts of the information separately? Then you have to add more UI-specific cruft into your database layout (to do this generically, you'll need a new table specifying which UI fields belong to which UI groupboxes). So you've only solved two problems and already your database layout has gotten twice as ugly, plus now instead of a simple O(1) layout operation when you load the UI, you've gotta do several database queries to find out what fields exist and dynamically lay them out while applying the correct widget order... and we haven't even dealt with sizing (should every field be the maximum size to fit its possible contents, or should all text fields be the same width? Wouldn't it be nice if you could say that some text fields should be one width and height, and some should be another combination? etc), or text justification, or formatting, or any other really common elementary usability requirements that will require further sacrifices from the clarity and simplicity of your database schema.
Most visual database editors. phpMyAdmin for instance.
Because the database structure isn't always a very good logical structure for a user to be using, especially in the case of databases that have been denormalised on purpose for efficiency reasons.
Yup, this route has already been traveled.
Simply pointing at a database will create an oversimplified UI, not giving much more than the CRUD of an Access UI. That's why Naked Objects (I'm one of its committers) builds its metamodel from a pojo domain model. This allows the UI to expose any public methods as menus ... we call this "behaviourally complete".
Per the comment about the UI not being suitable for end-users, I have two points:
distinguish between power users vs casual users. Most internal apps are for the former (we use Alan Coopers' term of a "sovereign application" for this), who understand the domain and don't want fancy UI stuff getting in the way. Most external apps, eg public web sites, are for the latter.
for the latter, there's nothing to prevent the autogenerated UI of a tool like Naked Objects being replaced with a custom or semi-customized viewer. One such viewer is Scimpi, I'm also working on an Eclipse RCP viewer that'll expose extension points. But even here, the auto gen UI is still very valuable for the development team and business analysts for exploration and prototyping.
Hope some of the above has piqued your interest. If you want more, google around, or you might want to check out my book on domain-driven design and NO, at pragprag.com.
HTH
Dan
List of projects that implement this idea.
.NET
dotObjects
Naked Objects
TrueView
Java
Domain Object Explorer
JMatter
Naked Objects
Sanssouci
Trails
Lablz
C++
Typical Objects
Specifically to your second question: Alot of it really depends on your data model. Some are very complicated and would lead to un-intuitive user interfaces. Perhaps for simply CRUD based systems, having your UI be a front end to the database would be preferable. In that case, I think that some of these tools would be great. However, for some more complicated systems where some db data needs to be hidden from the users, it would be better if you UI didn't mirror the db schema.
Microsoft Access has used this model for years - the database and UI development are very closely tied. You can auto-generate a form directly from a table definition with smart defaults and search built in. The model works well for developing applications with few concurrent users such as custom applications for small businesses where the amount of data stored is small.
If you are scaling to larger relational DBs with a number of concurrent users, or large databases then reliability and performance become more important, and separately constructed UI and databases make more sense. When more users are involved they often have different requirements so decoupling the UI from the DB schema makes it more efficient to develop.
Just a note on Java "projects that implement this idea" - tynamo is the new version of Trails framework
There are many systems that build an interface for you to edit stuff directly from table information. End-user interfaces, however, must be tweaked a little bit. You may not want to reveal to the user every field in your table.
Frameworks that make good use of the MVC design pattern can let you do all kinds of things with your models, which are the preferred way to build new systems (rather than creating database tables directly).
To answer your questions specifically:
django allows you to construct forms (and a complete admin CMS) out of models.
It is a common thing to do.
Naked Objects is about one step removed from this. They base the UI on an object model, and then persist the object model.
I think you are forgetting to consider the user in your design process if you are thinking like that. Bad mistake. Users don't like it when the interface changes, they would especially not like it if it changed frequently as they then wouldn't know what to do. Further, if you generate your UI on the fly based on the database structure, then what order would the objects be in? UIs need to have objects in an order that makes sense to the users not the database designers.
Further in a well-designed database there are fields that are not meant for the users to see. Things like numeric keys, insert date, last updated etc. You don't want to automatically expose these to the users and you certainly don't want them to have the ability to mess with the data in such fields.
Finally, if you don't think about the functionality of the page, then you aren't doing your job. A UI needs to be more than just a list of fileds that can be edited. You need to have constraints on who can see what, checks of the data before inserting to the database, business rules that need to be applied. You can't just autogenerate a lot of this (and you shouldn't even if you could!). Design needs thought and care.
Now as to drop down lists, of course you can generate them from the database and not the code, in fact it is the better choice. Just make the query the source for your particular object, not a list generated in code.
You can do it with the help of this cool tool from a developer in Philippines, it is called COBALT. You can download it here.
I am building a system that allows front-end users to define their own business objects. Defining a business object involves creating data fields for that business object and then relating it to other business objects in the system - fairly straight forward stuff. My question is, what is the most efficient storage strategy?
The requirements are:
Must support business objects with potentially 100+ fields (of all common data types)
The system will eventually support hundreds of thousands of business object instances
Business objects sometimes display data and aggregates from their relationships with other business objects
Users must be able to search for business objects by their data fields (and fields from related business objects)
The two possible solutions I can envisage are:
Have a dynamic schema such that when a new business object type is created a new table is created for storing instances of that object. The object's fields become columns in the storage table.
Have a fixed schema where instance data fields are stored as rows in basically a big long table.
I can see pros and cons to both approaches:
the dynamic schema allows me to index search columns
the dynamic tables are potentially limited in width by the max column size
dynamic schemas rule out / cause issues with replication
the static schema means less or even no dynamic sql generation
my guess is the static schema may perform like a dog when it comes to searching across 100,000+ objects
So what is the best soution? Is there another approach I haven't thought of?
Edit: The requirement I have been given is to build a generic system capable of supporting front-end user defined business objects. There will of course be restrictions on how these objects can be constructed and related, but the requirement itself is not up for negotiation.
My client is a service provider and requires a degree of flexibility in servicing their own clients, hence the need to create business objects.
I think your problem matches very well to a graph database like Neo4j, as it's built for the requested kind of flexibility from the beginning. It stores data as nodes and relationships/edges, and both nodes and relationships can hold arbitrary properties (in a key/value fashion). One important difference to a RDBMS is that a graph database won't need to lookup the relationships in a big long table (like in your fixed schema solution), so there should be a significant performance gain there. You can find out about language bindings for Neo4j in the wiki and read what others say about it in this stackoverflow thread. Disclaimer: I'm part of the Neo4j team.
Without much understanding of your situation...
Instead of writing a general purpose one-size-fits-all business objects system (which is the holy grail for Oracle, Microsoft, SAS, etc.), why not do it the typical way, where the requirements are gathered, and a developer designs and implements the users' business objects in an effective manner?
If your users are typical, they will create a monster, which will end up running slow, and they will hate it. Most users will view the data as an Excel sheet, and not understand relationships like: parent/child. As a result there will be some crazy objects built, and impossible-to-solve reports. You'll be forced to create scripts to manually convert many old objects to better and properly defined ones, etc...
Your requirements sound a little bit like an associative database with a front end to compose and edit entities.
I agree with KM above, unless you have a very compelling reason not to, you would be better off using a traditional approach. There are a lot of development tools and practices that allow you to build a robust and scalable system. Otherwise you will have to implement much of this yourself.
I don't know the best way to do this, because it sounds like something that has already been implemented by others. If I were asked to implement this feature, I would recommend buying a wheel instead of reinventing it.
Perhaps there are reasons you have to invent your own? If so, then you should add those reasons to the requirements you listed.
If you absolutely must be this generic, I still recommend buying a system that has been architected for this requirement. Not just the storage requirements, which are the least of the problems your customer will have; but also: how do you keep the customer from screwing up totally when given this much freedom. Some of the commercial systems already meet this challenge without going out of business because of customers messing up.
If you still need to do this on your own, then I suggest that your requirements (or perhaps those of another vendor?) must include: allow the customer to get it right, and help keep the customer from getting it wrong. You'll need some sort of UI to allow the customer to define these business objects, and the UI should validate the model that the customer builds.
I recommend a UI that works at a conceptual level. As an example, see NORMA, a Visual Studio add-in for Object-Role Modeling (the "other" ORM). Consider it as a example only, if your end users cannot afford a Visual Studio Standard license. Otherwise, you'll find that it is extensible, already produces many types of artifact (from SQL in various dialects to code), and will validate the model to see that it makes sense. End users would also be able to enter sample data that they believe should be valid, and the system will validate the data against the model.
If your customers are producing sensible (if dynamic) business objects, then the question of storage will be much simpler.
Have you thought about an XML based solution? The requirements suggested to me "Build a system that allows users to dynamically generate an XML Schema and work with XML documents based on that schema." I don't know enough about storing and querying XML documents to comment on your original question.
Another possibility might be to leverage NHibernate's ability to generate database schemas. If you can dynamically generate business objects, then you can generate XML mappings or Fluent mappings and use that to generate a normalized database schema.
Every user that I have ever talked to has always wanted "everything" in their project. Part of the job of gathering requirements is to guide the user, not just write down everything they say.
Your only hope is to build several template objects, that they can add properties to, you could code your application to handle each type of these objects, but allow the user to still slightly modify each as necessary.
You need to inform the user upfront of the major flaws this type of design has. This will help you in the end, when it runs slow, or if they screw up and need help fixing something. I'd put this in writing.
How many possible objects would they really need? Perhaps you could set these up using your system first. I have developed several very customizable systems over the years and when the user is sitting at an empty screen, it is like a deer in the headlights.
In any event, good luck.