I'm in the process of redesigning a SQL query builder UI for a application that I use, well it's a third party app but has a plugin model so I writing my own to replace the standard one. One area I am not very good at when it comes to application design is the UI.
I would be really interested to see/hear how other people would layout this dialog, to make it a little bit better, or do you think that it is good as it is.
I am using WPF for my new design so I can be pretty flexible with the UI.
This is the current SQL UI:
alt text http://img190.imageshack.us/img190/9492/sqlq.jpg
Just for clarity when you select a option from the Tables menu, it adds the table name to the "from Tables" text box, same with the Columns drop down into "Select Columns" box.
Who exactly would use such a UI? Since it's basically just a pre-defined SELECT statement template, I would expect such a tool to be used by people who know how to write SQL statements from scratch (since that's essentially what they're doing). The addition of available columns and tables and stuff like that is nice, I guess.
If the tool is meant to be used by experienced SQL users, the tool looks much too restrictive. For example, it appears that one would have to write his own JOIN clauses as WHERE clauses, making outer joins bothersome. It's much less useful than an open textbox.
If the tool is meant to be used by inexperienced users who might not know the ins and outs of relational database management systems, this thing is laughably awful. Too much needs to be done in spite of the restrictiveness of the tool. Too much knowledge of SQL is needed to understand what the form needs. It's more useful than an open textbox, but not much.
Ways to improve the tool depend on the end users. If they're experienced with SQL, the tool should be more of an open textbox where they can write whatever they want, including JOINs. Look at how Microsoft's Server Management Studio allows users to build queries: very nonrestrictive, with the ability to view tables and join them by dragging one column to another. That's a very natural way to approach queries in a visible, organized manner.
If the users aren't experienced with SQL, the tool should do a lot more for them. For example, if the user picks two tables, it should automatically join them so that he doesn't have to understand how joins work. Also, the comboboxes on the side should be able to work in any of the open text boxes, not just one specific text box each. Because you need column names in the SELECT part as well as the WHERE part.
Just think about your userbase, and tailor to the tool to them.
Makes perfect sense if you know how to write SQL.
I would consider perhaps moving the Tables-Columns-etc dropdownlists closer to where their contents will be displayed. You could even incorporate them with the left captions for the textboxes. So, for example, where your text says "Select Columns", you could put the Columns dropdownlist in that spot. Then the default selection there could be .
I think it's a nice UI, though.
What about joins?
What is the point of trying to redesign it? Query analyser has one of the best UIs in that it is unobtrusive and lets you do what you need.
I think if you try and shoehorn statements into a format this will not always work but may work in the majority of cases.
It depends on the business requirements really.
A better UI would be to show all the tables, fields in a tree view with check boxes. Trapping the click-event, one can easily aggregate the SQL.
Related
We're developing an application that has to query 3D shapes (and query based on other parameters as well) within a bounding box. The number of shapes is more than I want to keep in memory, so I need a database to handle it.
Specifically, our primary operations are inserts and queries. We never modify existing data.
Because it's a desktop application, I'm trying to avoid the PostgreSQL and MySQL separate server types of things, hoping for something more simple for deployment. I found Spatialite but it does not index on the 3rd dimension, so it won't work.
I tried searching for kd-tree database but haven't found anything yet. I know there are kd-tree implementations, but getting it in database form would take a lot of effort to roll our own, so I'm trying to see if there is something already out there.
The application is in Haskell, but if we have to integrate with some other language, we might deal with that.
SQLite R*Trees
Given a query rectangle, an R-Tree is able to quickly find all entries that are contained within the query rectangle or which overlap the query rectangle. This idea is easily extended to three dimensions for use in CAD systems.
I would respectfully challenge your attempt to avoid PostgreSQL/MySQL. I'm experienced in PostgreSQL and it does the job you want and it is not difficult to administer. Certainly, anything else you find is not going to have the level of development and testing PostgreSQL does - so why bother?
I have a very large database I need to diagram. The database is SQL Server 2008 on x64. It is large in that there are hundreds of related tables, each with up to 2000 fields (some are sparse), multiple relationships between tables (often hundreds per table, in fact), multiple schemas... you get the idea.
I tried to use the Database Diagrams feature of SQL Server Management Studio, but it crashed with a Win32Exception: "Not enough storage is available to process this command..."
I tried to use Visio's reverse engineering feature on a different machine to connect in and diagram it, but that's been going for a few hours with no sign of completion.
The scripts to build this giant schema are being by a tool we built for the job. While the tool is doing its job just fine, it's tricky to visualise its output.
I'm after a tool to kick out a diagram of this database so we can do this. Any suggestions?
EDIT:
Just to emphasize, the diagram is indeed not supposed to be used for actual useful reference. It's a client relationship management device to demonstrate the complexity/scale of the system.
I worked at a place that had several hundred tables (near 1k) and no one really knew what was going on in the system, company was growing and hiring a lot. A guy was tasked with doing a diagram, and he auto-magically created a gigantic tiled poster that contained every table with lines connecting various tables (going all over the place). I'm not sure what he used, it was Unix and Oracle years ago (way before Linux and open source). There was no real rhyme or reason to the layout of the the tables in his diagram. He had successfully created a diagram of every table. The "poster" was put on a wall in a common area, and got a few looks, but no one ever really used it, it was unusable, too cluttered, too unorganized. As a result, I used MS-Word to create a single page diagram containing the 20 main tables (it went through a few iterations as I "discovered" new main tables) with lines for each foreign key and each table located in a logical manner. I showed the column name, data type, nullability, PK, and all FKs. I put my diagram up on my wall by my monitor. Eventually everyone wanted a copy of my diagram, including the person that made the "poster". When I left that job they were still giving my diagram to new hires.
I recommend that you work like an explorer, find the key tables and map them as you go, making as many specific diagrams as necessary as you discover the system. Trying to make a gigantic "poster" automatically will not work very well.
Generating an image of any kind for a database of that size simply becomes eye candy that is stuck on a wall that draw's gasps, and honestly serves no real purpose except occasional glances. Why not use a tool like Red Gate's Documentation tool that will serve an actual purpose? Please understand I'm not saying this in a mocking way, but I've been down this road before trying to diagram a huge database, and I succeeded to some degree, but never found a good outlet where it was of some use.
Since you have multiple schemas maybe a good idea is to generate diagrams per schema instead
Use graphviz. Use some SQL statements to generate the digram, then run it through dot.exe to generate a PDF or PNG.
I've used it to generate digrams of data within SQL Server tables. No reason why you can use it for tables too.
http://www.graphviz.org/
There are also java, silverlight, and AJAX utilities for navigating extra large graphs, as PDF is only for one page.
I'd avoid doing the whole thing in a single diagram. As you mentioned, the tools crash, and it's probably not possible to easily comprehend a diagram with hundreds of tables with potentially thousands of records per table. Can you generate diagrams of smaller logical areas with some overlap to other
logical areas?
Alternately, you could try using something like graphviz to parse the DDL statements and then produce a graph. It will probably churn for a while, but I remember seeing in a university poster-sized diagrams with tiny print, that were probably of the same complexity as yours. Good luck!
FWIW, assuming you do want to go ahead with this I've personally found that the visual studio 2010 database modeller does the nicest diagrams I've come across so far - Just import your database as if you were going to use it for Linq2SQL
schemaspy
provides a handy interface to generate interactive diagrams that span multiple schemas using graphviz as a backend. I've never tried it on anything this size though.
IntelliJ (specifically IDEA as just tried with this, but I believe their other IDEs offer this feature https://www.jetbrains.com/) has a built in database client facility, from here you can connect to your database and analyse individual tables, specific combination or table or all your tables by highlighting the desired tables, 'right clicking' and selecting the 'diagram' option. You can save for later reference and also print. I have just tried this on a large DB of 500+ tables and it rendered in seconds, the vector diagram serves as an alternative way to digest database structures visually and the relationships and constraints between certain tables but not recommended for printing.
People seem to avoid building user interfaces that pull their information (names, field types, etc. as well as relationships) from a database; they instead hard-code forms (and tables, etc.) that have pretty much the same data names and types and things.
Am I making sense?
For instance, imagine an emumerated field in MySQL: why not just have the UI construct a drop-down list whenever it encounters an ENUM? Why put the same values in both the database and the code?
Perhaps I'm just missing something; perhaps there are projects out there that do this — sort of super-crud interfaces that can be pointed at any database and from it build a fully-functional relationally-aware user interface. Are there?
I'm possibly not quite conforming to the stackoverflow norms with this question; I shall summarise:
Can you please tell me of a project that constructs its user interface (solely) from analysis of the database schema?
Why is this not a common way to do it — surely it is good to only define data structure in one place (i.e. the database)?
Thank you, and may joyous code-love rain upon your IDE.
I'd like to point out that, last time I checked, .NET and Qt (and probably other environments) make it possible to use "database-aware widgets" (sometimes shortened to just data-aware widgets), which is probably the best pragmatic solution available. What I mean by data-aware widgets is that the widgets themselves know that they're linked directly to database fields, so you would have a combobox that knows that it's backed by an enum and fetches the possible values directly from the database at runtime, just like you suggested.
This is a really neat utility, and used well, it probably won't hurt anything. It still requires that you spend some time laying out widgets manually on a form, but then if you update the database to add a new value to that enum, you don't have to rebuild your app to see it show up in the UI.
But the reason most usability experts will cringe when they hear your question is because programmers tend to think that, well, why not just generate the entire UI, form layout and everything, from the database? And this is where it starts to get really nasty, really fast.
Let's say you have a simple Person table, with first_name, last_name, email_address, street_address, city, state, zip, and phone_number. You want to automatically generate a UI based on these fields. How do you sort the fields? I mean, ideally, first name and last name should be right next to one another. And it would look very silly if you had city and state before street address. So you have to add a new column to the table to specify sort order, if you go with the quickest method, or a new table to specify each field's order index to their field ID.
What if you want to group parts of the information separately? Then you have to add more UI-specific cruft into your database layout (to do this generically, you'll need a new table specifying which UI fields belong to which UI groupboxes). So you've only solved two problems and already your database layout has gotten twice as ugly, plus now instead of a simple O(1) layout operation when you load the UI, you've gotta do several database queries to find out what fields exist and dynamically lay them out while applying the correct widget order... and we haven't even dealt with sizing (should every field be the maximum size to fit its possible contents, or should all text fields be the same width? Wouldn't it be nice if you could say that some text fields should be one width and height, and some should be another combination? etc), or text justification, or formatting, or any other really common elementary usability requirements that will require further sacrifices from the clarity and simplicity of your database schema.
Most visual database editors. phpMyAdmin for instance.
Because the database structure isn't always a very good logical structure for a user to be using, especially in the case of databases that have been denormalised on purpose for efficiency reasons.
Yup, this route has already been traveled.
Simply pointing at a database will create an oversimplified UI, not giving much more than the CRUD of an Access UI. That's why Naked Objects (I'm one of its committers) builds its metamodel from a pojo domain model. This allows the UI to expose any public methods as menus ... we call this "behaviourally complete".
Per the comment about the UI not being suitable for end-users, I have two points:
distinguish between power users vs casual users. Most internal apps are for the former (we use Alan Coopers' term of a "sovereign application" for this), who understand the domain and don't want fancy UI stuff getting in the way. Most external apps, eg public web sites, are for the latter.
for the latter, there's nothing to prevent the autogenerated UI of a tool like Naked Objects being replaced with a custom or semi-customized viewer. One such viewer is Scimpi, I'm also working on an Eclipse RCP viewer that'll expose extension points. But even here, the auto gen UI is still very valuable for the development team and business analysts for exploration and prototyping.
Hope some of the above has piqued your interest. If you want more, google around, or you might want to check out my book on domain-driven design and NO, at pragprag.com.
HTH
Dan
List of projects that implement this idea.
.NET
dotObjects
Naked Objects
TrueView
Java
Domain Object Explorer
JMatter
Naked Objects
Sanssouci
Trails
Lablz
C++
Typical Objects
Specifically to your second question: Alot of it really depends on your data model. Some are very complicated and would lead to un-intuitive user interfaces. Perhaps for simply CRUD based systems, having your UI be a front end to the database would be preferable. In that case, I think that some of these tools would be great. However, for some more complicated systems where some db data needs to be hidden from the users, it would be better if you UI didn't mirror the db schema.
Microsoft Access has used this model for years - the database and UI development are very closely tied. You can auto-generate a form directly from a table definition with smart defaults and search built in. The model works well for developing applications with few concurrent users such as custom applications for small businesses where the amount of data stored is small.
If you are scaling to larger relational DBs with a number of concurrent users, or large databases then reliability and performance become more important, and separately constructed UI and databases make more sense. When more users are involved they often have different requirements so decoupling the UI from the DB schema makes it more efficient to develop.
Just a note on Java "projects that implement this idea" - tynamo is the new version of Trails framework
There are many systems that build an interface for you to edit stuff directly from table information. End-user interfaces, however, must be tweaked a little bit. You may not want to reveal to the user every field in your table.
Frameworks that make good use of the MVC design pattern can let you do all kinds of things with your models, which are the preferred way to build new systems (rather than creating database tables directly).
To answer your questions specifically:
django allows you to construct forms (and a complete admin CMS) out of models.
It is a common thing to do.
Naked Objects is about one step removed from this. They base the UI on an object model, and then persist the object model.
I think you are forgetting to consider the user in your design process if you are thinking like that. Bad mistake. Users don't like it when the interface changes, they would especially not like it if it changed frequently as they then wouldn't know what to do. Further, if you generate your UI on the fly based on the database structure, then what order would the objects be in? UIs need to have objects in an order that makes sense to the users not the database designers.
Further in a well-designed database there are fields that are not meant for the users to see. Things like numeric keys, insert date, last updated etc. You don't want to automatically expose these to the users and you certainly don't want them to have the ability to mess with the data in such fields.
Finally, if you don't think about the functionality of the page, then you aren't doing your job. A UI needs to be more than just a list of fileds that can be edited. You need to have constraints on who can see what, checks of the data before inserting to the database, business rules that need to be applied. You can't just autogenerate a lot of this (and you shouldn't even if you could!). Design needs thought and care.
Now as to drop down lists, of course you can generate them from the database and not the code, in fact it is the better choice. Just make the query the source for your particular object, not a list generated in code.
You can do it with the help of this cool tool from a developer in Philippines, it is called COBALT. You can download it here.
I am working on a social networking site with a family tree that is GEDCOM-compliant. We need to decide if we should use a horizontal or vertical database structure for User profiles. So, I would like to know if anybody can answer when to use a horizontal database structure and when to use vertical database structure.
I found some answers for shopping sites where fields are not decided: a vertical database structure should be used. But I am confused about what to use for a family tree site. Should I use vertical or horizontal?
I assume you use a relational database like Mysql, Ms sql, Sqlite, Postgresql or Oracle for storage?
Gedcom is a standard for information exchange so you know how many columns you will have. Maybe the standard is extended with new properties in the future but it probably wont be a lot of new properties. You can easily extend a table with a few new columns.
I would use a 'horizontal' table and not a entity-attribyte-value-system (vertical table). Vertical table systems tend to be slow. They can't be properly indexed and confuse the query optimizer.
It becomes a different story when your users can define new properties in their profiles like eye colo(u)r or favourite colo(u)r themselves. How flexible do you want those profiles to be?
Vertical databases are great for detawarehousing and read/only reporting. Normally you re-generate them overnight. Their write performance is usually very bad however SELECTs are 10-100 times faster.
A typical scenario for using a vertical database is olap reporting when you create a (daily) snapshot of data and then run queries against it. Most of benefit comes from the queries that request only a relatively small number of fields, e.g. when you select only a handful of fields from a wide and large table. Such a query query against millions of records (e.g. calculating SUM/COUNT/AVG) will take only a second or two.
Your case doesn't seem to be a good candidate for a vertical database.
I agree with tuinstoel, vertical table/EAV system is not only slow but also some time very complex. Sometimes it is required to write some of your own api methods that deal with those tables and developers only deal with those methods to avoid complexity.
So if you don't need to add more fields then stay with horizontal table. However you may be need to different table if you also if you are going to support multilingual capability. But I advise to still stick with horizontal tables.
I am also developing an site involving user profile and I am using horizontal tables and if there will be different languages support required in future then I will modify only for fields where language will matter.
I'm in search of a glorious reporting tool. I'm aware that glorious is a subjective term but... here are my desires:
Dead simple for business types
Does not require me to speficy any SQL
Can inspect the database's schema to interpret the users requests without programmer intervention
Costs next to nothing
Can save commonly crafted reports
This sounds an aweful lot like pivot tables, except those would be too complicated for this environment.
Does such a tool exist, or am I doomed to spend the rest of my corporate programming career writing one off reports?
A do-everything report wizard has been the dream of many a programming house.
Step 1 is realize its unnatainable. Step 2 is to find a happy medium between a perfect wizard and a coder writting custom reports all day.
Try creating a company specific wizard. I made one with some handy, "let the user do it" type filtering. I give the user a textbox for each field they want to filter, and let them add comma delimited filters, using % as the wildcard, - for ranges, and constructing the query from those. It also helps to dump results to excel, and let the marketing thugs play with it from there (they usually are pretty handy with excel).
Where it gets hairy is joins. Instead of trying to figure out which tables to join in a report, instead offer them 6-12 standard groupings, each pre-defined grouping should cover just about everything. I did this using a tab UI.
I recommend HTSQL. Queries are simply URLs that follow many predictable conventions (like .../sales?state='OH' for SELECT * FROM sales WHERE state='OH'). Setup is easy, presentation is good-looking. Definitely browse the showcase.
The scary thing about ad-hoc reporting is: you're certain to get an answer. Knowing that the answer is right is when you need us - the guys who understand the schema.
It's when you try to accomodate that by restricting the kinds or scope of reporting you can do to simple filters and such that they get less and less useful.
I recommend trying out (our very own) i-net Clear Reports. In my mind it's glorious (but I admit to being heavily biased... ;).
Check out a demo of our really dead simple ad-hoc reporting component.
Can export into any major format like PDF, HTML, SVG, XLS, etc., as well as into a Java applet viewer.
We offer a free and fully functional report designer.
Installs as a WAR file on your application server or can be used as a library within your own application.
Great technical support (you usually get an answer in minutes or hours rather than days or weeks)
Can read Crystal Reports templates. (for a lot of customers, this is the killer feature since you don't have to re-create all your old Crystal Reports templates)
Great and competitive pricing - effectively costing "less than open source" will cost you if you calculate in support costs (which you definitely should).