Software like OneNote has shown that auto-save can be implemented, and it works just as well (or better) as the manual save button / CTRL+S.
Anyways everything that you work on you want saved. Its just if you're trying out something destructive that you would close without saving.
So from a programmers/usability perspective, why is the manual "save" feature still seen in virtually all software today? Is it because everyone is too lazy to implement "auto-save" whenever data gets modified?
And is it a good idea for us implement auto-save, at least to start some traction in our specific industry and amongst our competitors?
autosave normally saves on a defined interval. What happens if you want to save in between intervals?
You should implement a manual save to stay consisent with other applications in the environment as well.
People expect file -> save, or CTRL + S to exist.
The save button is a well-known, comfortable UI feature that everyone from Jon Skeet to grandma is familiar with. If you got rid of it, it would be like removing the close button on a window for some people. Granted, they would eventually get used to it, but some people would not understand that their data has been saved automatically.
Also, if you're autosaving on the web, not only are you taking up a lot of space on your server with all those instances, you're also using up a lot of bandwidth with periodical saves. At least with manual save, you are only using the space and bandwidth when the user intends, which can be more infrequent, thus saving bandwidth. The advantage, of course, to autosaving is the retention of work should something go awry.
Check the definition of "skeuomorph" :)
Additionally with a "save" there is commonly "save as.." as well. Both give the user the feeling of control and security. Knowing that they clicked save lets them know at what state they can expect their data to be in when reloading it.
It really comes down to this: a Save button is cheaper to implement and maintain than Undo.
It is not hard to implement auto-save - just implement a normal save and call it when ever needed or just in a timer (if you are lazy).
Save buttons are common because of the common pattern learned by the users for decades.
Load data or files from a persistent storage into main memory.
Modify the data in main memory.
Save the modified data back to a persistent storage.
This pattern comes from the old distinction between harde drive and main memory. If you think about it in another way (as some experimental operating systems do), there is no need for loading and saving files - just think about a hard drive as your main memory and and the main memory as another cache level for the hard drive. In consequence all files (not on removeable medias) are always in memory and you will never again need to load or save files.
But doing this change is not easy because users are used to the old pattern for years. Further the old load and save pattern is an very easy way to get a kind of primitve undo system.
Auto-saving requires an undo system, too, and it is not that trivial to build one. Esspecialy if you perform image, audio or video editing and you are producing a lot of data it is hard to find a good time-memory-trade-off. And there is the risk that user will try to undo things by closing the application and then recognize that this did not work. So it might even be a good idea to persist undo information to protect users from this error or saving unwanted changes in the case of a crash.
So, yes, I would realy like to see the save (and load) buttons to disappear. I would like persisted undo-information or even complete edit histories, too. But I don't think this change can happen in a few years - if ever.
I work in the medical field and there are situations where you want the user to take responsibility for saving something. If you have an EHR and you are entering a prescription for a patient then you dont necessarily want it autosaving - you want the user to be aware of and take responsibility for their actions. Also, autosaving a value in a critical system like this could be disastrous for obvious reasons...
Should be tagged subjective maybe?
As a developer, I am always a little uneasy around apps like that. I like having control over when my data is saved, though perhaps this is just years of conditioning at work. I get that little "uh oh" feeling whenever I close a window into which I've entered data without explicitly pressing a close button (or shortcut).
That said, I have been "trained" to accept it in certain situations. OneNote, for example, or Tomboy. A great many OS X apps follow this pattern, especially utility apps like DB server GUI tools.
So, in short, different tools for different situations. IMO, most software these days would not benefit from a move from a manual save to an auto-save.
I think the answer to this is that 'it depends'!
You should consider not only your user's expectations in terms of consistency with other applications, but also the way in which the user is going to use your application.
A very common use-case for OneNote is that someone opens it up to dump in some information almost as an aside to what they're working on. They need to get in and out quickly. Any prompts about saving would be a nuisance.
Applications like Word, on the other hand, expect users to be spending a concerted amount of time working on a document. In this case, the chore of manually saving and responding to confirmation boxes etc will be seen as a relatively small task.
From programmers perspective implementing autosave would not be a huge deal. You just set up a timer and callback would do the saving.
However, from usability point of view autosave is very problematic. First of all users are used to having a manual saving and not offering it to them would confuse a majority of users and take feel of control away.
Even bigger issue would be that autosave overwrites contents of underlaying file whether you wanted it or not. Of course you could have autosave feature saving onto temporary file but the decision to overwrite original document must always come from the user, not from the software. And because you would anyways need the user to initiate at least one manual saving, why not enable manual saving to be available always?
An autosave feature is great when you are dealing with a document. What about a business application? If I edit a customer's account, should it update the account as I tab out of the edited fields? If so, what should it do when the account is in an invalid state? When do you enforce business rules and how do you enforce them? How will it perform when you have to take business rules into account on every edit?
You can certainly build an application that will take any of these considerations into account, but will it have been worth the extra effort?
So should we get rid of the Save button? It depends.
Short answer: "auto save" = "auto destroy" / "auto <expletive>".
For one project in university, my group and I built an application without explicit saving as an experiment.
We implemented an infinite undo stack and serialized the undo stack with the actual data so that even if you closed the app and re-opened it, you could always undo your last operation. Every operation wrote a new entry to the action list on disk so that the file was always consistent (well, mostly...), even if the power failed. It was a bit of a cross between a version control system and a journaling file system.
There were two problems: one, we didn't have time to get it completely right (ah, youthful hubris); two, everyone (fellow students and, most importantly, the TAs) hated it, because of all of the reasons mentioned already.
Sometimes, for all your best intentions, you just can't ignore ingrained behaviours.
Related
I am trying to get input from the user that the program will remember every time it runs. I want to do this in C. Based on my limited knowledge of computers, this will be stored in the storage/hard drive/ssd instead of the RAM/memory. I know I could use a database, or write to a text file, but I don't want to use an external file or a database, since I think that a database is a bit overkill for this and an external file can be messed with by the user. How can I do this (get the user to enter the input once and for the program to remember it forever)? Thanks! (If anyone needs me to clarify my question, I'd be happy to do so and when I get an answer, I will clarify this question for future users.)
People have done this all kinds of ways. In order from best to worst (in my opinion):
Use a registry or equivalent. That's what it's there for.
Use an environment variable. IMO this is error prone and may not really be what you want.
Store a file on your user's computer. Easy to do and simple.
Store a file on a server and read/write to the server via the network. Annoying that you have to use the network, but OK.
Modify your own binary on disk. This is fun as a learning experience, but generally inadvisable in production code. Still it can be done sometimes especially using an installer.
Spawn a background process that "never" dies. This is strictly worse than using a file.
You won't be able to prevent the user from modifying a file if they really want to. What you could do is create a file with a name or extension that makes it obvious that it should not be modified, or make it hidden to the user.
There isn't really any common way that you could write to a file and at the same time prevent the user from accessing it. You would need OS/platform level support to have some kind of protected storage.
The only real alternative commonly available is to store the information online on a server that you control and fetch it from there over the network. You could cache a cryptographically signed local copy with an expiration date to avoid having to connect every time the program is run. Of course if you are doing this as some kind of DRM or similar measure (e.g., time-limited demo), you will also need to protect your software from modification.
(Note that modifying the program itself, which you mentioned in a comment, is not really any different from modifying other files. In particular, the user can restore an earlier version of the program from backup or by re-downloading it, which is something even a casual user might try. Also, any signature on the software would become invalid by your modifications, and antivirus software may be triggered.)
If you simply wish to hide your file somewhere to protect against casual users (only), give it an obscure name and set the file hidden using both filesystem attributes and naming (in *nix systems with a . as the first character of file name). (How hidden you can make it may be thwarted by permissions and/or sandboxing, depending on the OS.)
Also note that if your goal is to hide the information from the user, you should encrypt it somehow. This includes any pieces of the data that are part of the program that writes it.
edit: In case I guessed incorrectly and the reason for wanting to do this is simply to keep things "clean", then just store it in the platform's usual "user settings" area. For example, AppData or registry on Windows, user defaults or ~/Library/Application Support on macOS, a "dotfile" on generic *nix systems, etc. Do not modify the application itself for this reason.
If you want to persist data, the typical way to that is to store it to a file. Just use FILE* and go about your business.
Using a database for this may be an overkill, it depends on how you want to later access the data once it is stored.
If you just load the data from the file and search through it, then there is no need for a database, if you have loads of data and want to make complex searches, then a database is the way to go. If you need redundancy, user handling, security then choose a database, since the developers of each one already spent a lot of time fixing this.
It doesn't matter if you're building an eshop or any other application which uses session to store some data between requests.
If you don't want to annoy the user by requiring him to register, you need to allow him to do certain tasks anonymously when possible (user really have to have a reason for registering).
There comes a problem - if user decides to login with his existing profile, he may already have some data in his "anonymous" session.
What are the best practices of merging these data? I'm guessing the application should merge it automatically where possible or let the user decide where not possible.
But what I'm asking more is if there are any resources about how to do the magic in database (where the session data are usually stored) effectively.
I have two basic solutions in my mind:
To keep anonymous session data and just add another "relation" saying what's actually used where and how it's merged
To physically merge these data
We could say that the first solution will probably be more effective, because the information about any relation will probably mean less data than data about the user. But it will also mean more effort when reading the data (as we firstly need to read the relation to get to actual user data).
Are there any articles/resources for designing data structures for this particular use case (anonymous + user data)?
An excellent question that any app developer using user data should ask, and, sadly very few do :(
In fact, there are two completely independent questions here:
Q1 - At what stage require user to sign in/up?
Q2 - Data concurrency and conflict resolution (see below).
And here some analysis for each of the questions. Please excuse my extra passion coming from my own "frustrated user" experience. :)
Q1 is a pure usability question. To which the answer is actually obvious:
Avoid or delay to force the user sign in as much as possible!
Even the need to save state is not enough a reason by itself. If I am as user not interested in saving that state, then don't force me to sign! Please!
The only reason for you (as website) to justify forcing me to sign is when I (as user) want to save my data for later use. Here I speak as user having wasted time on signing only to find the site useless. If you want to get rid of too many users, that is the right way. In any other case - please, delay it as much as possible!
Why so many sites completely disregard such an obvious rule? The possible reasons I can see:
R1- developer friendly vs user friendly. Yes, it is developer friendly to require sign in right away, so we don't need to bother with concurrency (Q2). So we can save developer costs, time etc. But every saving comes at a cost! Which in this case is called User Experience. Which is not necessarily where you would like to look for saving. Especially, since the solution should not be that hard (see below).
R2 - Designer or Manager making the decision is an "indoor enthusiast" :) She lives happy life surrounded by super-fast computers with super-fast internet connection and can't imagine singing up can be that hard for any user. So why is it such a big deal? So many reasons:
It breaks the application flow. Sites living in previous century still replace the whole screen with sometimes rather lengthy form. Some forms are badly designed, some have erratic instructions, some simply don't work. Some have submit buttons that are for some reason disabled in the browser used.
Some form designers have genius idea to lock certain fields with barely noticeable change or colour. Then don't show me the field if you don't want me to fill it!
If the site is serious about user's data, it must request Email and must verity it! Why? How else shall I get back to user who forgot all other credentials? Why verify? What if user mistyped the email? If I don't verify it, next time the user tries to recover password with her correct email, the recovery fails and all data are lost! Obvious, yet there are still sites out there not doing it. Then I need to wait till the verification email is received and click on, hopefully, well-formatted and uniquely identifiable link that does not break in my browser, nor get some funny characters due to broken encoding detection, making the whole link unusable.
The internet connection can be slow or broken, making every additional step a piece of pain. Even with good connection, it happens here and there that page suddenly takes much longer to load. Also the email may not arrive right away. Then impatient user starts furiously clicking the "resend verification" link. In which case 90% of sites resend their link with new token but also disable all previous tokens. Then several emails arrive in unpredictable order and poor user has to guess in vain, which one (and only one) is still valid. Now why those sites find it so hard to keep several tokens active, just for this case, is beyond my understanding.
Finally there is still this so hard to unlearn habit for sites to insist on the so-called "username". So now, beside my email, I have to think hard to come up with this unique username, different from any previous user! Thank you so much for making it sweet and easy! My own way of dealing with it is to use my email as username. Sadly, there are still sites not accepting it! Then what if some fun type used my email as his username? Not so unrealistic if your email is bill#gates.com. But why simply not use Email and Password to avoid all this mess?
Here some possible guidelines to relieve user's pain:
Only force me to sign in/up if you absolutely need and give me a chance to choose not to!
Make it one page form, so I know what I am up to and, needless to say, use as few input fields as possible. Ideally only Email and Password (possibly twice), no Username!
Show your sign in form as small window on top of your page without reloading, and allow me to get rid of it with single click away from that window. Don't force me to look for "close" button or, even worse, icon I could confuse for something else!
Account for user to click back/forth and reload buttons. Don't clear the form upon reload! Don't put clear button at all! It is too easy to click by accident. The data you are ask me to fill should not be so long in first place, that I could not re-enter it without the need of "assistance" to clear.
Now to question Q2. Here we have well known problem of conflict resolution that occurs any time two data need to be merged. For instance, the anonymous data and the registered user data, but also whenever two users modify the same data, or the same user modifies it from different devices at different times, or locally stored data conflict with server data, and so on.
But whatever the source is, the problem is always the same. We have two data, say two objects $obj1 and $obj2 and you need to produce your single merged object $obj3. The logic can be as simple as the rule that server's object always wins, or that the last modified object always wins, or the last modified object keys always win or any more complicated logic. This really depends on the nature of your application. But in each case, all you need to do is to write your logic function with arguments $obj1, $obj2 that returns $obj3.
A solution that will possibly work in many cases is to store timestamp on each object attribute (key) and let the latest changed key win at the moment of synchronisation. That accounts e.g. for the situation when the same user modifies different attributes when being anonymous from different devices.
Imagine I had modified keys A and B on device AA yesterday, then logged today from device BB to enter another B and saved it to the server, then switched back to my device AA, where I am anonymous, to enter yet another A without changing the old B from yesterday, and then realised I want to log in and synchronise. Then my local B is obviously old and should clearly not overwrite the value of B that I changed more recently on device BB. In this seemingly complicated case, the above solutions works seamlessly and effectively. In contrast, putting the timestamp only on whole objects would be wrong.
Now in some cases, it could make sense to keep both objects, and, e.g. distinguish them by adding extra properties, like in case 1 suggested in Radek's question. For instance, Dropbox adds something like "conflicted copy by user X" to the end of the file. Like in Dropbox case, this is sensible in case of collaboration apps, where users like to have some version control.
However, in those cases, you as developer simply save two copies and let the users deal with that problem.
If on the other hand, you have to write a complicated logic based on user's data, having two different copies hanging around can be a nightmare. In that case, I would split data into two groups (e.g. create two objects out of one). The first group has data representing the state of the application as a whole, that is important to be unique. For that data I would use the conflict resolution as above or similar. Then the second group is user-specific, where I would store both data as two separate entries in the database, properly mark them (like Dropbox does), and then let users deal with the list of two (or more) entries of their project.
Finally, if that additional complication of database management makes the developer uneasy, and since Radek asked to give a resource reference, I want to "kill two flies with one shot" by mentioning the blog entry StackMob offline Sync, whose solution provides both database and user management functionality and so relieves the developer from that pain. Surely there is a lot more info to be found when searching for data concurrence, conflict resolution and the likes.
To conclude, I have to add the obligatory disclaimer, that all written here are merely my own thoughts and suggestions, that everyone should use at own risk and don't hold me responsible if you suddenly get too many happy users making your system crash :)
As I am myself working on an app, where I am implementing all those aspects, I am certainly very interested to hear other opinions and what else folks have to say on the subject.
From my experience - both as a user of sites that require a login, and as a developer working with logged in users - I don't think I've ever seen a site behave this way.
The common pattern is to let a user be anonymous and the first time they do something that would require saving state, they are prompted to login. Then the previous action is remembered and the user can continue. For example, if they try to add something to their shopping cart, they are prompted to login and then after login, the item is in their cart.
I suppose some places would allow you to fill a cart and then login at which point the cart is associated with a concrete user.
I would create a SessionUser object that has the state of the site interaction and one field called UserId that is used to retrieve other things like name, address, etc.
With anonymous users, I would create the SessionUser object with an empty reference for UserId. This means we can't resolve a name or an address, but we can still save state. The actions they are performing, the pages they're viewing, etc.
Once they login we don't have to merge two objects, we just populate the UserId field in SessionUser and now we can traverse an object graph to get name, email, address or whatever else.
It's basically a CRUD application.. Are there any Access add-ons to simplify development of screens, reports, SQL stored procedures? I'm not familiar with VBA or Macros. I need to be able to control cursor movement, perform conditional if's, what happen before or after entering a value, table lookup's, etc. etc.
I've tinkered around with Access, already have a schema and relations defined, but I imagine the learning curve will take quite some time and heard it takes time to get up to speed on VBA and Macros.. I've even heard its better to develop this app with a VB add-on called RadVolution. I would actually rather develop this app with as a touch-screen POS-style app, but not aware of any RAD Tools or SDK's for that. I'm familiar with Informix-SQL and non-visual Basic, but have no C or Object-Oriented language experience, I'm old school procedural language (Basic, COBOL).. Anyone willing to work or help me with this conversion project?
A real strength of Access is its simplicity, it is described as a "rapid application database development and reporting tool". For the most part, you do not need any VBA for forms and reports, if it looks complicated, it is likely you are doing something wrong. The only question is how many users do you expect to have and where do you expect to run your application? A lot of bad press is due to misuse, used properly, Access can be very handy indeed.
You really don't want some type of automated conversion anyway. The reason is that you need to CHANGE how you done some these things to work with the architecture change that occurs when you move from one development system to another.
An automated conversion is probably not much worth it since so often how things were done in the previous system will be done differently in the new system. For example when people came from DOS based (text screen based) FoxPro applications to Access there was two significant changes the developer had to make:
1) No record numbers in Access
FoxPro (which was a dBase compatible clone) had its roots in being a singer user file based database system. So this was system designed from the ground up to operate on your personal computer. This meant the file and programming system used sequential records from a file on the hard drive. This model was much like punched cards data processing. And I should point out there's nothing wrong with this type of model, but the software approaches and designs you used for punch card data processing is somewhat different then compared to interactive multiuser systems.
What is SIGNIFICANT and most important here is on a conceptual level when you write software inside of Access then record numbers or so call record order as a conceptual idea that as you write software is NOT relevant. However in Foxpro assuming record order was a legitimate assumption. This is an architecture change. I remember back in early 90's that in many forums one the FIRST questions from long time Foxpro developer asked is:
How come access does not have record numbers like Foxpro does?
The answer was simple and that answer is/was Access considers data a big huge unordered bucket of data. In other words if you wanted some order to your data, you had to add something like an invoice number, perhaps even a time stamp, or something else. For something like a sub form as a general rule you can rely the auto number, but that auto number should never been seen by the users. No matter what, you had to use a SQL query that SAYS what order you want.
Another important detail is if you add 10 records to a table (even in single user mode), if you then retrieve those 10 records from that table, you can NOT assume the record order will be the same as when you added them. In other words if you need a particular order, you have to sort that data by some column. This is an assumption that FoxPpro, or people who used FORTRAN and punch cards could always assume. Such assumptions cannot be made when using Access. In fact this assumption cannot be made with any modern server based system such as SQL server.
However this "lack of" record order assumption was SIGNIFICANTLY important later on down the road. This "assuming" means that THEN your WHOLE Access design was now based on assumptions that ALLOWED easy conversion to multiuser systems OR that of client to server (both need this assuming).
So your software designs could never say go to the next or previous record (based on record number) since records are now a mumble jumble of records being entered by different people. The next two reocrds in that table (or previous ones) might not be yours! So keep in mind that while Access allows you go to the next/previous record inside of a form, it NEVER does so based on record number but ONLY on the data that's been CURRENTLY loaded into that form. In FoxPro you would often move around by actually using command that said go to record 4 in the table.
In Access we don't say go to the 4th record in the table. You might say go to the 4th record in some data that we pulled from a table into a form, but that 4th record has absolutely nothing to do with the actual physical fourth record in the table. A small change, but one that was required for multi-user systems that we started to use 10 years later (so the smale change in software produced beneifts 10 years later!).
As a general rule this architecture or philosophical concept of record order in a table is not a very big deal at all for most of the software you write, but if you needed to use SQL server later on, then it was a big deal. And I should point out that since your software is written using SQL in mind, then at least in this regards you are in good shape.
However for those that wrote applications over 4 to 5 years based on this simple record order assumption, it would have to be completely RE architectured for multiuser environment or even for Access.
I should point out that FoxPpro eventually became a brilliant object orientated client server development tool, but had to go through a significant metamorphosis than what the original architecture and designs that a typical FoxPro application had.
2) Event driven programming
In these older text based systems you tended to write one large major startup program with a main menu system included. Choosing a menu option would then perhaps branch to a section in the main program or perhaps branch out and call another portion or part of the application. However to its credit, FoxPro and a good number of dev tools did have some event type of setup, but they were not ideal. It best to re-do much of how those screens and UI will work when droping text based UI anyway - this is VERY much the case with new touch based and gesture based UI wew now see such as smartphone or iPad.
In event driven programing we as a general rule don't have that large startup program. And we also don't have a large code base for the main menu system. In event driven programming you have code that RESPONDS to a user click. Or you have code that responds to a record save. Or even navigation to the next record.
So in event driven programming you click on a button then a rather SMALL bit of code would respond to an event by the user (in this case a mouse click). So this type of programming is what we call event driven programming.
All of a sudden, your application is now not being driven or run by one big large main program, but in fact is a whole bunch a little tiny small programs stitched together by event driven code.
For people coming from an DOS based environment, or even QuickBasic, GW-basic or even many of the older text based database systems, then having one large startup program with some menu system was common.
And having a large program to "edit" one data entry screen was common.
Now such designs are turned upside down in which your menus and click events were now going to run and call code. Thus these very small routines would THEN call other bits of code to allow the application to run.
The main reason for this architecture change was the introduction of the mouse and the graphical user interface. In other words when looking at a data entry form, in place of tabbing along to the next field in a complex data entry form the user can now click on many different things and even click on the menu bar. So they can click just about ANY place on the form. This means that having one big program to run and maintain the form data entry is/was not possible. If your code was waiting for input in the Company field, then how could code be run when the user clicked on a menu bar option? Since the user could do many things in many different order than what the original programmer would anticipate then we need a different way of writing code.
At the end of the day with a GUI then code branching became too complicated for the developer to anticipate. Hence the concept of event driven programming was born to solve this dilemma. In other words our code now runs and responds to the users actions and our code is not waiting for the next user input sitting in some line of code.
Again this was a small architecture change, but many developers struggled with this conceptual change in approach to software development. On the other hand all of these changes occurred in the early 90's.
The next big change the course was object oriented programming. Access does allow you to build class objects, but it is not a full OO system.
Today the next big challenge and change and architecture is the ability to build web based systems. Again a zillion changes occur due to having "solve" different programs for the developer. Access 2010 allows one to build web based systems now, and this conceptual and architecture change is a GREATER challenge than that of learning the new macro language we have for developing web forms with Access. So a "change" in how you design software has to occur again.
I should point out that these major changes in the industry happen about once every 10 years or so.
My point of all of the above is, even if there was some automated conversion system, it really would NOT work very well because the architecture of the systems are very different. You would be handicapped by all of the old assumptions.
I also note you been around asking about using using Access in various Access forums for what about two years or even more now? You seem to be looking for some magical silver bullet that will solve your problems. There is not one!
At the end of the day you need a sit down and obtain some basic Access skills or hire someone. You need to learn the system you going to use and THEN come up wth a desing that works with Access and "is" for Access.
I should point out that the desing you choose for Access would not necessary say work with vb.net either. So don't try and take an existing desing and put a square peg in a round hole. What works in one system in terms of UI will NOT work in other systems.
I think you been fooling around much too long here. About the only good part of delaying so long is you now have to consider if you want to adopt a set of tools that allows some web integration with the application?
Office 365 works great with Access web publishing, but for complex mature applications, I think the web side is somewhat weak (but you can write hybrid applications with Access 2010 now). Here is an video of Access web in Action:
http://www.youtube.com/watch?v=AU4mH0jPntI
In the above, I swtich to running the applicaiton in a web browser. This app was 100% built in Access, including table trigger logic and code (no 3rd party tools or coding system was use - ONLY Access was used).
Considering technology right now, perhaps you have this run on iPads as they walk around in the store? There a lot of new choices here for software, but if you sit around for another 2-3 years, then you be looking to use something other then the iPad and some "other" new system.
You can certainly write your applications as more "touch" friendly in Access. However some of the new gesture based actions do not transfer to web. For example, we cannot disable keyboard input in a combo box, and this ability would REALLY help the Access application running on an iPad if we could. The reason being is when we tap or hit a combo box on iPad then it pops up on the screen the soft keyboard and we do NOT want this. And some of a really slick gesture date pickers etc. don't translate to web on the iPad (they do want you to write native apps after all!).
Microsoft Access is pretty simple as it is. There are wizards for form and report creation. You can then modify the events to perform all of the tasks you mention.
I suggest getting a good book on the subject and delving in. I learned a lot (years ago) from the Developer's Handbook series including Access Developer's Handbook and VBA Developer's Handbook.
It appears you've already made the decision to migrate your entire system to MS Access but have you ever thought about first skinning your IDS system with a platform like Grails? It's cross platform and can be deployed on any operating system that supports Java.
You could deploy the resulting application as a single user, single site or shared system depending upon the client's requirements.
Once you have migrated and enhanced all of the existing functionality it will then be trivial to convert the back-end database to another engine such as PostgreSQL.
I'm currently in the process of enhancing a legacy Informix 7 (SE/ACE/4GL) application for a client and it is working out really well.
I have to develop a database for a unique environment. I don't have experience with database design and could use everybody's wisdom.
My group is designing a database for piece of physics hardware and a data acquisition system. We need a system that will store all the hardware configuration parameters, and track the changes to these parameters as they are changed by the user.
The setup:
We have nearly 200 detectors and roughly 40 parameters associated with each detector. Of these 40 parameters, we expect only a few to change during the course of the experiment. Most parameters associated with a single detector are static.
We collect data for this experiment in timed runs. During these runs, the parameters loaded into the hardware must not change, although we should be able to edit the database at any time to prepare for the next run. The current plan:
The database will provide the difference between the current parameters and the parameters used during last run.
At the start of a new run, the most recent database changes be loaded into hardware.
The settings used for the upcoming run must be tagged with a run number and the current date and time. This is essential. I need a run-by-run history of the experimental setup.
There will be several different clients that both read and write to the database. Although changes to the database will be infrequent, I cannot guarantee that the changes won't happen concurrently.
Must be robust and non-corruptible. The configuration of the experimental system depends on the hardware. Any breakdown of the database would prevent data acquisition, and our time is expensive. Database backups?
My current plan is to implement the above requirements using a sqlite database, although I am unsure if it can support all my requirements. Is there any other technology I should look into? Has anybody done something similar? I am willing to learn any technology, as long as it's mature.
Tips and advice are welcome.
Thank you,
Sean
Update 1:
Database access:
There are three lite applications that can write and read to the database and one application that can only read.
The applications with write access are responsible for setting a non-overlapping subset of the hardware parameters. To be specific, we have one application (of which there may be multiple copies) which sets the high voltage, one application which sets the remainder of the hardware parameters which may change during the experiment, and one GUI which sets the remainder of the parameters which are nearly static and are only essential for the proper reconstruction of the data.
The program with read access only is our data analysis software. It needs access to nearly all of the parameters in the database to properly format the incoming data into something we can analyze properly. The number of connections to the database should be >10.
Backups:
Another setup at our lab dumps an xml file every run. Even though I don't think xml is appropriate, I was planning to back up the system every run, just in case.
Some basic things about the design; you should make sure that you don't delete data from any tables; keep track of the most recent data (probably best with most recent updated datetime); when the data value changes, though, don't delete the old data. When a run is initiated, tag every table used with the Run ID (in another column); this way, you maintain full historical record about every setting, and can pin exactly what the state used at a given run was.
Ask around of your colleagues.
You don't say what kind of physics you're doing, or how big the working group is, but in my discipline (particle physics) there is a deep repository of experience putting up and running just this type of systems (we call it "slow controls" and similar). There is a pretty good chance that someone you work with has either done this or knows someone who has. There may be a detailed description of the last time out in someone's thesis.
I don't personally have much to do with this, but I do know this: one common feature is to have no-delete-no-overwrite design. You can only add data, never remove it. This preserves your chances of figuring out what really happened in the case of trouble
Perhaps I should explain a little more. While this is an important task and has to be done right, it is not really related to physics, so you can't look it up on Spires or on arXive.org. No one writes papers on the design and implementation of medium sized slow controls databases. But they do sometimes put it in their dissertations. The easiest way to find a pointer really is to ask a bunch of people around the lab.
This is not a particularly large database by the sounds of things. So you might be able to get away with using Oracle's free database which will give you all kinds of great flexibility with journaling (not sure if that is an actual word) and administration.
Your mentioning of 'non-corruptible' right after you say "There will be several different clients that both read and write to the database" raises a red flag for me. Are you planning on creating some sort of application that has a interface for this? Or were you planning on direct access to the db via a tool like TOAD?
In order to preserve your data integrity you will need to get really strict on your permissions. I would only allow one (and a backup) person to have admin rights with the ability to do the data manipulation outside the GUI (which will make your life easier).
Backups? Yes, absolutely! Not only should you do daily, weekly and monthly backups you should do full and incremental. Also, test your backup images often to confirm they are in fact working.
As for the data structure I would need much greater detail in what you are trying to store and how you would access it. But from what you have put here I would say you need the following tables (to begin with):
Detectors
Parameters
Detector_Parameters
Some additional notes:
Since you will be doing so many changes I recommend using a version control like SVN to keep track of all your DDLs etc. I would also recommend using something like bugzilla for bug tracking (if needed) and using google docs for team document management.
Hope that helps.
Is it good practice to delegate data validation entirely to the database engine constraints?
Validating data from the application doesn't prevent invalid insertion from another software (possibly written in another language by another team). Using database constraints you reduce the points where you need to worry about invalid input data.
If you put validation both in database and application, maintenance becomes boring, because you have to update code for who knows how many applications, increasing the probability of human errors.
I just don't see this being done very much, looking at code from free software projects.
Validate at input time. Validate again before you put it in the database. And have database constraints to prevent bad input. And you can bet in spite of all that, bad data will still get into your database, so validate it again when you use it.
It seems like every day some web app gets hacked because they did all their validation in the form or worse, using Javascript, and people found a way to bypass it. You've got to guard against that.
Paranoid? Me? No, just experienced.
It's best to, where possible, have your validation rules specified in your database and use or write a framework that makes those rules bubble up into your front end. ASP.NET Dynamic Data helps with this and there are some commercial libraries out there that make it even easier.
This can be done both for simple input validation (like numbers or dates) and related data like that constrained by foreign keys.
In summary, the idea is to define the rules in one place (the database most the time) and have code in other layers that will enforce those rules.
The disadvantage to leaving the logic to the database is then you increase the load on that particular server. Web and application servers are comparatively easy to scale outward, but a database requires special techniques. As a general rule, it's a good idea to put as much of the computational logic into the application layer and keep the interaction with the database as simple as possible.
With that said, it is possible that your application may not need to worry about such heavy scalability issues. If you are certain that database server load will not be a problem for the foreseeable future, then go ahead and put the constraints on the database. You are quite correct that this improves the organization and simplicity of your system as a whole by keeping validation logic in a central location.
There are other concerns than just SQL injection with input. You should take the most defensive stance possible whenever accepting user input. For example, a user might be able to enter a link to an image into a textbox, which is actually a PHP script that runs something nasty.
If you design your application well, you should not have to laboriously check all input. For example, you could use a Forms API which takes care of most of the work for you, and a database layer which does much the same.
This is a good resource for basic checking of vulnerabilities:
http://ha.ckers.org/xss.html
It's far too late by the time the data gets to your database to provide meaningful validation for your users and applications. You don't want your database doing all the validation since that'll slow things down pretty good, and the database doesn't express the logic as clearly. Similarly, as you grow you'll be writing more application level transactions to complement your database transactions.
I would say it's potentially a bad practice, depending on what happens when the query fails. For example, if your database could throw an error that was intelligently handled by an application, then you might be ok.
On the other hand, if you don't put any validation in your app, you might not have any bad data, but you may have users thinking they entered stuff that doesn't get saved.
Implement as much data validation as you can at the database end without compromising other goals. For example, if speed is an issue, you may want to consider not using foreign keys, etc. Furthermore, some data validation can only be performed on the application side, e.g., ensuring that email addresses have valid domains.
Another disadvantage to doing data validation from the database is that often you dont validate the same way in every case. In fact, it often depends on application logic (user roles), and sometimes you might want to bypass validation altogether (cron jobs and maintenance scripts).
I've found that doing validation in the application, rather than in the database, works well. Of course then, all the interaction needs to go through your application. If you have other applications that work with your data, your application will need to support some sort of API (hopefully REST).
I don't think there is one right answer, it depends on your use.
If you are going to have a very heavily used system, with the potential that the database performance might become a bottleneck, then you might want to move the responsibility for validation to the front-end where it is easier to scale with multiple servers.
If you have multiple applications interacting with the database, then you might not want to replicate and maintain the validation rules across multiple applications, so then the database might be the better place.
You might want a slicker input screen that doesn't just hit the user with validation warnings when they try to save a record, maybe you want to validate a field after data has been entered and it losses focus; or even as the user types, changing the font colour as validation fails/passes.
Also related to constraints, is warnings of suspect data. In my application I have hard-constraints in the database (e.g. someone can't start a job before their date of birth), but then in the front-end have warnings for data that is possibly correct, but suspect (e.g. an eight year-old starting a job).