I just inherited a relatively small SQL Server database. We have a decentralized system operating on about ten sites, with each site being pounded all day by between sixty and one hundred clients. Upon inspecting the system, a couple of things jumped out at me: there are no maintenance plans or keys defined.
I have dozens of different applications that are already accessing the database. The majority of them are written in C with inline SQL. Part of what I was brought in to do was write stored procedures for everything and have our applications move to that. Before I do this, however, I really think I should be focusing on these seemingly glaring issues.
Also, we'll eventually be looking into replication to a central site, so I really think these things should be addressed before we even think of that.
Figuring out a redesign scheme and maintenance plan will be time-consuming but not problematic - I've done it before at single sites. But, how am I going to go about implementing these major changes to the database across ten (or more) production sites while ensuring data integrity and not breaking the applications?
I would suspect that with no keys officaly defined, that this database probaly has tons of data integrity problems. Lucky you.
For replication you will need GUIDs. I would do this, Add the GUIDs and PK definitions in the dev environment and test test test. You'll prbably find alot of crap where people did select * and adding the columns will cause probnalem or cause things to show up on reports that you don't want. Find and fix all these things. Be sure to script allthe changes to the data and put them in source control along with any code changes you need to make to the application. Then schedule down time for maintenance of the database during the lowest usage hours. Let the users know the application will be down ahead of time. During the down time, have the application show a down message, change the datbase to single user mode so no one except the team making this change can affect the database, make a fullbackup, run the scripts to make the changes to the database, run the code to change the application, test, take the database out of single user mode and turn the application back on.
Under no circumstances would I try to make a change this major without going to single user mode.
First ensure you have valid backups of every db, and test-restore them to make sure they restore OK.
Consider using Ola Hallengren's maintenance vs. Maintenance Plans if you need to deploy identical, consistent, scripted solutions to all your sites (Ola Hallengren's site)
Then I'd say look at getting some basic indexing in place, starting with heavy-hitter tables first. You can identify them with various methods - presume you know how, but just to throw a few out thoughts: code review, SQL Trace, Query Plan analysis, and then there are 3rd party tools e.g., Idera SQLdm, Confio Ignite, Quest's Spotlight on SQL Server or Foglight Performance Analysis for SQL Server.
I think this will get you rolling.
Some additional ideas.
One of the first thing's I'd check is: are all the database instances alike, as far as database objects are concerned? Do they all have the exact same tables, columns (and their order in the tables), nullability, etc. etc. Be sure to check pretty much everything listed in sys.objects. Once you know that the database structures are all in synch, then you know that any database modification scripts you generate will work on all the instances.
Once you modify your test environment with your planned changes, you have to ensure that they don't break existing functionality. Can you accurately emulate "...being pounded all day by between sixty and one hundred clients" on your test environment? If you can't, then you of course cannot know if your changes will break anything until they go live. (An assumption I'd avoid: just because a given instance has no duplicates in the columns you wish to build a primary key on does not mean that there are never any duplicates present...)
I went for a job interview this morning, and one of the scenarios that the IT manager (a former programmer) told me about was the following .... If anyone could give me any pointers that I can email him pointers to tomorrow (Wednesday 22nd) it may just help tip his decision in my direction. So there's beer in it if you answer a relevent answer, and I get the job.
Old systems using Access 2 databases. These systems are not due to be updated any time soon.
New systems need access to that data, so each night he exports information using SSIS. However, he needs to also write data back, and more often than just overnight.
However, when he connects to the database, using ODBC connections I think, he says that the app (SSIS) opens the tables / database exclusively, even if he selects the option NOT to open it exclusively.
Any ideas, suggestions or comments ?
Could it be that the Access db itself is set up to open exclusively. Gosh it's been about a million years since I used Access 2.
Another possibility to fix is to consider making the SQl server tables that mimic the access tables and making them linked tables in access and using them instead of access. I can't remember if you can do this with Access 2 or not. If you can do this, then you will be making changes directly to the sql servver tables and not locking them up in Access 2. It's at least something to try. The use of linked tables should't break anything in Access if they have the same names and all tables havea primary key. ANd then you can performance tune as well if there are problems.
I'm throwing out ideas here, up to you to research the feasibilty before sending them on. But honestly, do you really really want to work for an organization that still uses Access2? If they haven't upgraded that, what else are they years out of date on? Access2 came out in 1993. So they haven't upgraded it in 17 years, scary stuff.
Please, give me the most serious arguments against this.
Application directly opens a connection to ms sql server, directly executes queries.
So what I'd like to ask:
1) Why it is wrong when the number of users can be up to 1000 executing huge queries?
2) What serious problems can that cause?
3) What should I do?:)
Arguments, the most serious arguments against this kind of implementation!
One of the things to consider is how the queries are done. 1000 queries against a SQL Server DB might be manageable, but 1000 Access queries in which the table is locked, or which are actually joins or views, could use dramatically more memory. It really depends on how the application is written. Some Access apps open a recordset and page through the records one at a time, or fetch a few dozen and work on those, but sometimes Access grabs the whole recordset, for example to allow users to page through data. And I have seen Access lock a set of tables to allow editing of them. That would be bad in your scenario.
Of course, I wholeheartedly agree with the "10 years out of support" issue. That is a guaranteed problem. Mine is only a possibility. And you should probably update SQL Server to a current version also, for the same reason.
What about:
The version of access 97 is totally outdated and wont get any updates, has a crappy look, crappy functionality and in general - IF it requires rework - should be updated.
Problems? You run on a 10 year old out of support platform. What problem can that cause? Well - what about limited support?
Upgrade at least to 2007, better 2010 (coming in a couple of weeks) when you have a momnent time. I personally dispise access based applications (crappy architecture to start with etc.), but if there is one, the update to access 2010 is possible the most painless way to go.
Access 2003 or 2007 would be just fine for the scenario as long as you had an Access developer who was up to speed on how to develop for client/server with large user populations.
Access 97 is still an awfully nice version of Access. I think it's the best version ever produced.
But it is out of support and predates the alteration of default permissions in Windows implemented with the release of Windows 2000. This means that it has some problems in installing with its default permissions (it expects write access to its application folders and registry keys). An installation script can easily alter these appropriately, but you're still left with problems in certain contexts, like trying to run it in Windows Terminal Server/Citrix, where it very often just completely breaks.
I would like to hear an explanation of exactly why someone would choose A97 for new development. Of course, I may be misinterpreting. You may be asking about an existing app, in which case I'd go with "if it ain't broke, don't fix it," and then ask exactly what it is that is perceived as "broken." Those things can be fixed, though it's unlikely that simply upgrading from A97 to something more recent is going to do the job.
I’m currently nearly finished with a brand new application written in access 97 that stores its data in SQL server 2008. As has been said many times before the access/SQL server combination really works great.
Inline with my other applications it is completely unbound using ADO to get the data from the server. I wont drag up that debate again here but it is something you really want to look into as it can offer some great benefits.
Most of the SQL server guide you will find will ask you to check that you have the correct indexes and try to identify the slowest running parts of the system or the ones that get called a lot and then look at making them faster. That might cause you to make a covering index or to denormalise the data in someway.
Generally what is good practice for JET also works well for SQL server, make a good table schema with a good clustered index choice and good supporting indexes and you are 95% of the way there
Im not a developer (not a proper one) but always up for an excuse for self-development.
I work in a support team for a reputable software vendor, and we currently use a helpdesk piece of software called iSupport.
- Its not a bad piece of kit, and im not sure if it has been set up badly (trying to find out) but the biggest problem I face (being on the front line as an analyst who uses it 8hrs a day) is its inability to easily search.
A new 'incident' will come in. Client will report some errors in a log, perhaps mention some other keywords when describing his symptoms.
Now I know, I have probably answered a similar problem before (but cant remember the solution) or even more likely, a fellow-analyst may have answered the same question before.
I would like to have the ability to have one search box (think Google) that searches through EVERY incident that has ever been created and return me ALL incidents that contain that keyword.
At the moment the search is very poor - You can take time to set up searches and specify which fields you want to search on, which values to filter by (perhaps by an analyst or category, etc) but this takes time and more often than not, it returns poor results and it would have been easier to try and track it down yourself manually.
All of the data sits in underlying SQL Server tables (have requested a subset of the data).
What im thinking, is creating a separate front-end that is just a basic search box and thats it. This app will point to all the relevant fields in the SQL tables and pull out the relevant records into a table. Once I have the ID for the incident, it is then a simple job to pull out that incident back in the iSupport front end.
I was thinking along the lines of Google Desktop style app (shortcut key brings up the search box).
Now thats as much thinking as ive done. Looking for some advice on where to go next.
I know for instance, that Google Desktop crawls and indexes all your physical files on your machine. Would I have to do something similar for a database as I imagine there maybe a large number of records/fields/tables to search through.
TBH, if it works, im not that fussed (to begin with) if it takes awhile to process the query, as long as it returns relevant results. But ideally id like it to be quick.
Ill leave it at that for now.
Where should I begin?
If .NET is your thing, then Lucene.NET will work well with SQL Server to give you that google search feeling.
The StackOverflow websites use it, you can listen to the SO Podcast where Jeff/Joel bitch about why SQL Server full-text search sucks so much.
I'd suggest this might be a good candidate for a web application - an asp.net / jsf website. This means that you can control it from one machine, but all your colleagues can make use of it without a deployment headache every time you add a new feature...
The incident database is mission critical (critical to your relationship with customers), so if you came to me with this request I would insist that you accessed the database through a user that had select permissions to the appropriate tables, and very little else. This from your point of view is a good thing too - let's you operate knowing you're not going to cock anything up...
The SSMS Tool Pack (an add-in to Management Studio) contains a feature to search Table, View or Database Data.
Have a look into the Full-Text Search functionality of SQL Server.
iSupport includes a full-text search function. Add a 'Global Search' widget to a dashboard you create.
I am having a real problem at work with a highly ingrained developer obsessed with ms access. Users moan about random crashes, locking errors, freeze's, the application slowing down (especially in 2007) but seem to be very resistant to moving it. Most of the time they blame the computer and can't be convinced it's the fact its a mdb sat on a network drive and nothing to do with the hardware sat in front of them which is brand new.
There is a front end vb program hanging off it but I don't think it would take more than a couple of weeks to adjust, infact I would probably re-write it as it has year on year messy code from a previous developer.
What are my best arguments to convince them we need to move it?
Does anyone else have similar problems with developers stuck in their ways?
how about the random, crashes, locking errors, freeze's, slow downs (sic).
A quick search on the web finds some useful materials:
Best Practices When Using Microsoft Office Access 2003 in a Multi-user Environment - if the changes here can't be implemented, or would effectively take a rewrite, then that is good ammunition for doing it right.
SQL Server vs MS Access - pay special attention to feature limits. Eg You can only have 32,000 objects in an access DB. Caveat: though it says 255 concurrent users, and that is probably a technical limitation, the practical limitation is really MUCH lower.
It's hard to convince people that are not willing to learn and are not open to new ideas. You can go on about speed issues, concurrency issues, security problems.. but ultimately, some people will just never listen. Go over their heads. Rewrite it in tools from this decade and show them up. Refuse to be involved with the project and further. I don't know what the political situation is, but technically, MS access is wrong for what you are doing, from what you've described.
come in on a weekend, copy the database to sql server, change the app's connect-strings to sql server, retest the application, then uninstall ms-access...everywhere.
then don't say anything about it, let him think that the problems 'fixed themselves' and that the users are still using ms-access
To me it depends on how many concurrent users you have and how big the database is. If you have more than 5 concurrent users then you should be thinking about a database server. The network traffic starts to get out of hand and with each concurrent user you add it just gets worse.
I have created reliable access based systems for years. If you are having random crashes, locking issues, and slow downs then you aren't doing something right. I typically will have an mda local with the mdb on the network when creating an app in access. To have good performance it's key to have the proper indexes and queries optimized for getting just the data you need. Whether using a separate app, access, or some app running against sql server you need to actively handle record locking properly. You can't just blindly let access lock your records.
Forget the arguments about DB Size, it is an uninformed reason to shift to a client-server platform in 90% of the cases I hear it brought up.
Your best arguments are based on features explained at a low tech level:
(1) You can backup and perform maintenance on the DB without kicking out the users (which introduces costly downtime).
(2) Faster recovery if data is accidentally deleted/mangled or corrupted. Again, less risk and less downtime. This is always a good foundation for a business case.
(3) If (and only if) you anticipate the need to scale quite a bit, the upgrade will better allow that.
(4) If you need to run automated jobs/updates, SQL can do this much more elegantly.
Remember the contra-indications for SQL, it is easy to get on your technical high-horse about this platform versus that, but you have to balance the benefits against the costs.
SQL is a Helluva lot more expensive to maintain as it requires dedicated hardware, expensive licenses (Server OS and DB) and usually at least a part time DBA that is going to cost you a bare minimum of $75K (if you get luck AND work out of Podunk Iowa).
The best possible advice I can give you is to make sure that you have a good attitude and are known as someone who does quality work and gets things done. It sounds like you don't have any control in the situation so what you need is influence.
Find a way to solve a problem (probably a different one that is less threatening to the people involved) in the way you are suggesting. Make it work blindingly fast and flawlessly. Make it work so well that people start asking for you when they need something done. Get it done quickly, which you should be able to do because you'll be using the right tools for the job.
Be a good person to work with, not the PITA that knows how everyone else should write their code. Be able to give an answer for what you might do differently and why, but don't automatically assume that your ideas are always the best. Maybe there are trade-offs that you don't know about -- no money in the budget for the extra CALs, we have this other app that needs to be done first. This doesn't sound like your situation, but looking for opportunities to understand before making constructive criticisms can go a long way to helping people be receptive.
The other thing is that this probably has nothing to do with the technical aspects of the situation and everything to do with the insecurities of the other developer. "This is all I know. If we change it, I won't understand it and then where will I be." Look for ways to help the other guy grow -- when he's having a problem, find resources that will help him develop good technical solutions. Suggest that everyone in your department get some training in new technologies. Who knows, one good SQL Server course and the guy could become the SQL Server evangelist in the organization because now THAT'S what he knows.
Lastly, know when to cut your losses, so to speak. If you find that you're not able to do anything about the situation, don't add to the complaining. Move on to something that you can control and do it as well as you can. Maybe in the future you'll be in a position that you do have control or influence in the situation and can do something about it. If you find that you're in a company that's more dysfunctional than most, find a way to move on to a place where the environment is better.
It is possible, and actually fairly easy, to convert an Access database to having the tables/views in SQL Server while still using the Access app as a front-end.
From there, your Access-obsessed developer can still have fun with all that VBA code. Meanwhile, on the back-end, you add indexes and such to speed everything up. Maybe someday you get lucky, and he asks about stored procedures. Then, the app is just a front-end, and who cares what it's written in? Your data is safe in SQL Server.
It is possible for you to do this yourself, but just leave the production app ALOOOOOOOOOOOONE. Take a copy, and convert that copy. Then, host it for a couple of users to TEST drive .. make your version of the Access app show "TEST APP" in big red letters. If your developer asks what you're doing, you can say the truth -- you are testing to see if converting only the tables/views might be of some help to the overall app.
This way, you get the best of both worlds, keep your developer happy, make the users happier (hopefully), and if you play it right, your bosses will know that you handled a knotty personnel issue with your technological prowess and your maturity.
I once had similar problems with someone I would not hesistate to call a complete idiot.
It was not possible to convince them of the issues with access. In the end it was easier to force the issue than do it "nicely", cruel to be kind.
If they resist then you can always go above their head. Management must be aware of crashes and stability related issues. Present a plan to them to improve stability and they are likely to at least listen. They will probably then want a meeting with all developers to discuss so go into it armed with plenty of ammo.
More than "How to convince them", let's talk about "How to do it without anybody noticing"!
First of all I advice you not to mix together the code optimisation issue and the SQL server one. Do not give users a chance to complain about SQL while bugs are related to something else.
If your code is really unbearable, rewrite the app before switching to SQL, keeping in mind the following points to make the final transition to SQL Server completely transparent for final users.
This is what we did 18 months ago, and I am sure we still have users thinking our database is Access:
Export current access database to SQL through available Wizard in access for testing purposes (many problems might occur, and you could need another tool such as the one proposed here).
Create a unique connection object at the application level, so that you can freely switch from Access to SQL at any time (at development level, you can even add an input box at startup to ask which connection to use). We chose an ADODB connection object, but it will also work with ODBC connection.
In case you use SQL syntax to update tables, make sure that all SELECTs, INSERTs, UPDATEs and DELETEs make use of this connection. In case you use recordset, make sure that all of them use this connection at opening time.
When needed, update all connexion specific code by adding a "SELECT CASE" type_Of_TheConnexion options
Switch to SQL connection ..and debug till you're done!
The problems you will find are mainly linked to SQL syntax, where MSSQL uses ' instead of " and # as separators. Date format is also an issue, where standard SQL format is 'YYYYMMDD' while MS-Access format depends on computer locals (beware of conversions from date to string!) and is stored as "YYYY-MM-DD" (if I remember ...). Boolean in SQL are 0 and 1, while they are True/False or 0/-1 in Access ...
Test, update code, and when you are ok, make a new data transfer, lock your app on the SQL connection, and distribute a new runtime.
It depends on the type of application and data load of your database but Access is quite efficient, even over the network.
Depending on the amount of data your users deal with you could easily scale up to a 100 users on a network just using a from and back-end Access database.
Looks like in your case a rewrite may be in order. If your application is data-centric if doesn't make much sense to develop it in VB6: the tools given by Access are much better than anything you'd be able to make, especially when considering Access 2007.
Upsizing to SQL Server is only really required if you're getting into issues of:
Security:
you need to make sure that only the rights users can access data. You can do your own security in Access, but it's never going to be as strong as SQL Server.
Scalability:
you're dealing with lots of data and complex queries or a lot of users and it would be better to have dedicated hardware to handle the load for the clients. The issue with this though is that while removing the pressure from the less-capable clients machines, you're adding a lot more to the server.
Integrity:
With the back-end database being just a file that needs R/W access for all connected clients, there's always the possibility that someone is going to do something bad or that a client may crash and leave the database corrupted.
If your number of users is average (I'd say 30), then there's probably no real need to upscale:
Use MS Access 2007 to develop your application, then just use the MS Access 2007 Runtime (it's free!) on all client machines to get a more modern user interface (uses the Ribbon and has lots of UI enhancements over previous versions).
You can't be the cheapness of that solution : you only need full retail version of MS Access and all the rest is free, regardess of the number of users!
Don't think that moving to SQL Server is going to improve performance of your queries: MS Access often does a better job of optimizing the queries for you (it knows what needs to be displayed and does lots of caching and optimization).
Make sure you only edit small amounts of data at any one time (don't use dynaset queries just to display vast amounts of data in a datasheet; use a snapshot instead and open a detail form that only contains the data to edit when necessary.
Cache complex queries locally.
Built some caching mechanism that leaves a copy of the results of a complex query on the local machine. The gain in performance is pretty amazing and if the query doens't change much (for instance a log of stock operations) you can just persist the complex/big query locally and append new records as necessary.
There is so much more to say.
Bottom line is: you may be looking at a rewrite, but don't dismiss Access as the solution because your current application was poorly written.
Try bechmarking and showing the stats to him
Making people change can sometimes be a real pain in the butt.
I would have to say the main argument would have stability and speed, but of course like you have said they already know this a still won't move.
Another thing to try would be to show them the power of LINQ to SQL and how much cleaner it would make your application. Like Daniel Silveira said you could try and throw a couple of stats there way and see if they are convinced.
We have a app build using MS access as a back end and I can't wait till we get our new SQL server so I can move everything to that.
You could show him the perf results comparing the two, but if he's really set in his ways and refuses to change, there isn't much you can do except force him somehow.
If you're his boss then just force him to change it to use SQL. If not, then convince your boss to force the change by showing him the perf results and explain it'll fix the issues you're having.
Errr, leave the team? You seem to be working with the totally wrong set of people. Now, if the team IS your company, then you are working with the wrong company.
Of course once you leave the company, you could tell your clients that you could solve the network problems on their own and make them leave the company as well. Then give them an improved system that works on SQL Server Express.