First of all, I'm not a DB expert .. so I'm not sure if my terminology or anything is correct here, but if you can bear with me I hope you'll get the idea.
I have a SQL Azure database which powers a social network. Millions of transactions occur every day from very simple ones to complex SELECTS which sort through tens of thousands of users based on distance etc.
My user numbers grow daily and I know (believe) that at some point I'll need to implement sharding, or make use of SQL Azure Federation to keep my app up and running due to 1 SQL Azure having limited resources ... but my question is, how do I figure out when I'm going to need to do this?
I know that when I start to use too much resources, my queries will be throttled ... but for all I know this could start happening tomorrow, or be years away.
If I know I'm hitting 80% of what I'm allowed to, then I know I need to prioritise a solution to help me scale things out, but if I'm only using 10% then I know I can put this on the back-burner and deal with it later.
I can't find any way, or even mention, of how to measure this?
Any suggestions?
Thanks,
Steven
I don't know of any inbuilt way to measure this. If somebody does then I would be very interested to hear about it.
However there is a great library from the Microsoft AppFabric CAT Best Practices Team which is a transient fault handling framework. See here
This does a number of things including handling retry logic for opening connections and running queries. You could use that but extend it slightly to log when you were being throttled by SQL Azure.
This probably wont give you as much warning as you want, but will help you know when you are getting closer to the limit. If you combined this approach together with some kind of application / database stress testing then you can find your limits now before your real usage gets there.
Based on the numbers you have given I would definitely start looking at sharding now.
I would recommend you read the article below if you haven't done so; it contains interesting information about the underlying reasons for SQL Azure throttling conditions. Understanding what is being monitored for throttling can help you figure out why your database is being throttled.
Technet Article: SQL Azure Connection Management
Thank you for mentioning the Enzo library by the way (disclaimer: I wrote it)!
However understanding the reason of the throttling would be my first recommendation because depending on the reason, sharding may or may not help you. For example, if the issue of the thottling is excessive locks, sharding may indeed reduce locks to a single database, but it could come back and bite you at a later time.
Thank you
Herve
best practices to fight throttling 1) keep queries as short as possible 2) run workloads in batches 3) employ retry mechanisms.
I would also like to point you to couple of resources.
1) sql azure throttling and decoding reason codes: http://msdn.microsoft.com/en-us/library/ff394106.aspx#throttling
2) http://geekswithblogs.net/hroggero/archive/2011/05/26/cloud-lesson-learned-exponential-backoff.aspx
I've got 10 GB of files to back up daily to another site. The client is way out in the country so bandwidth is an issue. Does anyone know of any existing software or libraries out there that help with keeping a folder with its files synchronized across a slow link, that is it only sends files across if they have changed? Some kind of hash checking would be nice, too, to at least confirm the two sides are the same.
I don't mind paying some money for it, seeing as how it might take me several weeks to a month to implement something decent on my own. I just don't want to re-invent the wheel, here. BTW it is a windows shop (they have an in house windows IT guy) so windows is preferred.
I also have 10 GB of SQL Server 2000 databases to go across. Is the SQL server replication mode reliable?
Thanks!
Look into rysnc.
This might be what you're looking for seeing as you're an MS shop
Microsoft Sync Framework File Synchronization
Two more suggestions are Robocopy and XXCopy.
This is something I would like to see while doing my day today programming works, But I've never seen such application yet. You input is highly appreciated.
Lets say we have an application that needs MSSQL server as DBMS. And suppose you just need to install it and do something. (i.e You are not going to deply it in production servers etc.)
In such a case it might be an overhead, of installing MSSQL first. I am suggesing something like a software bridge that can use another DBMS to store data. In other words the application "sees" an MSSQL instance but underneath that it might be Access. The bridge sholud do some sort of conversion.
Another example : You have MSSQL but a certain application needs Oracle. You have to purchase Oracle then. But with a something like a bridge, You can put information into your MSSQL DBMS. The bridge listens to port 1521 like Oracle so application "Thinks" there is an oracle installation.
Is it an idea that cannot be implemented?
Are there any such applications?
If so what are they?
Thanks... :)
Adding a Clarification : The application might be from a third party. You don't have any knowledge on internal architecture of that. you just know it uses a certain DBMS. I am trying to use a different DBMS other than the third party software needs.
Application usually don't depend on a specific database server, OR they depend on it for a reason.
If an application asks for oracle, or sql server, or whatever, it's because it relies on the implementation details of this specific vendor to run its SQL, its stored procedures, etc. There's no way you could emulate that with an access database, for example...
If your application just needs to run some very simple SQL (ie basic insert/select statements), it probably uses a standard driver (odbc, ado, etc.), and those drivers can accommodate every major sql database engines. In my experience, "simple applications" don't ask for a specific database vendor.
This is the problem that ODBC was supposed to solve :-) .
But in response to your questions:
Is it an idea that cannot be implemented?
It can be implemented.
It would be tedious and thankless work, and you would have a very limited audiance. In my opinion it's not worth doing.
Are there any such applications?
None that I know of.
If so what are they?
None that I know of.
......
Bringing in Chandrasekar's note in the comments section:
Have a look in a super user's perspective... He has a nice application but he can't use it without some DBMS. But still he is not a programmer to do something. So they need such a product
I agree it has applications, but it has a very limited audiance :) .
What you're proposing is something like the firefox plugin 'ietab', Only you won't have ie installed... so instead of embedding ie, you would need to entirely re-implement ie using firefox's rendering window.
Just my opinion : that's too much effort... It's simpler to just install a second database.
If this application uses ADO to connect to SQL Server and you can modify the connection string, then it's quite easy to use a different database: change the connection string! However, the other database must be able to support all features of SQL Server. Besides, the software was never tested on another database so the application might Crash & Burn.
If you can't change the connection string, or the application doesn't use ADO, things are more complicated and very close to impossible.
I've worked in the past on a project that needed to be reasonable database-independent. The database had to support stored procedures but there weren't any other restrictions. By default, we tried to support both SQL Server and Oracle. (We also supported Interbase but never advertised this.) While we did manage to keep it mostly database-independent, we did have to work around quite a few minor issues. Especially joins in our queries had some nasty problems which we just solved by adding more logic to stored procedures.
"This is the problem that ODBC was supposed to solve :-) ."
And it is the very same problem that SQL was intended to solve too.
It seems to me that the reason why this problem exists is that the world seemingly fails to agree sufficiently on what the data manipulation language/interface ought to look like.
I suspect that if this were solvable, it would already have been done.
The closest I've heard is EnterpriseDB where they have built a layer on top of Postgres so it looks more like Oracle.
But remember these databases have features covered by patents and copyright so there's a limit on how closely a competitor product can imitate the real thing.
It would probably be easier to imitate 'down' than up. For example, MS-Access wouldn't be able to imitate much of the functionality for Oracle or SQL Server, whereas there's a much better chance of SQL Server imitating a simpler DB like Access.
Applications usually DO depend on a specific database server. Every database implements things slightly differently - even MSSQL and Sybase, which have a common ancestor.
Any bridge, however well it attempts to abstract the differences, would leave some exposed. These would be likely to create subtle bugs in the application, which might appear initially to work, but then fail, or worse, corrupt data.
Moreover, the application vendor would not support you in such a case - they'd simply say they don't support that use case, and you should remove the bridge and install a proper instance of whatever database it was intended for.
In short, I don't think it's worth the risk of the application malfunctioning subtly, and being left without support, even if the application isn't especially important. If you dislike the underlying database the application uses, choose a different application.
I am having a real problem at work with a highly ingrained developer obsessed with ms access. Users moan about random crashes, locking errors, freeze's, the application slowing down (especially in 2007) but seem to be very resistant to moving it. Most of the time they blame the computer and can't be convinced it's the fact its a mdb sat on a network drive and nothing to do with the hardware sat in front of them which is brand new.
There is a front end vb program hanging off it but I don't think it would take more than a couple of weeks to adjust, infact I would probably re-write it as it has year on year messy code from a previous developer.
What are my best arguments to convince them we need to move it?
Does anyone else have similar problems with developers stuck in their ways?
how about the random, crashes, locking errors, freeze's, slow downs (sic).
A quick search on the web finds some useful materials:
Best Practices When Using Microsoft Office Access 2003 in a Multi-user Environment - if the changes here can't be implemented, or would effectively take a rewrite, then that is good ammunition for doing it right.
SQL Server vs MS Access - pay special attention to feature limits. Eg You can only have 32,000 objects in an access DB. Caveat: though it says 255 concurrent users, and that is probably a technical limitation, the practical limitation is really MUCH lower.
It's hard to convince people that are not willing to learn and are not open to new ideas. You can go on about speed issues, concurrency issues, security problems.. but ultimately, some people will just never listen. Go over their heads. Rewrite it in tools from this decade and show them up. Refuse to be involved with the project and further. I don't know what the political situation is, but technically, MS access is wrong for what you are doing, from what you've described.
come in on a weekend, copy the database to sql server, change the app's connect-strings to sql server, retest the application, then uninstall ms-access...everywhere.
then don't say anything about it, let him think that the problems 'fixed themselves' and that the users are still using ms-access
To me it depends on how many concurrent users you have and how big the database is. If you have more than 5 concurrent users then you should be thinking about a database server. The network traffic starts to get out of hand and with each concurrent user you add it just gets worse.
I have created reliable access based systems for years. If you are having random crashes, locking issues, and slow downs then you aren't doing something right. I typically will have an mda local with the mdb on the network when creating an app in access. To have good performance it's key to have the proper indexes and queries optimized for getting just the data you need. Whether using a separate app, access, or some app running against sql server you need to actively handle record locking properly. You can't just blindly let access lock your records.
Forget the arguments about DB Size, it is an uninformed reason to shift to a client-server platform in 90% of the cases I hear it brought up.
Your best arguments are based on features explained at a low tech level:
(1) You can backup and perform maintenance on the DB without kicking out the users (which introduces costly downtime).
(2) Faster recovery if data is accidentally deleted/mangled or corrupted. Again, less risk and less downtime. This is always a good foundation for a business case.
(3) If (and only if) you anticipate the need to scale quite a bit, the upgrade will better allow that.
(4) If you need to run automated jobs/updates, SQL can do this much more elegantly.
Remember the contra-indications for SQL, it is easy to get on your technical high-horse about this platform versus that, but you have to balance the benefits against the costs.
SQL is a Helluva lot more expensive to maintain as it requires dedicated hardware, expensive licenses (Server OS and DB) and usually at least a part time DBA that is going to cost you a bare minimum of $75K (if you get luck AND work out of Podunk Iowa).
The best possible advice I can give you is to make sure that you have a good attitude and are known as someone who does quality work and gets things done. It sounds like you don't have any control in the situation so what you need is influence.
Find a way to solve a problem (probably a different one that is less threatening to the people involved) in the way you are suggesting. Make it work blindingly fast and flawlessly. Make it work so well that people start asking for you when they need something done. Get it done quickly, which you should be able to do because you'll be using the right tools for the job.
Be a good person to work with, not the PITA that knows how everyone else should write their code. Be able to give an answer for what you might do differently and why, but don't automatically assume that your ideas are always the best. Maybe there are trade-offs that you don't know about -- no money in the budget for the extra CALs, we have this other app that needs to be done first. This doesn't sound like your situation, but looking for opportunities to understand before making constructive criticisms can go a long way to helping people be receptive.
The other thing is that this probably has nothing to do with the technical aspects of the situation and everything to do with the insecurities of the other developer. "This is all I know. If we change it, I won't understand it and then where will I be." Look for ways to help the other guy grow -- when he's having a problem, find resources that will help him develop good technical solutions. Suggest that everyone in your department get some training in new technologies. Who knows, one good SQL Server course and the guy could become the SQL Server evangelist in the organization because now THAT'S what he knows.
Lastly, know when to cut your losses, so to speak. If you find that you're not able to do anything about the situation, don't add to the complaining. Move on to something that you can control and do it as well as you can. Maybe in the future you'll be in a position that you do have control or influence in the situation and can do something about it. If you find that you're in a company that's more dysfunctional than most, find a way to move on to a place where the environment is better.
It is possible, and actually fairly easy, to convert an Access database to having the tables/views in SQL Server while still using the Access app as a front-end.
From there, your Access-obsessed developer can still have fun with all that VBA code. Meanwhile, on the back-end, you add indexes and such to speed everything up. Maybe someday you get lucky, and he asks about stored procedures. Then, the app is just a front-end, and who cares what it's written in? Your data is safe in SQL Server.
It is possible for you to do this yourself, but just leave the production app ALOOOOOOOOOOOONE. Take a copy, and convert that copy. Then, host it for a couple of users to TEST drive .. make your version of the Access app show "TEST APP" in big red letters. If your developer asks what you're doing, you can say the truth -- you are testing to see if converting only the tables/views might be of some help to the overall app.
This way, you get the best of both worlds, keep your developer happy, make the users happier (hopefully), and if you play it right, your bosses will know that you handled a knotty personnel issue with your technological prowess and your maturity.
I once had similar problems with someone I would not hesistate to call a complete idiot.
It was not possible to convince them of the issues with access. In the end it was easier to force the issue than do it "nicely", cruel to be kind.
If they resist then you can always go above their head. Management must be aware of crashes and stability related issues. Present a plan to them to improve stability and they are likely to at least listen. They will probably then want a meeting with all developers to discuss so go into it armed with plenty of ammo.
More than "How to convince them", let's talk about "How to do it without anybody noticing"!
First of all I advice you not to mix together the code optimisation issue and the SQL server one. Do not give users a chance to complain about SQL while bugs are related to something else.
If your code is really unbearable, rewrite the app before switching to SQL, keeping in mind the following points to make the final transition to SQL Server completely transparent for final users.
This is what we did 18 months ago, and I am sure we still have users thinking our database is Access:
Export current access database to SQL through available Wizard in access for testing purposes (many problems might occur, and you could need another tool such as the one proposed here).
Create a unique connection object at the application level, so that you can freely switch from Access to SQL at any time (at development level, you can even add an input box at startup to ask which connection to use). We chose an ADODB connection object, but it will also work with ODBC connection.
In case you use SQL syntax to update tables, make sure that all SELECTs, INSERTs, UPDATEs and DELETEs make use of this connection. In case you use recordset, make sure that all of them use this connection at opening time.
When needed, update all connexion specific code by adding a "SELECT CASE" type_Of_TheConnexion options
Switch to SQL connection ..and debug till you're done!
The problems you will find are mainly linked to SQL syntax, where MSSQL uses ' instead of " and # as separators. Date format is also an issue, where standard SQL format is 'YYYYMMDD' while MS-Access format depends on computer locals (beware of conversions from date to string!) and is stored as "YYYY-MM-DD" (if I remember ...). Boolean in SQL are 0 and 1, while they are True/False or 0/-1 in Access ...
Test, update code, and when you are ok, make a new data transfer, lock your app on the SQL connection, and distribute a new runtime.
It depends on the type of application and data load of your database but Access is quite efficient, even over the network.
Depending on the amount of data your users deal with you could easily scale up to a 100 users on a network just using a from and back-end Access database.
Looks like in your case a rewrite may be in order. If your application is data-centric if doesn't make much sense to develop it in VB6: the tools given by Access are much better than anything you'd be able to make, especially when considering Access 2007.
Upsizing to SQL Server is only really required if you're getting into issues of:
Security:
you need to make sure that only the rights users can access data. You can do your own security in Access, but it's never going to be as strong as SQL Server.
Scalability:
you're dealing with lots of data and complex queries or a lot of users and it would be better to have dedicated hardware to handle the load for the clients. The issue with this though is that while removing the pressure from the less-capable clients machines, you're adding a lot more to the server.
Integrity:
With the back-end database being just a file that needs R/W access for all connected clients, there's always the possibility that someone is going to do something bad or that a client may crash and leave the database corrupted.
If your number of users is average (I'd say 30), then there's probably no real need to upscale:
Use MS Access 2007 to develop your application, then just use the MS Access 2007 Runtime (it's free!) on all client machines to get a more modern user interface (uses the Ribbon and has lots of UI enhancements over previous versions).
You can't be the cheapness of that solution : you only need full retail version of MS Access and all the rest is free, regardess of the number of users!
Don't think that moving to SQL Server is going to improve performance of your queries: MS Access often does a better job of optimizing the queries for you (it knows what needs to be displayed and does lots of caching and optimization).
Make sure you only edit small amounts of data at any one time (don't use dynaset queries just to display vast amounts of data in a datasheet; use a snapshot instead and open a detail form that only contains the data to edit when necessary.
Cache complex queries locally.
Built some caching mechanism that leaves a copy of the results of a complex query on the local machine. The gain in performance is pretty amazing and if the query doens't change much (for instance a log of stock operations) you can just persist the complex/big query locally and append new records as necessary.
There is so much more to say.
Bottom line is: you may be looking at a rewrite, but don't dismiss Access as the solution because your current application was poorly written.
Try bechmarking and showing the stats to him
Making people change can sometimes be a real pain in the butt.
I would have to say the main argument would have stability and speed, but of course like you have said they already know this a still won't move.
Another thing to try would be to show them the power of LINQ to SQL and how much cleaner it would make your application. Like Daniel Silveira said you could try and throw a couple of stats there way and see if they are convinced.
We have a app build using MS access as a back end and I can't wait till we get our new SQL server so I can move everything to that.
You could show him the perf results comparing the two, but if he's really set in his ways and refuses to change, there isn't much you can do except force him somehow.
If you're his boss then just force him to change it to use SQL. If not, then convince your boss to force the change by showing him the perf results and explain it'll fix the issues you're having.
Errr, leave the team? You seem to be working with the totally wrong set of people. Now, if the team IS your company, then you are working with the wrong company.
Of course once you leave the company, you could tell your clients that you could solve the network problems on their own and make them leave the company as well. Then give them an improved system that works on SQL Server Express.