I have a sql database that I dump data into every 15 minutes using SSIS. The transaction log get's huge and I back it up and shrink it a few times a week. But I know I'm doing something wrong. What is the best practice for me to maintain it? Should it stay ~1GB and I should be backing it up hourly? Since it's a datawarehouse, should I be backing it up at all? Show I be doing something different in SSIS? The datawarehouse recovery model is Bulk Logged.
You should also look to make sure SSIS is setup correct to do minimally logged operations as you may not have it set correctly. Then once that is working correctly, evaluate if you really need bulk logged recovery model: http://msdn.microsoft.com/en-us/library/ms175987(v=sql.105).aspx is good link on subject. If you can quickly (your definition of quick) redo the data that is lost and don't want point in time recovery of non-bulk operations, move to simple recovery, IMHO.
You have a few options with logs and they are related to how you are backing up your database and how much data loss you can stand. If you are doing full backups nightly for example...change you database to simple mode. This truncates your log on commit (meaning once logged operations complete the log cleans itself up) ... there are a lot of articles on this topic ... Google it up and if you need more help post back.
Related
I am very new to sql server, does anyone know what kind of recovery model for each system databases. I don't know if I should all make it to simple or full in my databases. Because if the database is in full so I need to create a log backup for that.
There's basically an option between simple and full. Which one to use depends on your requirement on what data can be lost in case of a disaster and to what point in time you can restore back to.
In simple recovery model you can only restore up to the latest backup and the only point in time you can restore into is the end time of the backup. If that is not enough, you need to use full. Then you can restore back to any point in time.
The third option is bulk logged, which is basically full with the exception that bulk load operations can be done with minimal logging and you can lose the changes done in the bulk load if the disaster happens before it ended.
I understand that the transaction logs keep a record of historical transactions in order to facilitate a restore if needed. However do I need to keep creating transaction log backups for inactive databases that are hanging around on the server? No DDL statements are run against them and they are just used for reference.
I am just a bit worried that I might run out of log space if I get this wrong.
Have you considered changing the recovery model of your databases to the SIMPLE recovery model? Doing so would negate the need to backup the transaction log as it would be automatically re-used in the "unlikely" event that you need it to be.
I would still advise that regular FULL database backups be taken.
Also, if these database are indeed true read only databases then why not consider setting them to be so. This action would have the advantage of immediately highlighting any queries/users that are "still" issuing DML operations when you believe there to be none.
Other options for identifying queries that are performing more than just READ operations include running a Profiler Trace of activity on your database server and also an aggressive option would be to revoke all data modification rights from the relevant database Users.
Transaction logs are actually truncated when they're backed up. So, if these databases are actually inactive, you shouldn't be backing up any transaction logs for them since the logs would be empty.
Also, common practice for "inactive" databases would be to make them READ ONLY with a SIMPLE recovery model.
I recently had to root out the cause of intermittent ODBC errors in a 12 year old SQL Based application. The culprit turned out to be this statement at the end of one of the stored procedures:
DUMP TRAN NotTheRealDBName WITH NO_LOG
I wasn't familiar with this command, so I did some research and found that it makes a backup of the transaction log. I believe this would only be useful if you at some point restored the transaction log, which didn't seem to be happening. The stored procedure in question was a simple sequence of inserts, updates, selects, and deletes on various tables.
I'm trying to imagine why one would want to backup and restore the transaction log in the middle of business logic, but I can't get my mind around it. Can anyone give me an example of an efficacious use of DUMP TRAN?
They key part is that the program also adds the WITH NO_LOG clause. So basically it truncates the log. Probably 12 years ago the author had run into log growth issues, 'thoroughly' researched the issue and concluded that the best avenue is to truncate the log whenever he was doing some lengthy updates. Nevermind that in the process he breaks the backup chain of any maintenance plan...
This command is used when you want to shrink your database; I never saw it used in business logic and I believe it to be actually dangerous when used that way.
DUMP is now BACKUP and I'm pretty sure the NO_LOG option has been discontinued in SQL Server 2008 (and with good cause).
Basically, if you configured your database improperly - if you used the "Full" recovery model but didn't actually do proper backups - then this was a way to shrink the log.
I can think of very few good reasons why you'd want to truncate the log without backing it up, and I also can't think of many reasons why you'd want to back up the log in the middle of business logic, but there is a very good reason to backup the transaction log in general, and that is to meet RPO targets.
Generally you're only going to be doing one database backup per day, but if you have a business requirement to be able to recover half a day's worth of transactions if the server blows up or some rogue DBA manages to hose your production database and all mirrors, then you'll likely be making hourly (or more frequent) transaction log backups. That way, you can restore the backup from yesterday night and use the transaction log backups to restore up to an hour ago (or whatever the RPO is).
After reading an article about the subject from O'Reilly, I wanted to ask Stack Overflow for their thoughts on the matter.
Write locally to disk, then batch insert to the database periodically (e.g. at log rollover time). Do that in a separate, low-priority process. More efficient and more robust...
(Make sure that your database log table contains a column for "which machine the log event came from" by the way - very handy!)
I'd say no, given that a fairly large percentage of server errors involve problems communicating with databases. If the database were on another machine, network connectivity would be another source of errors that couldn't be logged.
If I were to log server errors to a database, it would be critical to have a backup logger that wrote locally (to an event log or file or something) in case the database was unreachable.
Log to DB if you can and it doesn't slow down your DB :)
It's way way way faster to find anything in DB then in log files. Especially if you think ahead what you will need. Logging in db let's you query log table like this:
select * from logs
where log_name = 'wcf' and log_level = 'error'
then after you find error you can see the whole path that lead to this error
select * from logs
where contextId = 'what you get from previous select' order by timestamp
How will you get this info if you log it in text files?
Edit:
As JonSkeet suggested this answer would be better if I stated that one should consider making logging to db asynchronous. So I state it :) I just didn't need it. For example how to do it you can check "Ultra Fast ASP.NET" by Richard Kiessig.
If the database is production database, this is a horrible idea.
You will have issues with backups, replication, recovery. Like more storage for DB itself, replicas, if any, and backups. More time to setup and restore replication, more time to verify backups, more time to recover DB from backups.
It probably isn't a bad idea if you want the logs in a database but I would say not to follow the article's advice if you have a lot of log file entries. The main issue is that I've seen file systems have issues keeping up with logs coming from a busy site let alone a database. If you really want to do this I would look at loading the log files into the database after they are first written to disk.
Think about a properly setup database that utilizes RAM for reads and writes? This is so much faster than writing to disk and would not present the disk IO bottleneck you see when serving large numbers of clients that occurs when threads start locking down due to the OS telling them to wait on currently executing threads that are using all available IO handles.
I don't have any benchmarks to prove this data, although my latest application is rolling with database logging. This will have a failsafe as mentioned in one of these responses. If the database connection can not be created, create local database (h2 maybe?) and write to that. Then we can have a periodic check of database connectivity that will re-establish the connection, dump the local database, and push it to the remote database.
This could be done during off hours if you do not have a H-A site.
Sometime in the future I hope to develop benchmarks to demonstrate my point.
Good Luck!
So our SQL Server 2000 is giving me the error, "The log file for database is full. Back up the transaction log for the database to free up some log space."
How do I go about fixing this without deleting the log like some other sites have mentioned?
Additional Info: Enable AutoGrowth is enabled growing by 10% and is restricted to 40MB.
To just empty it:
backup log <dbname> with truncate_only
To save it somewhere:
backup log <dbname> to disk='c:\somefile.bak'
If you dont really need transactional history, try setting the database recovery mode to simple.
Scott, as you guessed: truncating the log is a bad move if you care about your data.
The following, free, videos will help you see exactly what's going on and will show you how to fix the problem without truncating the logs. (These videos also explain why that's such a dangerous hack and why you are right to look for another solution.)
SQL Server Backups Demystified
SQL Server Logging Essentials
Understanding Backup Options
Together these videos will help you understand exactly what's going on and will show you whether you want to switch to SIMPLE recovery, or look into actually changing your backup routines. There are also some additional 'how-to' videos that will show you exactly how to set up your backups to ensure availability while managing log file sizing and growth.
ether backup your database logs regularly if you need to recover up to the minute or do other fun stuff like log shipping in the future, or set the database to simple mode and shrink the data file.
DO NOT copy, rename, or delete the .ldf file this will break your database and after you recover from this you may have data in an inconsistent state making it invalid.
I don't think renaming or moving the log file will work while the database is online.
Easiest thing to do, IMO, is to open the properties for the database and switch it to Simple Recovery Model. then shrink the database and then go back and set the DB to Full Recoery Model (or whatever model you need).
Changing the logging mode forces SQL Server to set a checkpoint in the database, after which shrinking the database will free up the excess space.
My friend who faced this error in the past recommends:
Try
Backing up the DB. The maintenance plan includes truncation of these files.
Also try changing the 'recovery mode' for the DB to Simple (instead of Full for instance)
Cause:
The transaction log swells up due to events being logged (Maybe you have a number of transactions failing and being rolled back.. or a sudden peaking in transactions on the server )
You may want to check related SO question:
How do you clear the transaction log in a SQL Server 2005 database?
Well you could take a copy of the transaction log, then truncate the log file, which is what the error message suggests.
If disk space is full and you can't copy the log to another machine over the network, then connect a drive via USB and copy it off that way.
You have the answer in your question: Backup the log, then it will be shrunk.
Make a maintenance plan to regularly backup the database and don't forget to select "Backup the transaction log". That way you'll keep it small.
If it's a non production environment use
dump tran <db_name> with no_log;
Once this has completed shrink the log file to free up disk space. Finally switch database recovery mode to simple.
As soon as you take a full backup of the database, and the database is not using the Simple recovery model, SQL Server keeps a complete record of all transactions ever performed on the database. It does this so that in the event of a catastrophic failure where you lose the data file, you can restore to the point of failure by backing up the log and, once you have restored an old data backup, restore the log to replay the lost transactions.
To prevent this building up, you must back up the transaction log. Or, you can break the chain at the current point using the TRUNCATE_ONLY or NO_LOG options of BACKUP LOG.
If you don't need this feature, set the recovery model to Simple.
My dear friend it is vey important for a DBA to check his log file quite frequently. Because if you don't give much attention towards it some day it is going to give this error.
For this purpose you have to periodically take back up so that the logs file would not faced such error.
Other then this the above given suggestion are quite right.
Rename it it. eg:
old-log-16-09-08.log
Then the SQL server can use a new empty one.