As part of our DR solution, we have attempted to enable log shipping for a heavy transaction load database. While the configuration completes successfully, the first transaction log backup job to kick off after the completion of the log shipping configuration runs continuously and grows exponentially in size. On one occasion, that first transaction log backup job ran for 12 hours with a file size 3X greater than the 27 GB full backup file for the database. We killed that process. Recently, we tried a twist on the approach using a differential as explained below, but the transaction log backup job still ran with an ever growing file size.
This process was run during weekend low use hours
7:46 am – log shipping configuration kicks off
9:32 am – backup file is stored in network share folder. File size is 26.1 GB
9:30 pm – Log shipping configuration completes.
– I disable the log shipping backup, copy, and restore jobs
9:31 pm – I enter command to backup database with differential
9:33 pm – Differential completes with a file size of 768 MB.
– I re-enable the backup and copy jobs to get that process moving along after the differential
– I copy the differential file to the secondary location
9:45 pm – The first transaction log backup job kicks off
9:59 pm – After the Differential file is copied, I restore the database on Secondary using the differential
11:02 pm – The restore of the differential is still running
– The transaction log backup job that was created at 9:45 am is still running with a file size of 28 GB and still growing.
We ultimately killed this process due to space issues as the transaction log backup job never completed.
Has anyone experienced this scenario before? Is there anything we could change to improve the process time on the transaction log backup job? Given the heavy transaction load, I wonder if it would be best to implement an alternative DR solution for this particular database.
I know this may be old,but adding some pointers which will help you.
1.When database is set to Bulklogged recovery model,Tlog will contain copy of data files too,so your Tlogs size will be big
2.further you might want to check what is happening during backups and restores using below trace flags.
dbcc traceon(3004,3605,-1)
3.Same trace flag can be applied to restore as well
4.further if restore is taking some much time,this might be due to huge transactions which are rolledback.See below link for more details
http://www.sqlskills.com/blogs/paul/why-could-restoring-a-log-shipping-log-backup-be-slow/
5.You also can enable Instant file initilization to speedup restores as this will help in growing data files instantly
you also can check if there is network latency by using perfmon counters.
Related
Just started with a company and noticed that their database was set to Simple Recovery.
I talked to the owner and suggested to convert it to Full Recovery, explained to him the benefit of using a transaction log and backed up every hour. After he agreed I did a full DB backup prior to conversion. Then scheduled hourly backup for the Transaction log file and Full Nightly backup for the Data File.
It was my impression that once the hourly backup started running, the size of the transaction log (60GB) would shrink. It's been more than a month but the size of the transaction log is still the same.
Is it okay to run DBCC ShrinkDB against the Log file without detaching and attaching the database?
I have the impression that once the hourly backup started running the size of the Tlog, 60GB will start to shrink. It's been more than a month but the size of the Tlog is still the same.
Log file will not shrink automatically
Is it okay to run DBCC ShrinkDB against the Tlog
Don't shrink log file,unless you are in shortage of space.Reason behind this is, file growth operations are expensive
you can see below command to view free space in log file
dbcc sqlperf('logspace')
The log backups that you're taking will help to keep the log file from growing, but the log file won't shrink on its own. Internally, the log file is segmented into virtual log files (VLFs) and are used in (more or less) a cyclical manner. While you're running your work load, transactions are being recorded into these VLFs. When the log backup runs, it will read from any VLF that had transactions since the last log backup, write those transactions to disk, and then clear the VLF and mark it as available for reuse.
I have Windows scheduler calling a program that does a full database backup every day at 3:00am.
I would also like to do a transaction log backup every ten minutes.
What is the best way to sync these?
I understand that transaction logs are independent of full back-ups, but is it a problem if the two different tasks both do a backup at 3:00am? i.e. the database is asked to produce a full back-up and a transaction log back-up at exactly the same time.
Perhaps I should have one task and query SQL Server to see if the last full backup was more than 24 hours ago. If not, then do a transaction log backup.
As long as you're running SQL 2005 or higher, you can run log backups at the same time as a full backup without issue. Glossing over a bit of detail, log backups are based on the last completed full backup at the time that the log backup started. The only time it really matters is when you'd need to do a recovery and even then, it's only ever "contentious" (i.e. "which full backup should I restore given that I want to restore to this point in time?") for log backups that are taken while a full backup is in progress.
If ever you need to do a point-in-time restore for such a time, you'd work backwards. That is, first find the log backup that contains said point in time. Then, to find the full backup that it's based on, look at the database backup lsn for that log backup in either msdb.dbo.backupset or by running restore headeronly on the log backup. From there, find the full backup that has that value as the checkpoint lsn. Now, you just need to restore that full backup (using the norecovery option) and every log backup based on it up to and including the log backup you identified in the first step (again, all using norecovery). Then run restore database [yourDB] with recovery to run crash recovery and you're back in business.
I am a programmer, with a side job as an involuntary DBA.
I have a maintenance plan that does a full backup and a 'check database integrity' every night. I backup transaction logs every 10 minutes. The transaction log backup size spikes after the database backup - exponentially bigger. I used to rebuild indexes and statistics every night - I thought that is what was causing the transaction log spike - but removing those steps didn't change anything.
Mirroring our backups on slow connections would be helped considerably if there wasn't this massive spike - so I am hoping it is something I am doing wrong. Can anyone suggest anything?
If you are only running the log backup from 6am to midnight, then the very first log backup at 6am is backing up all the database activity that has occurred in the 6 hours since the last log backup.
This is entirely normal, and probably has nothing to do with the fact that your database backup takes place at 4am.
Since you are on SQL2008, the warning in my other answer doesn't apply, and you should be fine with running the log backups 24 hours.
Is this SQL 2000?
In SQL 2000, you're not supposed to run the log backup while the full backup is executing, or "bad things can happen", like blocking, or hugely bloated log files.
See this ServerFault post for "The Word" from "The Man", Paul Randal, who used to be in charge of the SQL engine at Microsoft.
See this follow-up post for some ideas for skipping the log backup while the full backup is executing.
In SQL 2005 or later, this restriction no longer exists, and you shouldn't have trouble running log backups and full backups at the same time.
While your full backup is running, transaction backups will not run. So how long does your full backup take? The transaction log will not be truncated during this time by transaction log backups.
I have a nightly job that does a bunch of inserts. Since I have a full recovery model, this increases my transaction log size.
Currently I have my log file big enough to accommodate these transactions, but the issue is that the transaction log is mostly empty throughout the day.
Is it an issue (besides disk space) to have a huge (mostly empty) transaction log?
I'm thinking about switching the database to simple recovery before the job, running the job and then switching it back to full recovery. I can have the transaction logs just not be backed up until our nightly differential backup comes around and then i can start the transaction log backup again.
Suggestions?
I'd do nothing. Or have SIMPLE permanently.
Changing the recovery model back to full will require a full backup anyway to preserve integrity later on. You'll have a gap in your LSN chains otherwise.
You've mentioned differential backup, so I assume your full is not each night.
So, putting this together means you'll use more disk space for your full backup than for the LDF file.
I'd keep it FULL and, if push comes to shove, force log backups during the nightly job to keep the log size small. Also, is OK to shrink the log if you really have to, ie. it grew once due to a one time operation and it won't grow back. The fragmentation concerns are more valid for data files, log files have a totally different structure and allocation pattern. Just don't get into the habit of constant shrinking.
I'm saying this because if you do have a backup restore strategy in place already it seems silly to me to increase the window of data loss from 'last log backup' to 'last differential backup'. We're talking a change from 10-30 minutes (typical log backup frequency) to 3-24 hours (typical differential frequency). You won't be able to do differentials as often as log backups because differentials are growing in size (starting with second differential after full each differential is at least as big as previous differential). Log backups only backup the log since last backup, so they stay relative constant in size. Also with a SIMPLE mode you won't be able to do a log tail backup attempt and recover all data in case of crash.
It just seems you're trading off a lot more that you gain with reducing the log file size.
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx