We have a lot of sql server databases in Simple Recovery Mode.
We need to move these databases on a different datacenter. These databases can not be closed (they are serving 24x7 mission critical services).
We are thinking to use log shipping to create a remote standby database and then switching the remote copy to be primary.
How we can estimate the amount o transaction log written daily by each of these databases?
We need to know this amount to estimate netowork bandwith and size of transaction log backup.
Regards
Giovanni
You don't have to estimate: set the database to full recovery and start taking log backups. You'll have to do this anyways to do log shipping and you should really be doing this anyways if the db is that important to you. Once you're taking the log backups, you'll know how big the log backups are over any given prior of time so you can estimate bandwidth based on that.
I wrote a stored procedure that do the following:
read num_of_bytes_written from sys.dm_io_virtual_file_stats
random insert and delete rows from tables
read num_of_bytes_written from sys.dm_io_virtual_file_stats
calculate difference in megabytes between the two values of
num_of_bytes_written
make compressed transaction log backup
check the difference between log backup size and the num of bytes
written
After 1000 executions I'll calculate the ratio between log size and backup size
regards
Giova
Related
I have a database with log file size of 700 MB. Now I have fixed its max file size to 1 GB.
When it reaches 1 GB,transaction failed the reason for the same is that "The transaction log for database is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases"
This is same case if I uncheck Autogrowth for log file.
When I checked log_reuse_wait_desc column in sys.databases it says "Active_Transaction".
I am not able to understand why Sql server is not maintaining the max file size limit.Why it cannot delete old logs or something like that to maintain the max file size.
How does it work.
What I want is to limit the log file size that not to exceed 1 GB in any case.
There are a few things you need to consider here, especially if you want to restrict the log file size to 1GB.
as already mentioned, you need to understand the difference
between the three recovery models. Taking log backups is a key task
when using the full recovery model. However this is only part of the
problem; log backups only truncate the inactive part of the log,
therefore a transaction could fill the log file with 1GB+ of data,
and then you are in the same position you are in now... even if you are in simple recovery model (a log backup will not help you here!).
In an ideal situation, you would not restrict the log file in such a way, because of this problem. if possible you want to allow it to auto-grow so, in theory, it could fill the disk.
Transaction log management is a science in itself. Kimberly Tripp has some very good advice on how to manage transaction log throughput here
understanding VLF's will allow you to better manage your transaction log, and could help towards better proportioning your log file for large transactions.
If, after everything you have read, you are still required to limit the transaction log growth, then you will need to consider batch updating large result sets. this will allow you to update, say, 1000 rows at a time, meaning that only 1000 records are written to the log. SQL Server uses write-ahead logging, so in order to complete a transaction, you first need to have enough space in the transaction log to write all the details. If using a simple recovery model, this write-ahead logging is automatically truncated, meaning you don't need to backup the log. Therefore, writing 1000 records at a time (for example) will causing less of a problem than a huge 1,000,000 record insert (say)
Redgate provide a free e-book to help you on your way!
EDIT:
P.s. I've just read your comment above... If you are in full recovery model you MUST do log backups, otherwise sql server WILL NOT recover the space from the log, and will continue to write to the log causing it to expand! However note, you MUST have a full backup for transaction log backups to take effect. SQL Server cannot backup the log if it doesn't have an initial restore point (i.e. the full backup).
One of our servers is an SQL Server 2008 R2 database (full recovery) of less than 100GB in size that is backed up daily with transaction log backups taken every 15 minutes. The Log file growth is set to 500MB.
But no matter what I do, I cannot get the VLF count down to a respectable figure (e.g. 50) and is now constantly sitting between 500-700.
I've used the expand, backup logs, shrink, expand, backup, shrink, etc... technique and whilst it works to drop the VLF count from 700 to 500, that is as far as it will go.
The database is partially replicated (Transactional - continuous). We rely heavily on doing reports off the subscription database and up-to-date reports are run 24-7, so we cannot change the replication model. We do nightly index rebuilds/reorganising and update statistics weekly.
There aren't any long running transactions according SYSPROCESSES.
I am at a bit of a loss. Is there any cure apart from temporarily performing a Backup -> Simple -> Full operation?
TIA
Most of the time users will hit the database to read news. There are very few number of queries executed under transactions. 95% of the database hits would be for read-only purposes.
My database log files size is growing 1 GB per day. Even if I shrink the database, the log file size is not decreasing. What could be the reason for growing the log file size more and more? How can I control this? As per my knowledge log file does not increase when we read data from tables.
Any suggestions on how to deal with the log file growing? How can it be kept a manageable or reasonable size? Does this effect performance in any way?
There are couple of things to consider. What type of backups you do, and what type of backups you need. If you will have the answer to this question you can either switch Recovery Mode to simple or leave it full but then you need to make incremental backups on a daily basis (or whatever makes you happy with log size).
To set your database logging to simple (but only if you do Full Backups of your database!).
Right click on your database
Choose Properties
Choose Options
Set Recovery mode to simple
This will work and is best if your backup schedule is Full Backup every day. Because in such scenario your log won't be trimmed and it will skyrocket (just like in your case).
If you would be using Grandfather&Father&Son backup technique, which means Monthly Full backup, Weekly Full backup, and then every day incremental backup. Then for that you need Full Recovery Mode. If 1GB of log per day is still too much you can enable incremental backup per hour or per 15 minutes. This should fix the problem of log growing more and more.
If you run Full Backup every day you can switch it to simple recovery mode and you should be fine without risking your data (if you can leave with possible 1 day of data being lost). If you plan to use incremental then leave it at Recovery Mode Full.
Full backups will not help, you must regularly backup the transaction log (as well as the regular database full and differential backups) for it to be emptied. If you are not backing up the log and you are not in simple recovery mode, then your transaction log has every transaction in it since the database was set up. If you have enough action that you are growing by a gig a day, then you may also have large imports or updates affecting many records at once. It is possible you need to be in simple recovery mode where transactions are not recorded individually. Do NOT do that however if you have a mix of data from imports and users. In that case you need to back up the transaction log frequently to be able to keep the size manageable and to a point in time. We backup our transaction log every 15 minutes.
Read about transaction log backups in BOL to see how to fix the mess you have right now. Then get your backups set up and running properly. You need to read and understand this stuff thoroughly before attempting to fix. Right now, you would probably be in a world of hurt if your server failed and you had to recover the database. Transaction log backups are critical to being able to recover properly from a failure.
Do you backup your database frequently? You need to perform full- and/or transaction log- backups in order for SQL Server to consider shrinking your log file.
The major reason for the log file large size is due to bulk transaction in DB. To reduce the log file size best option to take the transaction log backup after certain interval of time.
This question might be kind of elementary, but here goes:
I have a SQL Server database with a 4 GB log file. The DB is 16GB and is backed up nightly.
Can I truncate the log regularly because the entire DB+Log is backed up each night?
you can something like this to you maintenance schedule to run every night before the backup. This will try to shrink/truncate your log file to 1 meg
BACKUP LOG DBNAME
TO disk = 'D:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Backup\DBNAME.log'
DBCC SHRINKFILE('DBNAME_Log', 1)
Are you sure the log is backed up nightly and not just the database?
If so, then what does this database do? Are you deleting and refreshing whole tables? If so, your log might be the right size for the amount of transactions you have. You want the log to be large enough to handle your normal transaction load without having to grow. A small log can be a detriment to performance.
If this database is not transactional in nature (i.e., the tables are populated by full refreshes rather than one record ata time), the change the recovery mode to simple. Do not do that though if you have transactional tables that you will need to be able to recover from the log rahter than simply re-importing the data.
If you can run log backups during the day (depending on load, etc. this may or may not be possible for you) you can keep the log file under control by doing so. This will prevent the log file itself from growing quite so large, and also provide the side benefit of giving you the ability to restore closer to the point of failure in the event of a problem.
You'll still need to shrink the log file once using DBCC SHRINKFILE, but if it's backed up regularly after that point it shouldn't stabilize at a smaller size.
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx