I have several SSIS packages jobs running, and some months ago my disk got full because the size of the SSISDB database.
I noticed that the cleanup_server_retention_window was set to 365 days, and I changed it to one day. (It is a development server, and at this point I really dont care about the history).
The (big) problem, obviously, is now that the transaction log grows a lot and fast.
To prevent this, I start performing a full backup every week and a transaction log backup every day, and the size of the database is now controlled.
However, some more experienced guys are telling me that this is not the best aproach to this issue, but I cant see any problem with it..
I would like to know if there is a better solution for this.
I tried just about everything including changing retention window; it was deleted the transactions but not reducing the log size. For me the allocated log file size grew to 75 GB. Nothing seemed to help.
The main issue has to do with the recovery model
of the SSIS DB that was set to 'Full'. Once I set this to 'Simple' and changed the initial log file size, all was fixed!
I have been monitoring this for the last couple of days just to make sure all is well and it looks fine to me so this operation is safe.
The current log file size is 512KBMB as opposed to 75GB!
The (big) problem, obviously, is now that the transaction log grows a lot and fast.
You will not see this everyday..The Cause of transaction log growth was changing cleanup_server_retention_window ..when you changed the value from 365 to 1,internally it has to do a lot of deletes
I start performing a full backup every week and a transaction log backup every day, and the size of the database is now controlled
I don't see an issue with backing up SSISD.In our instance ,we changed the recovery model to simple and do daily full backups
I fixed this in 3 ways:
adding some missing indexes on the SSISDB database; they should be there after installation of CU4
changed the parameter #delete_batch_size from 1000 to 25 in the stored procedure internal.cleanup_server_retention_window
changed the recoevry model from Full to Simple
Now, when running the SSISDB maintenance job, the transaction log no longer fills up beyond repair causing a database 'crash' / rollback
Related
I've recently taken on the database administration of a few SQL servers varying from SQL Server 2005 to 2014, where many of the DB's are in Full recovery mode, however no good ongoing backup maintenance plans were ever setup.
It seems to me that the previous DBA would only deal with transaction log files when they got out of control and fill up the hard drive. So i'd like to change this and fix the issue once and for all. I've been doing some reading and think I have a decent understanding of what need to be done, so i'd like to validate my understanding as well as ask a few question to clarify a few points that I still don't fully get.
So based on my understanding to date i would need to create a maintenance plan which starts with a Full Backup. I still need to talk to management to figure out things like RTO, acceptable data loss, etc so let's assume for this example that we'll do Full Backup's on Sunday.
Next I would add to this maintenance plan Differential backups every night... so Monday to Saturday. I realize that this could also be Full backup's or run the differentials more frequently, but again this is just as an example to make sure i'm understanding things correctly.
Now as for the transaction log backups. I get that i would need to back these up and truncate the log file to prevent it from continually growing and getting out of control. I don't know if there are any specific recommendation for how often to back this up, but i've seen 15 minutes suggested. I guess this would more so fall under the acceptable data loss window. Is that correct?
So the other thing that i've discovered is that when you backup the transaction log file with truncation, that if the log file has already grown out of control that it doesn't shrink the file. I've also read that it isn't good to shrink these files, at least on regular basis because once you shrink it, it would need to grow again and this will cause fragmentation and performance issues.
Now since I am currently in a situation that the files have already grown out of control, i'd assume that i should in fact shrink the log files this one time once i've but my maintenance in place. Is that assumption correct?
Also once i Shrink the transaction log file this one time, are there any maintenance task that i should be running to avoid performance issue due to shrinking the log files?
One other question that i was wondering about would be with respect to point in time recovery. So let's say I take a full backup at 5:00 AM and i also take a transaction log backup every 15 minutes. I get alerted that at 6:18 AM something has gone wrong (let's say a table was deleted). So i know i can restore by Full backup that happened at 5:00 AM and leave it in NO RECOVERY mode and restore all of the Transaction log backup from 5:15 AM to 6:15 AM, but here is what i'm interested in...since i have my DB in full recovery mode, is it possible to somehow use my existing transaction log file (not the backups) to roll forward all transaction between 6:15 and 6:17 just before the table was deleted? If so how would you do this? I guess this obviously wouldn't work in the case of you loosing the hard drive with your transaction log files, or your server exploding...but in a case like i've outlined is it do able?
Thanks
I would recommend doing a full backup after everyone stopped working, e. g. at 10 p.m. (if that is the case), not in the morning shortly before people start working. Just in order to give it enough time to run.
Personally, I prefer doing daily full backups instead of incremental backups if the database is not too big to save backups for, say, 14 days. I feel better to rely on less files. If database and full backups are too big, incremental backups might be the better choice.
As you said: How many transaction log backups you create during the day depends on the acceptable data loss window. In an environment > 5 people working on the system (just as a gut feeling) I would configure them to run all 15 minutes, on very big systems maybe even more.
After the first transaction log backup, you might want to shrink the LOG file ONCE.
I think it's not necessary to run any optimizations after a log shrink.
As far as I know it's not possible to restore transactions between 06:15 and 06:17.
When activating the transaction log backups, keep in mind that the first transaction log backup will be quite big (around the size of the current, large log). Ensure to have enough space on disk until you shrink the log file and delete the first transaction log (usually done automatically within the maintenance plan, e. g. after 14 days.).
I'm currently scanning 70 projects with a total of about 2 million lines of code. All has been going well until a few weeks ago I was notified that a few projects failed because we ran out of hard drive space on the SonarQube server. I was sure we had more than enough space according to the HW/SW requirements. I read that restarting the Sonarqube server service does clear up temp files, but after doing that several times, something is still eating up more space. The culprit is coming from SQL server:
...\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA\sonarqube_log.ldf
The size of this file is currently at 66.8 GB. Does anyone know if I can just truncate the stuff in there, know of any best practices for reducing this file size, and keeping the size down during future scans?
OK, if your database recovery is set to Full and you're not backing up the transaction logs, that's the problem.
If you want help understanding the differences between recovery models, then start here. The short version is that Full recovery allows point-in-time (aka, point-of-failure) recovery, meaning you can restore the DB to the exact moment before the problem occurred. The drawback of Full recovery is that you must backup your transaction logs or this exact issue happens: the logs grow endlessly. Simple recovery eliminates the need for transaction log backups (indeed, I don't think you can do a transaction log backup on a Simple database) but limits you to restoring the DB to what it was when you last ran a database backup. Note that simple recovery still uses transaction logs! The system just periodically flushes them.
So, you will need to do one of two things: Use Full recovery and backup your transaction logs periodically (I've seen systems that do it every hour or even every 15 minutes for high traffic systems), or switch to Simple recovery. Those are your only real options.
Whichever one you do, switching to Simple or backing up the logs will flush the transaction logs. You can verify that with DBCC SQLPERF(logspace). However, you'll notice that sonarqube_log.ldf will not change size at all. It will still be 66.8 GB. This is by design. In a properly managed system, the transaction logs will reach the size that they need to operate, and then the backups and simple flushes will keep the size constant. The log file will be the proper size to run the system, so it will never need to grow (which is expensive) and will never run out of space (which would cause all transactions to fail).
"So, how do I get my disk space back?" you ask. "I've fixed the problem so the log file is going to be 95% wasted space now."
What you'll need to do is shrink the log file. This is something that you will often see written that you should never do. And, I agree, in a properly managed system, you should never need to do this. However, your system was not running properly. The log files were on runaway. In general, though, you shouldn't ever need to do this.
First, take a full database backup. I repeat: Take a full backup of your database. This shouldn't cause any problems, but you don't want to be doing stuff like this without a fresh backup.
Next, you'll need to find the file id for the log file for the database in question. You can do that like this:
select d.name, mf.file_id
from sys.databases d
join sys.master_files mf
on d.database_id = mf.database_id
where mf.type = 1
and d.name = 'SonarQube'
The mf.type = 1 returns only transaction log files. Use the name of your database if it's not SonarQube.
--Switch to the SonarQube database. If you're not in the right context, you'll shrink the wrong file.
use [SonarQube];
--Do a checkpoint if you're on Simple recovery
checkpoint;
--Do a log backup if you're on Full recovery
backup log ....
--Shrink logfile to 1 GB. Replace #file_id with the id you got above.
dbcc shrinkfile ( #file_id, 1024 );
The size there is the size in MB. You'll have to make a guess based on how large your DB is and how many changes you make to your system. Something between 50% the size of the DB and 100% the size of the DB is pretty safe for most systems. In any case, I would not shrink the logs further than 1 GB. You'll need to monitor the log space and whether or not the log continues to grow. The Disk Usage report in SSMS is a good way to do that.
The first time you run this, you may not see much gain in disk space at all. It might go to 66.0 GB or 62 GB. The reason for that is because the current tail of the log file is still at the end of the transaction log file. What you should do is, if the system is under activity, wait a few hours and then run the above again. That will give the system time to cycle the log back to the beginning of the log file, and you'll be able to shrink it down. Here is a good article covering more about how shrinking the log file actually works, if you're interested.
Most of the time users will hit the database to read news. There are very few number of queries executed under transactions. 95% of the database hits would be for read-only purposes.
My database log files size is growing 1 GB per day. Even if I shrink the database, the log file size is not decreasing. What could be the reason for growing the log file size more and more? How can I control this? As per my knowledge log file does not increase when we read data from tables.
Any suggestions on how to deal with the log file growing? How can it be kept a manageable or reasonable size? Does this effect performance in any way?
There are couple of things to consider. What type of backups you do, and what type of backups you need. If you will have the answer to this question you can either switch Recovery Mode to simple or leave it full but then you need to make incremental backups on a daily basis (or whatever makes you happy with log size).
To set your database logging to simple (but only if you do Full Backups of your database!).
Right click on your database
Choose Properties
Choose Options
Set Recovery mode to simple
This will work and is best if your backup schedule is Full Backup every day. Because in such scenario your log won't be trimmed and it will skyrocket (just like in your case).
If you would be using Grandfather&Father&Son backup technique, which means Monthly Full backup, Weekly Full backup, and then every day incremental backup. Then for that you need Full Recovery Mode. If 1GB of log per day is still too much you can enable incremental backup per hour or per 15 minutes. This should fix the problem of log growing more and more.
If you run Full Backup every day you can switch it to simple recovery mode and you should be fine without risking your data (if you can leave with possible 1 day of data being lost). If you plan to use incremental then leave it at Recovery Mode Full.
Full backups will not help, you must regularly backup the transaction log (as well as the regular database full and differential backups) for it to be emptied. If you are not backing up the log and you are not in simple recovery mode, then your transaction log has every transaction in it since the database was set up. If you have enough action that you are growing by a gig a day, then you may also have large imports or updates affecting many records at once. It is possible you need to be in simple recovery mode where transactions are not recorded individually. Do NOT do that however if you have a mix of data from imports and users. In that case you need to back up the transaction log frequently to be able to keep the size manageable and to a point in time. We backup our transaction log every 15 minutes.
Read about transaction log backups in BOL to see how to fix the mess you have right now. Then get your backups set up and running properly. You need to read and understand this stuff thoroughly before attempting to fix. Right now, you would probably be in a world of hurt if your server failed and you had to recover the database. Transaction log backups are critical to being able to recover properly from a failure.
Do you backup your database frequently? You need to perform full- and/or transaction log- backups in order for SQL Server to consider shrinking your log file.
The major reason for the log file large size is due to bulk transaction in DB. To reduce the log file size best option to take the transaction log backup after certain interval of time.
I recently had to root out the cause of intermittent ODBC errors in a 12 year old SQL Based application. The culprit turned out to be this statement at the end of one of the stored procedures:
DUMP TRAN NotTheRealDBName WITH NO_LOG
I wasn't familiar with this command, so I did some research and found that it makes a backup of the transaction log. I believe this would only be useful if you at some point restored the transaction log, which didn't seem to be happening. The stored procedure in question was a simple sequence of inserts, updates, selects, and deletes on various tables.
I'm trying to imagine why one would want to backup and restore the transaction log in the middle of business logic, but I can't get my mind around it. Can anyone give me an example of an efficacious use of DUMP TRAN?
They key part is that the program also adds the WITH NO_LOG clause. So basically it truncates the log. Probably 12 years ago the author had run into log growth issues, 'thoroughly' researched the issue and concluded that the best avenue is to truncate the log whenever he was doing some lengthy updates. Nevermind that in the process he breaks the backup chain of any maintenance plan...
This command is used when you want to shrink your database; I never saw it used in business logic and I believe it to be actually dangerous when used that way.
DUMP is now BACKUP and I'm pretty sure the NO_LOG option has been discontinued in SQL Server 2008 (and with good cause).
Basically, if you configured your database improperly - if you used the "Full" recovery model but didn't actually do proper backups - then this was a way to shrink the log.
I can think of very few good reasons why you'd want to truncate the log without backing it up, and I also can't think of many reasons why you'd want to back up the log in the middle of business logic, but there is a very good reason to backup the transaction log in general, and that is to meet RPO targets.
Generally you're only going to be doing one database backup per day, but if you have a business requirement to be able to recover half a day's worth of transactions if the server blows up or some rogue DBA manages to hose your production database and all mirrors, then you'll likely be making hourly (or more frequent) transaction log backups. That way, you can restore the backup from yesterday night and use the transaction log backups to restore up to an hour ago (or whatever the RPO is).
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx