I have ran into an error today in regards to unavailable space on the drive where my SQL tables and logs are stored which has caused me to be unable to update any of the databases.
I have looked at the database server and deleted a database of approximatedly 1.5GB to allow me to continue. On looking on the server drive I can see backups for the database that I have deleted in the location:
E:\Program Files\Microsoft SQL Server\MCSSQL12.SQL2014\MSSQL\Backup
Inside this folder there are 5 backup copies for the last 5 days which I would like to delete to clear up the space. However when I delete the .BAK files from this folder it does not reallocate the free space. Am I missing a step here somewhere?
Quick wins:
If this is on a SAN, get the drive expanded by 20% just to give you some breathing room while you sort everything else.
Dropping databases does not delete their backups. You have to delete those too (and make sure they don't just land in the Recycle Bin). But not until you've verified that you either A) really don't need them anymore or B) have copies of those backups somewhere more durable.
Get your backups moved to a different drive - both now and for your scheduled backups. Your backups should not be on the same drive as your data. File this under "putting all of your eggs into one basket". Consider this: If your E drive fails, it's going to take both your data files and your backups with it. What do you have left?
Review your backup retention. Do you really need all 5 of those backups? On my instances, we do daily Full backups and transaction log backups every 15 minutes. We keep at most 2 Fulls on local storage - everything else is on tape. And once you have a Full, the transaction logs between that Full and the previous one aren't really needed unless you need to a Point In Time Restore to somewhere in between them. All managed by Agent jobs executing Ola Hallengren's backup scripts so we're hands-off and just monitoring.
Are you using backup compression? If not, this may help you get more backups on your volume if you can't change your retention period or backup location (but really, put your backups on another volume, in the interest of safety).
Less-quick wins:
Consider moving your logs to a separate drive. Transaction logs and data files have different I/O profiles/requirements, so by moving them to a different drive you can tune each appropriately - for both performance and space requirements.
Review your indexes and eliminate any that you don't really need (or consolidate them). Indexes take up space! Dropping unused indexes won't give free space back to the volume, but it will free space in your MDF files that can be used for other data instead of growing right away.
Do not shrink your database unless absolutely necessary and you're confident that it won't re-grow significantly in the very near future.
What are your data and log file growth increments? If you have free space on the drives where the log and data files live, and your are allowing for automatic file growth, sql server will grow the files when space is needed.
You may also need to review the information here MSDN regarding dbcc command shrinkfile.
Related
I am a newbie in SQL Server, I have a task to move the whole SQL Server to another.
I am trying to estimate how much space I need in the new SQL Server.
I ran EXEC sp_spaceused
And the following came up:
When I look into the output, it seems that the Database is using ~122GB (reserved), but when looking in the total database size (mdf + ldf) it is ~1.8 TB.
Does that mean when I copy the Database from the existing SQL Server to a new one I will need ~1.8 TBs into the new?
I am thinking about creating a back-up and copy the back-up to the new Server. How does the back-up takes into consideration the unallocated space? Does the back gets closer to the reserved or the database_size? I understand that this is without taking into consideration the uncompressed in the back-up, which will improve the file size.
Thx for the help.
The backup file will be much smaller than 1.8TB, since unallocated pages are not backed up. But the log and data files themselves will be restored to an identical size, so you will need 1.8TB on the target in order to restore the database in its current state.
Did you check to see if your log file is large due to some uncontrolled growth that happened at some point and is maybe no longer necessary? If this is where all your size is, it's quite possible you can fix this before the move. Make sure you have a full backup and take at least one log backup, then use DBCC SHRINKFILE to bring the log file back into the right stratosphere, especially if it was caused by either a one-time abnormal event or a prolonged log backup neglect that has now been addressed.
I really don't recommend copying/moving the mdf/ldf files (background) or using the SSMS UI to "do a copy" since you can have much greater control over what you're actually doing by using proper BACKUP DATABASE and RESTORE DATABASE commands.
How do I verify how much of the log data is being used?
If you're taking regular log backups (or are in simple recovery), it should usually be a very small % of the file. DBCC SQLPERF(LogSpace); will tell you % in use for all log files.
To minimize the size that the log will require in the backup file itself, then:
if full recovery, back up the log first.
if simple recovery, run a couple of CHECKPOINT; commands first.
I am working on my laptop for my study.
I have a table that has 150 million records.
I have to insert these records into another table.
While doing this process after around 2 hours I get my 150GB harddisk full notification.
when I check created files I see that .ldf size around 120GB while .mdf file around 30GB
I googled and found that ldf is the transaction log.
How can stop this file from creating or empty it during the process of insertion?
You can think of below options to reduce the log file growth:
You can go for SELECT * INTO to load to another table. It is minimally logged operation.
If possible, You can change the recovery model to simple for reducing the amount of log generated
Alternatively, if you want to keep the recovery model as Full, want to recover from the current scenario, then you can do below things to free up space:
Take backup of the log to free up the log file size. Safer approach, in case something goes wrong with full backup in next step, you can restore from the log backup and last available full backup.
Take Full backup of the database.
Once you take Full backup of database, you can delete the log backup taken in step no. 1
I've recently taken on the database administration of a few SQL servers varying from SQL Server 2005 to 2014, where many of the DB's are in Full recovery mode, however no good ongoing backup maintenance plans were ever setup.
It seems to me that the previous DBA would only deal with transaction log files when they got out of control and fill up the hard drive. So i'd like to change this and fix the issue once and for all. I've been doing some reading and think I have a decent understanding of what need to be done, so i'd like to validate my understanding as well as ask a few question to clarify a few points that I still don't fully get.
So based on my understanding to date i would need to create a maintenance plan which starts with a Full Backup. I still need to talk to management to figure out things like RTO, acceptable data loss, etc so let's assume for this example that we'll do Full Backup's on Sunday.
Next I would add to this maintenance plan Differential backups every night... so Monday to Saturday. I realize that this could also be Full backup's or run the differentials more frequently, but again this is just as an example to make sure i'm understanding things correctly.
Now as for the transaction log backups. I get that i would need to back these up and truncate the log file to prevent it from continually growing and getting out of control. I don't know if there are any specific recommendation for how often to back this up, but i've seen 15 minutes suggested. I guess this would more so fall under the acceptable data loss window. Is that correct?
So the other thing that i've discovered is that when you backup the transaction log file with truncation, that if the log file has already grown out of control that it doesn't shrink the file. I've also read that it isn't good to shrink these files, at least on regular basis because once you shrink it, it would need to grow again and this will cause fragmentation and performance issues.
Now since I am currently in a situation that the files have already grown out of control, i'd assume that i should in fact shrink the log files this one time once i've but my maintenance in place. Is that assumption correct?
Also once i Shrink the transaction log file this one time, are there any maintenance task that i should be running to avoid performance issue due to shrinking the log files?
One other question that i was wondering about would be with respect to point in time recovery. So let's say I take a full backup at 5:00 AM and i also take a transaction log backup every 15 minutes. I get alerted that at 6:18 AM something has gone wrong (let's say a table was deleted). So i know i can restore by Full backup that happened at 5:00 AM and leave it in NO RECOVERY mode and restore all of the Transaction log backup from 5:15 AM to 6:15 AM, but here is what i'm interested in...since i have my DB in full recovery mode, is it possible to somehow use my existing transaction log file (not the backups) to roll forward all transaction between 6:15 and 6:17 just before the table was deleted? If so how would you do this? I guess this obviously wouldn't work in the case of you loosing the hard drive with your transaction log files, or your server exploding...but in a case like i've outlined is it do able?
Thanks
I would recommend doing a full backup after everyone stopped working, e. g. at 10 p.m. (if that is the case), not in the morning shortly before people start working. Just in order to give it enough time to run.
Personally, I prefer doing daily full backups instead of incremental backups if the database is not too big to save backups for, say, 14 days. I feel better to rely on less files. If database and full backups are too big, incremental backups might be the better choice.
As you said: How many transaction log backups you create during the day depends on the acceptable data loss window. In an environment > 5 people working on the system (just as a gut feeling) I would configure them to run all 15 minutes, on very big systems maybe even more.
After the first transaction log backup, you might want to shrink the LOG file ONCE.
I think it's not necessary to run any optimizations after a log shrink.
As far as I know it's not possible to restore transactions between 06:15 and 06:17.
When activating the transaction log backups, keep in mind that the first transaction log backup will be quite big (around the size of the current, large log). Ensure to have enough space on disk until you shrink the log file and delete the first transaction log (usually done automatically within the maintenance plan, e. g. after 14 days.).
I need the best shortest possible way to handle SQL Server transaction log files as my disk will be having a problem in saving them, in future. I don't know how to tackle this Low Disk Space issue, as my drive's free space is already in MB's.
Now, I don't know that whether the logs will be lost or affected if I move them to some other drive or will it effect the transactions or how to save the future logs depending on some previous transactions, if the disk space is completely full. Someone please help!!
Lets first take something off the table: do not delete or move any database mdf or ldf file. You'll end up corrupting and loosing the database.
You need to investigate why is the log growing. Go read Factors That Can Delay Log Truncation. Follow the steps in the article to identify why is the log growing.
If the reasons is anything else but 'LOG_BACKUP', post an update with the reason you discovered and we can take give further advise.
If the reason is LOG_BACKUP then we can proceed. You have a database in non-SIMPLE recovery mode which is not being backed up correctly. You need to answer a question: Why is the database not in SIMPLE recovery mode? This is a business decision question so we cannot possibly know the answer.
If you don't know the answer to above or if you realize that SIMPLE recovery mode is acceptable, then we can do the quick fix. Change the recovery model to SIMPLE then run DBCC SHRINKFILE to shrink the log.
If you need a non-SIMPLE recovery model then you need to set up a proper log backup plan and start taking log backups. Read Transaction Log Backups and Use the Maintenance Plan Wizard. See See How to shrink the SQL Server Log to understand why you need to take repeated log backups until the SHRINKFILE is effective, due to the circular nature of the log.
First, you should read the excellent answer written by Remus Rusanu.
At the beginning, he wrote:
Lets first take something off the table: do not delete or move any database mdf or ldf file. You'll end up corrupting and loosing the database.
To clarify: It is possible to move the log file to another drive in order to free space on the current drive.
It's just that you can't/shouldn't just move the file in Windows Explorer while the database is in use by SQL Server.
Disclaimer: You probably won't need to do this if you follow the steps in Remus' answer.
But it's possible that you still may want to move the log file to a different drive.
Either for performance reasons (SQL Server ist faster if database and log file are on two different physical drives, because there are lots of writes to the log file), or if you still have a disk space problem (even if you shrink the log file or back it up regularly, some day the database size will grow so that the drive is too small to hold both files).
To move the log file (or the database file) to another location, you need to:
detach the database from SQL Server
move the file(s)
re-attach the database and specify the new file location(s)
Here are two tutorials with screenshots how to do this:
Move SQL Server transaction log files to a different location via TSQL and SSMS
Move SQL Server transaction log to another disk
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx