I have to take a Production SQL Database backup (Full Backup ) while the connection is on and transaction is happening. Does it affect any transactions. Do they maintain any transaction isolation levels ??
Not a development question.
Backups are transactional integer and will contain the transactions closed at the time of backup. ALl new / open transaction during the backup will be in the transaction log and not in the backup.
Database servers are not written by mediocre developers and the specialists know that backups are of paramount integrity. And like usual for high level products they write good documentation which - one can acutally read.
Technet magazine had a simple explanation for the different backup strategies; for Full Backup it starts out with:
"A full database backup provides a complete copy of the database and provides a single point-in-time to which the database can be restored. Even though it may take many hours for the backup process to run, you can still only restore the backup to a single point (effectively at the end of the backup, but I'll discuss exactly what that point is later in this article). A full backup does not allow recovery to any point in time while the backup was running. This is also the same for differential backups."
Your users will be able to use the application subject to the server having the disk / memory resources to carry on running whilst the backup is processed.
Related
I'm having an issue that only happens sometimes where my SQL Server differential backup job fails with an error message similar to
Msg 3035, Sev 16, State 1, Line 1 : Cannot perform a differential backup for database "MyDatabaseName", because a current database backup does not exist. Perform a full database backup by reissuing BACKUP DATABASE, omitting the WITH DIFFERENTIAL option. [SQLSTATE 42000]
Msg 3013, Sev 16, State 1, Line 1 : BACKUP DATABASE is terminating abnormally. [SQLSTATE 42000]
I'm currently using Ola Hallengren's SQL Server Mantenance Solution script for backup's, integrity check's and index maintenance. I've scheduled the backup job's as such:
Full Backup of system databases every day # 1:30 AM
Full Backup of all user databases every week on Monday, Wednesday & Friday # 2:30 AM
Differential Backup of all user databases every week on Sunday, Tuesday, Thursday & Saturday # 2:30 AM
Transaction Log backup or all user databases every 30 minutes
I've also set the cleanup time to 168 hours...which is 7 days.
I know that usually when this error message comes up, it is due to a full backup not being present, or possible the recovery mode on a database being changed. I have checked both of these and neither seems to be the case. I can confirm that my Friday full backup was successful, however my Saturday and Sunday Differentials failed. There have also been no changes to the recovery model, and no manual backup taken through SQL Server.
It's worth noting that this only happens sometimes. Sometimes the differential backups work without a problem and other times they fail.
This server is a VM, and we are using VMWare vSphere/vCenter 6.5. I've talked to my server admin and asked how his backup is running an he told me that we are using Quest AppAssure which leverage VMWare snapshot technology, and that he is backuping the drives every x number of minutes, so it is possible that the time of his backup changes and eventually overlaps with mine.
We looked back at the time that his backups ran on the weekend and they happened within a few minutes before mine started. I'm wondering if this is causing my backup issues? if so is there a way to prevent this, or do we simply need to plan the backups at different times that don't overlap?
Thanks
We had another call with Quest today and figured out a solution to the problem.
It would seem that when configuring the backup through Quest Rapid Recovery, you have the option to do Machine level backups or Volume level backup. When it's configured to do volume level backups you can choose for it to either do a block level backup which isn't SQL Server aware, or for it to do a backup that is SQL Server aware, which ends up using the volume shadow copy service and these backups show up as Full (Copy Only) backups in the SQL backup history...even though you can't restore them from SQL Server.
Rapid Recovery can only do backups on a schedule, plus has the option to Truncate Logs after the backup complete to avoid filling the log files, but can't do transaction log backups, so you loose the ability to do a more granular recovery down to the second like with the native SQL Timeline Recovery...Which is why we chose to use native SQL Server Backups.
So to fix the issue, you need to either do a machine level backup which has not SQL Server Awareness/integration. Or you can do a volume level backup, but disable the SQL Server Writer extension, and the truncate logs options to remove the integration.
We've run a bunch of tests, and from the point/time that this change was done, we only see SQL Server backups and no longer the Rapid Recovery backups in the SQL Server Backup history.
So now i am in talks with my server/backup admin see if we can do a machine level backup once a day, so that we can do a machine level restore in case of disaster, and also add a volume level backup of my Backup drive, so that he captures my backups more frequently during the day. I think once this is complete, we'll have the best of both backup solutions.
Ability to do machine level restores (Rapid Recovery)
Retention of SQL Server Backups (Rapid Recovery)
Flexible point in time recovery (SQL Server)
Just wanted to post a update to this issue
We scheduled a call with Quest yesterday, and they assured me that their backup is only taking volume snapshots and will have no impact on my SQL backups. They said that the reason that i was seeing these errors is likely do to Rapid Recovery(I guess AppAssure was renamed to Rapid Recovery) and my SQL backup jobs both trying to use the volume shadow copy service at the same time and so we simply needed to stagger the backup jobs. I end up finding out that this wasn't completely true because the Rapid Recovery backups were configured to truncate my SQL Logs. I also told the guy from rapid recovery that when i queried the msdb backupset table that i was seeing backup jobs listed that lined up with the times of the rapid recovery backup. Still he assured me that it would have zero impact on my backups.
I was still concerned that the Rapid Recovery backups might be impacting the chain of my backup files, so in our test environment i right clicked on one of our databases and clicked tasks > restore >database just to see the recovery history. I see a database backup listed as type Full (Copy Only) which coincides with the rapid recovery backup, then a number of my transaction log backups.
It would seem to me that Rapid Recovery is definitely impacting my SQL Backups.
One other thing to note that i just tried in out test environment. I did a few tests backups using Full's , Transaction Log, Differentials and Full (Copy Only) just to see how things showed up in the restore window in SQL Server Management Studio.
So i realized that in the default restore screen, it tries to recover to the nearest point in time using the least combination of various backup files. To go past the last full backup, I have to use the Timeline option.
I could see the Full backup as the first item, and then the transaction logs backups.
Once i do the Differential backup, i see the Full plus the differential, but no more transaction log backups.... which makes sense since it's trying to get the the closest time possible for the recovery.
Next if i do another Transaction Log backup, I see the Full, Differential and the Transaction Log back
One thing that surprised me however is if i do a Full (Copy Only) followed by a transaction log backup, that i see these two item in the list of files for the recovery, yet if i do a differential after a Full (Copy Only) that it show me the last Full (non copy only), plus the differential. I expected that the backups would always be based off of the last full, both for transaction log and differential backups. I thought that the Copy Only backups would be ignored in the backup chain.
Next I decided to use the timeline restore feature and select a point in time during my tests where the Rapid Recovery backup wasn't part of the listed backups, and do a verify backup. As expected it was successful. After this i tried a restore to another point in time where the Rapid Recovery Full (Copy Only) backup was listed and the verify failed on the Full (Copy Only) backup file from rapid recovery, since it doesn't exist on the sql server.
Any recommendations on how to fix this? The point of the Rapid Recovery backup is supposed to be to backup the machine, in case we ever loose the server and have to restore the entire server, plus for it to pick up my sql server backups to keep for retention, since i only keep 7 days worth of backup on the server itself.
You can check the error log file.
Your job may be the victim of deadlock transactions.
Backup file location may not have enough permission.
So, you should check all the possible error log files.
I'm using Microsoft SQL server and storing lot of data daily. I'm taking full backup daily which takes more than 5 hours to complete. Is there any idea to complete my backup process within a hour ? An alternative things to do?
Daily differential backup (only changes will be backed up) and CloudBerry software will make you deal:)
Definitely consider taking differential backups. Depending on how much data is changing you may be able to take differential backups every hour or so, giving you a quick way to restore the hourly backup if needed. Perhaps complete a full backup once a week or every few days.
There are several considerations and I suggest you research it well to see if it will meet your specific needs, for example (source):
If your databases are small or compress effectively enough so that your full and transaction log backups fall within storage and SLA limits, differential backups are unnecessary.
If your databases change a lot between backups, you might as well perform full backups.
If the changes to your database are few and the transaction log backups would take longer to restore than the differential backups, using differential backups might make sense and are worth investigating.
Other things to consider:
Are you currently taking transaction log backups? Should you be? What recovery model are you using?
If you decide to do differential backups, be aware of things such as the "Copy only backup" option in SQL Server Management Studio. These have implications for restoring data in a disaster recovery situation.
In essence, you should educate yourself on SQL Server backup and restore before you make any changes.
To improve the situation and to speed the backup you may consider differential and transaction log backups. You can do it through the studio, but to schedule it you will have to refer to a 3-rd party tool like CloudBerry.
The alternate to taking full backups daily is taking incremental/ progressive incremental or reverse incremental backups.
An incremental backup copies the changes only from the last successful backups.
Progressive incremental has only on one full backup copy for the servers and takes incremental backups for the rest of cycle.
Reverse incremental supported by Veeam and DPM has its own mechanism to backup and store only incremental copies from the server, and store each copy as a full copy which allows quick restoration in case of disasters.
To configure this kind of backups, you can go for any enterprise level backup tool, but if you focus on cost effectiveness I would recommend Cloudberry backup.
I tried CloudBerry Cross-Platform Cloud Backup which provides a simple GUI to manage backup and restores and cloud storage account comes bundled with the software.You can also use the cloudberry explorer to view, move and manage your data on the cloud storage account.
It also provides enterprise backup features like job scheduling, CLI, compression and encryption.
I have SQL Server 2012, and a Backup drive apart, but it is really full. I want to move all the backups, after done, to a Network Drive. My concern is with the differential and log backups, as I don't know if they need to see the full backup to be done, or how does it works in SQL Server.
Is it safe to move the Full Backup to another drive, and then execute the daily differential backups?
Let us remember the golden rule for restoring first.
‘After restoring full database backup, restore latest differential database backup and all the transaction log backup after that to get database to current state.’
So if you have only one full backup and then keep on taking differential backup's the size of the differential backup will keep on growing everyday. In case of disaster you may have to find the older full backup and then restore the latest differential backup.
So in theory: If you take one full backup and move to somewhere else and keep on taking differential backup works.
However, there are changes that your full backup may get corrupted or your differential backup have issues. You will in that case miss another full backup.
It is always a good idea to take full backup at regular intervals. I take full backup of my databases every day when they are relatively small and every other day when they are very large.
Here is the blog post where I have written about backup timeline.
https://blog.sqlauthority.com/2009/07/14/sql-server-backup-timeline-and-understanding-of-database-restore-process-in-full-recovery-model/
To answer the question you asked, the SQL Server doesn't access the full backup itself when it makes differential and log backups. It instead keeps track of what needs to be backed up in the database itself.
The short version of that for differential backups is "the notion of what's changed (and therefore needs to be backed up) is kept track of in what are called differential change map pages" and for log backup is "the notion of what's changed (and therefore needs to be backed up) is kept track of in the transaction log itself".
The longer (and IMO more interesting) version is this excellent article by Paul Randal.
To recover, you need to restore the full backup before the differential. The normal recovery sequence is to restore the latest full backup, latest differential backup after the full backup, and log backups since the differential in sequence.
The backup files obviously need to be available in order to restore regardless of where you store them. If you were to lose the local full backup, you could not restore at all. You might consider copying the full backup to the network share as an extra layer of safety.
I have been building an application to Backup SQL Databases using VDI using SMO.
I have a doubt whether i should be capturing the last backup time of a database along with the LSN or simply, just capture the LSN.
More precisely, i wanted to take Last Backup Time to know that if end-user is using another software to backup their SQL DB's using Copy-only method since copy-only method doesn't update LSN or truncate anything but does update the Last backup time.
So, if i am ignoring the last backup time information of any DB and simply capturing the LSN, is it going to cause any problems for me? in terms of recovery/backup if there are other 3rd party softwares into play?
:::: Edit :::::
I am a backup application developer and i would like to know, should i be capturing the Last Backup time of the database that my end-user would like to backup ? or just simply capture the LSN to maintain the Log chain?
If you're writing your backup software for the purposes of ensuring recoverability (that is you're not taking copy-only backups yourself), don't worry about any copy-only backups that have been taken by others. As you've said, copy-only backups don't affect the backup chain.
The only thing that would mess you up is if someone else is taking a non-copy-only backup interleaved with yours. In that case, you could get into a situation where one of their backups relies on one of yours (or vice versa) and the restore scenario is complicated at a point where it really shouldn't be (i.e. the database is down and you need to recover it).
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx