I have been building an application to Backup SQL Databases using VDI using SMO.
I have a doubt whether i should be capturing the last backup time of a database along with the LSN or simply, just capture the LSN.
More precisely, i wanted to take Last Backup Time to know that if end-user is using another software to backup their SQL DB's using Copy-only method since copy-only method doesn't update LSN or truncate anything but does update the Last backup time.
So, if i am ignoring the last backup time information of any DB and simply capturing the LSN, is it going to cause any problems for me? in terms of recovery/backup if there are other 3rd party softwares into play?
:::: Edit :::::
I am a backup application developer and i would like to know, should i be capturing the Last Backup time of the database that my end-user would like to backup ? or just simply capture the LSN to maintain the Log chain?
If you're writing your backup software for the purposes of ensuring recoverability (that is you're not taking copy-only backups yourself), don't worry about any copy-only backups that have been taken by others. As you've said, copy-only backups don't affect the backup chain.
The only thing that would mess you up is if someone else is taking a non-copy-only backup interleaved with yours. In that case, you could get into a situation where one of their backups relies on one of yours (or vice versa) and the restore scenario is complicated at a point where it really shouldn't be (i.e. the database is down and you need to recover it).
Related
I'm using Microsoft SQL server and storing lot of data daily. I'm taking full backup daily which takes more than 5 hours to complete. Is there any idea to complete my backup process within a hour ? An alternative things to do?
Daily differential backup (only changes will be backed up) and CloudBerry software will make you deal:)
Definitely consider taking differential backups. Depending on how much data is changing you may be able to take differential backups every hour or so, giving you a quick way to restore the hourly backup if needed. Perhaps complete a full backup once a week or every few days.
There are several considerations and I suggest you research it well to see if it will meet your specific needs, for example (source):
If your databases are small or compress effectively enough so that your full and transaction log backups fall within storage and SLA limits, differential backups are unnecessary.
If your databases change a lot between backups, you might as well perform full backups.
If the changes to your database are few and the transaction log backups would take longer to restore than the differential backups, using differential backups might make sense and are worth investigating.
Other things to consider:
Are you currently taking transaction log backups? Should you be? What recovery model are you using?
If you decide to do differential backups, be aware of things such as the "Copy only backup" option in SQL Server Management Studio. These have implications for restoring data in a disaster recovery situation.
In essence, you should educate yourself on SQL Server backup and restore before you make any changes.
To improve the situation and to speed the backup you may consider differential and transaction log backups. You can do it through the studio, but to schedule it you will have to refer to a 3-rd party tool like CloudBerry.
The alternate to taking full backups daily is taking incremental/ progressive incremental or reverse incremental backups.
An incremental backup copies the changes only from the last successful backups.
Progressive incremental has only on one full backup copy for the servers and takes incremental backups for the rest of cycle.
Reverse incremental supported by Veeam and DPM has its own mechanism to backup and store only incremental copies from the server, and store each copy as a full copy which allows quick restoration in case of disasters.
To configure this kind of backups, you can go for any enterprise level backup tool, but if you focus on cost effectiveness I would recommend Cloudberry backup.
I tried CloudBerry Cross-Platform Cloud Backup which provides a simple GUI to manage backup and restores and cloud storage account comes bundled with the software.You can also use the cloudberry explorer to view, move and manage your data on the cloud storage account.
It also provides enterprise backup features like job scheduling, CLI, compression and encryption.
I am working with a data warehouse with SQL Server 2012 and was wondering what would be the most optimized, automated procedure for a backup/restore strategy.
Current observations and limitations:
1) Cannot use transaction logs as it would affect my load performance - datasets are potentially huge with large transactions
2) Current plan is to do full backup every week and differential backup every day
I am not sure when DML operations will happen as it depends on my application's usage, but is there a way to just track the NUMBER of changes to a database that would trigger a differential backup? A way that would not affect performance? I do not want to be taking unnecessary differential backups.
Would Change tracking be a good solution for my scenario? Or would there be overhead involved? I do not need to know the actual data that was changed, just the fact that it was changed by a certain amount.
Thanks in advance!
Well, there's this ( http://www.sqlskills.com/blogs/paul/new-script-how-much-of-the-database-has-changed-since-the-last-full-backup/ ). I'm just trying to figure out what problem you're trying to solve. That is, if you find that the size is below some threshold, it will be (by definition) cheap to do.
It all depends on your DWH configuration.
1. Is your DWH database partitioned? If yes, It would be easier to do the daily db backup(diff backup) for the current partition ONLY. It's much more smaller set of data to be backed up.
If not, Current plan is to do full backup every week and differential backup every day is the only way since you cannot use transaction log file.
You could also try 3rd party disk (block) level backup software (i.e. Doubletake)....
Hope it helps.
You seem to have a mistaken notion of what a differential backup is. Don't worry; it's common.
When you say things like "track the number of changes to a database that would trigger a differential backup", it implies that you think that a differential backup gets all of the changes since the latest full or differential.
However, a differential backup gets all of the data that has changed since the last full backup only. So, you'd expect the size of subsequent differential backups to get larger and larger. For example, let's say you take a full backup on Sunday and a differential backup every other day. You'd get something like:
Monday: All of the data changed since Sunday's backup.
Tuesday: All of the data changed since Sunday's backup (including Monday's data)
Wednesday: All of the data changed since Sunday's backup (including Tuesday's data)
etc
Additionally, you'd only ever restore at most one differential backup if/when you need to restore your database. For instance, if your database crashed on right before Thursday's backup, you'd restore your last full backup (from Sunday in my example), then Wednesday's differential, and you're done.
As for when to schedule it, that's typically dictated by the rhythm of your business. For instance, you might decide to take a backup just before you kick off your ETL or just after. Doing it during doesn't make much sense as you'd have an inconsistent (with respect to your ETL process) database if you ever need to restore it.
I am currently trying to backup an empty SQL Server 2008 R2 database that I designed for a project that is getting shelved for the time being. I was going through the back up procedure through the SQL Management Studio when I noticed there was an option to make a Copy Only Back Up. I looked it up to see what it was but I didn't fully understand the options I was getting.
http://technet.microsoft.com/en-us/library/ms191495.aspx
I read the entry above as well as other entries and I keep seeing the phrase "independent of the sequence of conventional SQL Server backups."
Can anyone elaborate what this statement means or more about Copy Only Backups in general? I'm not sure if it's the backup I should do in this case? (My first reaction is no)
It's a full dump of a database, where you intent to take that dump and load it into some OTHER sql server instance. e.g. It's a nice way of making a complete copy of a DB without having to take down the db, detach the db, copy the .mdf files, re-attach, etc...
Naturally, since you're not using this "backup" as an actual backup, you don't want it to interfere with your normal backup schedules, hence the copy-only functionality. It's a full backup, but will not reset the backup schedule, so your normal next incremental/snapshot backup will work as usual.
This mechanism is necessary since the built-in hotcopy/migration tools in MSSMS are basically useless and can't handle its own databases in many cases.
Normally when you take a backup, it starts (or continues, depending on the type of backup that you took) what is called a log chain. Let's say that you need a copy of your database and, for whatever reason, you can't use your normally scheduled backups for this purpose. Let's walk through the scenario where you don't use a copy_only backup
Normal full backup
A bunch of differential backups
Another full backup (to make your copy database)
More differential backups
Delete the backup from step 3 (you know... to save space)
Disaster on your actual database that necessitates restore from backup
In this case, you can only restore to the last differential backup made in step 2 because the differential backups made in step 4 depend on the full backup from step 3. Now, if the backup in step 3 were a copy_only backup, you'd be fine because you're not re-establishing a log chain (which is to say that the differential backups in step 4 depend on the full backup from step 1.
If you are creating an archive backup and continuing to back it up on the server is not a concern, then it doesn't matter whether you use it or not. It will be restorable as the database either way.
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx
I want to implement an automated backup system for my site's SQL Server 2005 DB that will backup nightly to Amazon's S3 service. But since S3 charges both for space and bandwidth used, I would like to minimize the size of the files that I transfer in. What is the best way to achieve this?
I should clarify that I'm not really talking so much about compression, which is pretty straightforward, but concerning backup strategies like whether to do differential backups all the time, whether I need to copy transaction logs, etc.
Differential backups will be smaller than full backups, of course. However, you should consider the restoration side as well. You'll need your last full backup as well as your differentials to perform the restore which can add up to a lot of bandwidth/transfer time for a restore. One option is to perform a full backup weekly and do differentials daily (or a similar type of schedule).
As for transaction logs, it depends on what granularity you're looking for in restoring your data. If restoring to the last full or differential backup is sufficient, then you don't need to worry about taking transaction log backups. If that's not the case, then transaction log backups will be necessary.
Either use a commercial product do compress the backups like Red Gate Backup Pro or just zip-compress it after you're done.
Write a .batch script or powershell script that will find the file/s created in the past day and zip them up. Then FTP or whatever you have to do.
A powershell example that I just came across.