About DocumentDB manual snapshot AWS official documentation says Full backups — When a manual snapshot is taken, a full backup of your cluster's data is created and stored.
So it means, just as an example I have created 5 snapshots of a cluster ( 1 snapshot per day ) named ss_day_1, ss_day_2,ss_day_3,ss_day_4,ss_day_5. After that if I delete ss_day_1, ss_day_2,ss_day_3,ss_day_4 then I will be able to restore whole database ( which contains data from day1 to day 5 ) from ss_day_5 snapshot.
Am I right? or do I have to keep previous snapshots to restore the database?
It will be so kind of you if you clear my doubt. Thanks in advance.
When you restore from a manual snapshot, you restore to when the manual snapshot was taken, it will be the database image at that time. In your case, if you keep just ss_day_5, it will not contain the image of the data that was in ss_day_1.
If you need to keep 5 days of backups, it is better for you to use the automatic backup feature and choose a retention of 5 days. Automatic backups are continuous and incremental, there's a full snapshot at the beginning of the retention period and then only changes are backed up. This way it will use less backup space, lower cost for your cluster. And because these changes are streamed all the time, you can also do point in time restore (PITR) up to the exact second in the past (within the retention period), which you can't do with manual snapshots.
Check docs for the comparison between the two backup methods.
Related
I was curious on other's Snowflake backup archival/PITR strategy on how you keep your Snowflake database backups past the maximum 97 backup time (assumes Enterprise edition, 90 days for time-travel and 7 extra days for fail-safe) for a database's point-in-time-recovery (PITR)?
97 days is a lot of time for PITR but problems are not always caught until its too late in my experiences thus the need for backups past the max 97 days.
Example Scenario
The RPO is 12 months for a business, my initial train of thought to meet this RPO is:
Setup a task to create a clone of the database on day 97 (via maximum restore time supported by Snowflake)
The clone is created and the snapshot of the previous 97 days ensures PITR capability for the database (as time-travel is now being overwritten)
Another 97 days pass, the previous clone of the database is backed up "physically" to an external storage location (AWS/Azure/GCP) to archive the backup. Once the database backup is archived, the previous clone is dropped and another clone of the database is created to backup the previous 97 days (like step 2).
Repeat the process to ensure PITR and business as usual in case of DR
I might not 100% understand how time-travel/fail-safe "resets" when it reaches its maximum age for a database but i if understand correctly how time-travel resets, I am curious on others thoughts and different methods on how this would be achieved?
Also if there is a current snowflake best practice for this type of backup archival/PITR strategy, I'd be more than happy to hear it.
If you need to keep 12 months of historical data and need to be able to restore data using the Point in Time Recovery (PITR) approach,
then I see 2 alternatives, which actually can be implemented in parallel without affecting each other:
to take a snapshot of an ORIGINAL object using Zero Copy Clone every day (365 days a year; similar to taking daily a snapshot (files in S3) in Redshift). But there might be a limit to how many snapshots of ORIGINAL object could be taken over the time period (TBD in Question #2 above). So, in this case we would have 365 + 90 days of Time Travel (Customer controlled) + 7 days of Disaster recovery (Snowflake Admin controlled);
to backup daily SnowFlake data to S3 bucket - to use COPY INTO command.
I've confirmed with Snowflake You can backup the Original source as many times as you want using Zero Copy Clone. There is a soft limit of number of databases that can be created in an account, which is in the thousands.
Also note, Time Travel cannot be used on the CLONED object to restore data as it was on the original object before the cloned object was created. Though, you can use Time Travel on the CLONED object after it was created to keep tack of feature changes.
Our database is MSSQL and we are currently using High Availability Group with fail over in multiple node cluster so the idea of redundancy and backup is already there.
All of our current servers are in the same location; imagine a scenario where an earthquake takes out the entire hosting facility then we are sitting duck.
I'm exploring a disaster recover (DR) strategy to have another DR backup at a different location so when this happens I can bring back the entire database using the DR's backup set with a minimum down time and the data needs to be guarantee to be up to the minutes if possible.
I've read around the Microsoft doc but I don't really see one that talks about this in details.
I need a true backup that is up to the minutes, do I need to do this full backup (once every day) along with transaction log backup (one every minute) and then save it to the other different geological location? Can you point me to a guide or best practice documentation on how to achieve this?
I'm exploring a disaster recover (DR) strategy to have another DR backup at a different location so when this happens I can bring back the entire database using the DR's backup set with a minimum down time and the data needs to be guarantee to be up to the minutes if possible
Following are your options, since you already have Availability Groups in-place.
Multi sub-net WSFC You need to add additional node into WSFC, that work as DR Replica in availability group, consider it's another copy of secondary that you already have in same location but this copy laying different geographical location, since it's multi sub-net WSFC required attention for care-full Quorum configuration.
Log-Shipping it's simple solution compare to Multi sub-net WSFC and easy to administrate. It basically takes log backups on scheduled basis from Primary replica in you current Availability Group, and restores into secondary replica. You can have multiple secondary replicas and each of them different geographical locations depending on network bandwidth.
I need a true backup that is up to the minutes, do I need to do this full backup (once every day) along with transaction log backup (one every minute) and then save it to the other different geological location? Can you point me to a guide or best practice documentation on how to achieve this?
This post at DBA.SE would help you..
I am working with a data warehouse with SQL Server 2012 and was wondering what would be the most optimized, automated procedure for a backup/restore strategy.
Current observations and limitations:
1) Cannot use transaction logs as it would affect my load performance - datasets are potentially huge with large transactions
2) Current plan is to do full backup every week and differential backup every day
I am not sure when DML operations will happen as it depends on my application's usage, but is there a way to just track the NUMBER of changes to a database that would trigger a differential backup? A way that would not affect performance? I do not want to be taking unnecessary differential backups.
Would Change tracking be a good solution for my scenario? Or would there be overhead involved? I do not need to know the actual data that was changed, just the fact that it was changed by a certain amount.
Thanks in advance!
Well, there's this ( http://www.sqlskills.com/blogs/paul/new-script-how-much-of-the-database-has-changed-since-the-last-full-backup/ ). I'm just trying to figure out what problem you're trying to solve. That is, if you find that the size is below some threshold, it will be (by definition) cheap to do.
It all depends on your DWH configuration.
1. Is your DWH database partitioned? If yes, It would be easier to do the daily db backup(diff backup) for the current partition ONLY. It's much more smaller set of data to be backed up.
If not, Current plan is to do full backup every week and differential backup every day is the only way since you cannot use transaction log file.
You could also try 3rd party disk (block) level backup software (i.e. Doubletake)....
Hope it helps.
You seem to have a mistaken notion of what a differential backup is. Don't worry; it's common.
When you say things like "track the number of changes to a database that would trigger a differential backup", it implies that you think that a differential backup gets all of the changes since the latest full or differential.
However, a differential backup gets all of the data that has changed since the last full backup only. So, you'd expect the size of subsequent differential backups to get larger and larger. For example, let's say you take a full backup on Sunday and a differential backup every other day. You'd get something like:
Monday: All of the data changed since Sunday's backup.
Tuesday: All of the data changed since Sunday's backup (including Monday's data)
Wednesday: All of the data changed since Sunday's backup (including Tuesday's data)
etc
Additionally, you'd only ever restore at most one differential backup if/when you need to restore your database. For instance, if your database crashed on right before Thursday's backup, you'd restore your last full backup (from Sunday in my example), then Wednesday's differential, and you're done.
As for when to schedule it, that's typically dictated by the rhythm of your business. For instance, you might decide to take a backup just before you kick off your ETL or just after. Doing it during doesn't make much sense as you'd have an inconsistent (with respect to your ETL process) database if you ever need to restore it.
Most of the time users will hit the database to read news. There are very few number of queries executed under transactions. 95% of the database hits would be for read-only purposes.
My database log files size is growing 1 GB per day. Even if I shrink the database, the log file size is not decreasing. What could be the reason for growing the log file size more and more? How can I control this? As per my knowledge log file does not increase when we read data from tables.
Any suggestions on how to deal with the log file growing? How can it be kept a manageable or reasonable size? Does this effect performance in any way?
There are couple of things to consider. What type of backups you do, and what type of backups you need. If you will have the answer to this question you can either switch Recovery Mode to simple or leave it full but then you need to make incremental backups on a daily basis (or whatever makes you happy with log size).
To set your database logging to simple (but only if you do Full Backups of your database!).
Right click on your database
Choose Properties
Choose Options
Set Recovery mode to simple
This will work and is best if your backup schedule is Full Backup every day. Because in such scenario your log won't be trimmed and it will skyrocket (just like in your case).
If you would be using Grandfather&Father&Son backup technique, which means Monthly Full backup, Weekly Full backup, and then every day incremental backup. Then for that you need Full Recovery Mode. If 1GB of log per day is still too much you can enable incremental backup per hour or per 15 minutes. This should fix the problem of log growing more and more.
If you run Full Backup every day you can switch it to simple recovery mode and you should be fine without risking your data (if you can leave with possible 1 day of data being lost). If you plan to use incremental then leave it at Recovery Mode Full.
Full backups will not help, you must regularly backup the transaction log (as well as the regular database full and differential backups) for it to be emptied. If you are not backing up the log and you are not in simple recovery mode, then your transaction log has every transaction in it since the database was set up. If you have enough action that you are growing by a gig a day, then you may also have large imports or updates affecting many records at once. It is possible you need to be in simple recovery mode where transactions are not recorded individually. Do NOT do that however if you have a mix of data from imports and users. In that case you need to back up the transaction log frequently to be able to keep the size manageable and to a point in time. We backup our transaction log every 15 minutes.
Read about transaction log backups in BOL to see how to fix the mess you have right now. Then get your backups set up and running properly. You need to read and understand this stuff thoroughly before attempting to fix. Right now, you would probably be in a world of hurt if your server failed and you had to recover the database. Transaction log backups are critical to being able to recover properly from a failure.
Do you backup your database frequently? You need to perform full- and/or transaction log- backups in order for SQL Server to consider shrinking your log file.
The major reason for the log file large size is due to bulk transaction in DB. To reduce the log file size best option to take the transaction log backup after certain interval of time.
I have a SQL Server 2005 database that is backed up nightly. There backup consists of:
FULL backup of the database.
backup of the transaction log.
These are currently two separate jobs.
The log is huge and I'd like to set things up so that:
the database is backed up in full nightly
the log is set such that I can recover the database from any point between one backup and the next.
How can I set this up so that the log files are manageable? I suspect that the log has never been shrunk, as the log is huge.
You are currently implementing the FULL Recovery Model from the sound of things. This will allow you to restore to a point in time provided that you have a transaction log backup that covers the desired point in time (post full backup).
In order to reduce the size of your required transaction log file, you should look to increase the frequency of your transaction log backups. I would suggest hourly. Once you have gauged the actual usage of your log file, you can then look to shrink it to a more suitable size. The key point to note here is that once a transaction log backup has been completed, the inactive portion of the log file becomes available for use once again. The reason why a transaction log file grows continuously is if the transaction log backups are either, not being taken at all or their frequency is not sufficient.
I would also suggest that you consider performing a mix of DIFFERENTIAL and FULL Backups in order to reduce the collective size of your backed up data. An example schedule would be a weekly FULL Backup, say every Sunday, with daily DIFFERENTIAL backups.
I hope what I have detailed makes sense. Please feel free to contact me directly and I will happily assist you in deploying an appropriate backup strategy for your environment.
Essential References:
How to stop the transaction log
file from growing enexpectedly
Backup and Restoring Databases in
SQL Server
One of the things I find with backups is that people typically don't run them frequently enough - especially when it comes to log file backups. And it sounds like you're correct, that the log file isn't being truncated regularly (which means you're likely wasting premium disk space [1]). More importantly though, that's leaving you completely exposed from a recoverability standpoint.)
Happily though, getting things up and running as you need them isn't so hard. In fact, I'd recommend the following three videos as they should give you the background info you need, and then the step-by-step instructions you'll want to follow to get everything working correctly:
http://www.sqlservervideos.com/video/logging-essentials
http://www.sqlservervideos.com/video/sql2528-log-files
http://www.sqlservervideos.com/video/sqlbackup-best-practices
1 Maximize Storage Performance: http://www.sqlmag.com/Article/ArticleID/100893/sql_server_100893.html
What you are doing is effectively a SIMPLE mode backup with bonus disadvantage of not shrinking the log. There is no point to back up both at the same time. If you're doing a full backup, you can just truncate the log.
If you're going to be able to restore to any point of time, you will have to do a full backup once a day (say) and back up the log few times during the day. See http://msdn.microsoft.com/en-us/library/ms191429(SQL.90).aspx