Our database is MSSQL and we are currently using High Availability Group with fail over in multiple node cluster so the idea of redundancy and backup is already there.
All of our current servers are in the same location; imagine a scenario where an earthquake takes out the entire hosting facility then we are sitting duck.
I'm exploring a disaster recover (DR) strategy to have another DR backup at a different location so when this happens I can bring back the entire database using the DR's backup set with a minimum down time and the data needs to be guarantee to be up to the minutes if possible.
I've read around the Microsoft doc but I don't really see one that talks about this in details.
I need a true backup that is up to the minutes, do I need to do this full backup (once every day) along with transaction log backup (one every minute) and then save it to the other different geological location? Can you point me to a guide or best practice documentation on how to achieve this?
I'm exploring a disaster recover (DR) strategy to have another DR backup at a different location so when this happens I can bring back the entire database using the DR's backup set with a minimum down time and the data needs to be guarantee to be up to the minutes if possible
Following are your options, since you already have Availability Groups in-place.
Multi sub-net WSFC You need to add additional node into WSFC, that work as DR Replica in availability group, consider it's another copy of secondary that you already have in same location but this copy laying different geographical location, since it's multi sub-net WSFC required attention for care-full Quorum configuration.
Log-Shipping it's simple solution compare to Multi sub-net WSFC and easy to administrate. It basically takes log backups on scheduled basis from Primary replica in you current Availability Group, and restores into secondary replica. You can have multiple secondary replicas and each of them different geographical locations depending on network bandwidth.
I need a true backup that is up to the minutes, do I need to do this full backup (once every day) along with transaction log backup (one every minute) and then save it to the other different geological location? Can you point me to a guide or best practice documentation on how to achieve this?
This post at DBA.SE would help you..
Related
I have a scenario in which I need to maintain a replica of existing database.
Is there a solution to achieve the below mentioned approach.
1. Take a full back once and restore to a destination database.
2. Scheduled( ex: Every day) differential backup(Only the data which has changed since last backup) of the source database and restore into the destination database
This is to avoid taking full backup and restore each time.
You can use Differential Backups, but you would need to ship a new Full backup periodically or the Differentials will continue to grow.
A better solution might be Log Shipping, where you can ship just the changes on whatever schedule you want.
You can consider configuring an availability group and use a secondary SQL server instance with asynchronous data sync. This should be considered only if the primary(original live SQL server) and secondary servers are in the same location\data centre. So you don't need to take backup-restore or do any extra work other than properly configuring it at the first time.
If that is not the case (copy should be available in another location\data center), it would be better to go with configuring log shipping.
First option is a lot better because it would contain the exact copy of the primary database (with a sync delay depending on various factors...probably seconds) and you can directly fail over to the secondary in case of any issues with the primary server.
Relative newcomer to sql and pg here so this is a relatively open question regarding backing up daily data from a stream. Specific commands / scripts would be appreciated if it's simple, otherwise I'm happy to be directed to more specific articles/tutorials on how to implement what needs to be done.
Situation
I'm logging various data streams from some external servers on the amount of a few GB/day every day. I want to be able to store this data onto larger harddrives which will then be used to pull information from for analysis at a later date.
Hardware
x1 SSD (128GB) (OS + application)
x2 HDD (4TB each) (storage, 2nd drive for redundancy)
What needs to be done
The current plan is to have the SSD store a temporary database consisting of the daily logged data. When server load is low (early morning), dump the entire temporary database onto two separate backup instances on each of the two storage disks. The motivation for storing a temp db is to reduce the load on the harddrives. Furthermore, the daily data is small enough that it will be able to copy over to the storage drives before server load picks up.
Questions
Is this an acceptable method?
Is it better/safer to just push data directly to one of the storage drives, consider that the primary database, and automate a scheduled backup from that drive to the 2nd storage drive?
What specific commands would be required to do this to ensure data integrity (i.e. while a backup is in progress, new data will still be being logged)
At a later date when budget allows the hardware will be upgraded but the above is what's in place for now.
thanks!
First rule when building a backup system - do the simplest thing that works for you.
Running pg_dump will ensure data integrity. You will want to pay attention to what the last item backed up is to make sure you don't delete anything newer than that. After deleting the data you may well want to run a CLUSTER or VACUUM FULL on various tables if you can afford the logging.
Another option would be to have an empty template database and do something like:
Halt application + disconnect
Rename database from "current_db" to "old_db"
CREATE DATABASE current_db TEMPLATE my_template_db
Copy over any other bits you need (sequence numbers etc)
Reconnect the application
Dump old_db + copy backups to other disks.
If what you actually want is two separate live databases, one small quick one and one larger for long-running queries then investigate tablespaces. Create two tablespaces - the default on the big disks and the "small" one on your SSD. Put your small DB on the SSD. Then you can just copy from one table to another with foreign-data-wrappers (FDW) or dump/restore etc.
I have two databases - a CRM database (Microsoft Dynamics crm) and a company database.
These two databases are different.
How to copy the company database (all objects) into CRM database every 5 seconds?
Thanks
The cheapest way to do this (and one of the easiest) is to use a method called log shipping. This can (on a schedule even every 5 minutes or so) copy the log file to another machine and from the shipped log file restore to the target data base. Please ignore geniuses that will claim it can be done every minute because it takes a little while to close the log backup file, move it and reapply but a 5-10 minute window is achievable.
You can also use mirroring,transactional replication, and other High Availability solutions but there is no easy way to keep two machines in sync.
Do you need to duplicate the data? Can;t you query the source system directly if they're on the same server?
Else this might point you in the right direction: Keep two databases synchronized with timestamp / rowversion
I've replicated a large database (close to 1TB) to three remote servers using pushing transactional replication. The subscribers are read-only. A lot of data is inserted and updated (from other sources) in one day every month. It always fail the replication after the day and we manually initialize the replication from backup every month.
Is it possible to switch to log shipping before the inserting day and switch back to transactional replication after the bulk insertions are log shipped? So there is no need to copy the big backup file for re-initialization?
No. Transactional replication is logical while log shipping is physical. You can't switch at will between the two. But if your subscribers are read only to start with then transactional replication can be replaced out of the box with log shipping, at the cost of a slight delay in updates and having to disconnect readers on the stand-by sites every time a log is being applied (usually this is nowhere near as bad as it sounds). Given how much more efficient and less problematic log shipping is compared to transactional replication, I would not hesitate for a single second in doing this replace for good.
I question your need to re-initialize on a scheduled basis. I've had replication topologies go for a really long time without having the need to re-init. And when we did, it was only because there was a schema change that didn't play nice. When you say that the large amount of data fails replication, what does that mean? Replication will gladly deliver large data changes that to the subscribers. If you're running afoul of latency limits, you can either increase those or break down large transactions into smaller ones at the publisher. You also have the option of setting the MaxCmdsInTran option for the log reader agent to have it break up your transactions for you.
I have the following scenario:
Our system is running a SQL Server Express 2005 database locally (on each users desktop, if you will). The system is storing a lot of production data from a machine. There are high demands on the safety of the data, and doing a backup each night, or even each hour is not enough. We need a backup strategy that will ensure almost instantaneous/continuous backup of the database.
Is there anyone out there that has successfully implemented a system similar to this, and/or has got some ideas of how to accomplish it? The only thing I can think of right now is to have mirrored drives (raid) to hold the data, but that would be complicated and expensive.
I would appreciate any and all thoughts on this, since it is a real issue for me and my company. Thanks in advance!
Update:
I was not clear enough in my description of the scenario. The system is storing data in a vehicle that has no connection to anything. A centralized database is therefor not possible. Neither can we use a standard/enterprise version of SQL Server, since it would be to expensive (each vehicle would need a license). Thanks for your input!
Switch your database into "Full" recovery mode. Do full backup every night and do delta backup after major user action. The delta backups can be done to the flash memory or different hard-drive, and all data can be synchronized with server when online.
Another simple way is to trace all user changes and important data in a text file that stored on a separate drive. If SQL database crashes the user or other operator can repeat steps to restore data.
One way I've seen this done is by using DoubleTake.
I will assume that a central database on a server is not feasible because your systems are running standalone and are not connected to anything. So this is what I would do
Set up RAID on the computer. This insures you against simple disk failure.
Any SQLSever database can be recovered to the point of the last commited transaction if you have a full database backup and a set of transaction logs available. Basically you simply restore the last full backup then apply the transaction logs going forward. See these links.
http://www.enterpriseitplanet.com/storage/features/article.php/11318_3776361_3
https://web.archive.org/web/1/http://blogs.techrepublic%2ecom%2ecom/datacenter/?p=132
So what you need to do is set up a periodic full backup of both the database and transaction logs, and more regular transaction log backups (and ensure that your transaction log can never run out of space).
In the event of failure you restore the last full backup, then apply the transaction logs going forward.
Myself, if these are critical systems, I would be inclined to add an additional drive to the system and make sure that the backups are copied over to that. This is because as good as raid is it does sometimes have issues - raid controllers fail, disks get wiped accidentally in parallel, disk failures go unnoticed so your just running on one disk etc. If you ensure backups are copied to a separate disk then you can always recover to the last transaction log backup. You should also ensure tape backups of course, but they are generally a last resort in the event of trouble.
If for some reason you cannot set up raid then you should still install a second disk, but place the database file on one drive and the transaction log on the other and copy backups to both disks. In the event of failure of the C drive, or some other software issue crashing the database you can still recover to the last commited transaction. Failure on the D drive limits you to the last transaction log backup (Oracle used to allow you to mirror the transaction log from the database, which again would completely cover you, but I don't think this facility exists in SQL Server)
If you are looking for a scheduler for SQL Server Express (which doesn't come with one) then I've been using SQLScheduler quite happily without problems, and it's free.
The most obvious answer would be to ditch SQL Server Express running locally and use a single source for your data (such as a standard SQL server install on a central storage location). Unless your system requires individual back ups of every single person's own individual instance of SQL Server Express.
If your requirements are so stringent as to call for instantaneous backups on every operation, you should definitely think about a different method of storage than local instances of SQL Server Express.
Wouldn't it be easier to just use one centralized SQL Server and back that up every hour or so? If you truly need instantaneous backup, your company (which seems not to want to spend money by installing Express on each machine) will need to spring for two servers and two SQL Server Enterprise licenses to implement Mirroring.
Raid isn't that expensive, but it is also not the best option. If you really want high availability data you should upgrade to sql server standard on a remote server where each user connects to and use transaction based replication to an sql server (express) instance on another machine. Raid doesn't always protect you from dataloss. If the data is that important for you then the costs should not be that much of an issue.
Update in response to the question update.
If you can't use remote servers then there a couple of options:
You write a trigger which initiates a backup script on each insert or update and stores it on a seperate harddrive.
You use raid. But beware that if the raid controller fails that you still got a problem.
RAID is not expensive. Use RAID to protect against hard drive failure. You also need monitoring though. No point in having this if you let both drives fail.
Also, implement hourly incremental backups, then daily incremental backups and finally weekly full backups.
You need all of these strategies working together because they protect against different things. RAID does not protect against human or coding errors destroying data. Hourly and weekly backups don't protect against hard drive failure.