I'm new to SQL Server replication options.
I want to set up a system such that a backup database at a remote location is used for data analysis over some large set of data in the primary database. The analysis does not need to have access to live data, and I want to run this analysis daily.
I don't have access to transaction logging on the primary SQL Server 2008 database.
What is the best way to synchronize a primary database to a secondary one in SQL Server without using transaction logging?
Is there maybe another option I'm not seeing?
Thanks,
You could do one of the following:
Some sort of automated backup, copy and restore.
Use a SSIS job to transfer the data from one to the other.
Use a linked SQL Server, and copy/update as appropriate.
The best solution though, is to use log shipping, scheduled once a day, if you can sort out your access to the primary database.
Related
I'm setting up an Azure SQL DB for our Web App. We have enabled Point In Time Retention (PITR) and Long Term Retention (LTR). Our process is to keep backups for 1 year.
Periodically, we need to upgrade the DB by applying SQL scripts. Sometimes there is a problem with the upgrade scripts and the upgrade fails. We need to rollback the database to the previous version.
To rollback the DB I tried the restore feature. However, the restore feature seems to only create new DBs; therein lies the problem. Restoring to a new DB and removing the old one works great, but we lose all our backup history. It appears backups are tied to the DB (probably to the ResourceId).
So, how can I use Azure SQL DB and periodically restore a DB and still maintain all the back up history?
Unfortunately, restoring from a backup in Azure SQL Database always creates a new database. The secret here may be to rename the newly restored database with the name of the original database. You will even see that the restored database once renamed it then shows all the security recommendations, automatic tuning recommendations of the original database.
So delete existing database, restored the database, and rename it as the original database.
You can reference this document Recover an Azure SQL database using automated database backups , it gives the answer that all the recover are creating new database.
By default, SQL Database backups are stored in geo-replicated blob storage (RA-GRS). The following options are available for database recovery using automated database backups:
Create a new database on the same SQL Database server recovered to a specified point in time within the retention period.
Create a database on the same SQL Database server recovered to the
deletion time for a deleted database.
Create a new database on any SQL Database server in the same region
recovered to the point of the most recent backups.
Create a new database on any SQL Database server in any other region
recovered to the point of the most recent replicated backups.
If you configured backup long-term retention, you can also create a new database from any LTR backup on any SQL Database server.
improtant:
You cannot overwrite an existing database during restore.
"So, how can I use Azure SQL DB and periodically restore a DB and still maintain all the back up history?"
You can use Database replacement:
If the restored database is intended as a replacement for the original database, you should specify the original database's compute size and service tier. You can then rename the original database and give the restored database the original name using the ALTER DATABASE command in T-SQL.
Hope this helps.
I have an Oracle database and a SQL Server database. There is one table say Inventory which contains millions of rows in both database tables and it keeps growing.
I want to compare the Oracle table data with the SQL Server data to find out which records are missing in the SQL Server table on daily basis.
Which is best approach for this?
Create SSIS package.
Create Windows service.
I want to consume less resource to achieve this functionality which takes less time and less resource.
Eg : 18 millions records in oracle and 16/17 millions in SQL Server
This situation of two different database arise because two different application online and offline
EDIT : How about connecting SQL server from oracle through Oracle Gateway to SQL server to
1) Direct query to SQL server from Oracle to update missing record in SQL server for 1st time.
2) Create a trigger on Oracle which gets executed when record is deleted from Oracle and it insert deleted record in new oracle table.
3) Create SSIS package to map newly created oracle table with SQL server to update SQL server record.This way only few records have to process daily through SSIS.
What do you think of this approach ?
I would create an SSIS package and load the data from the Oracle table use a Data Flow / OLE DB Data Source. If you have SQL Enterprise, the Attunity Connectors are a bit faster.
Then I would load key from the SQL Server table into a Lookup transformation, where I would match the 2 sources on the key, and direct unmatched rows into a separate output.
Finally I would direct the unmatched rows output to a OLE DB Command, to update the SQL Server table.
This SSIS package will require a lot of memory, but as the matching is done in memory with minimal IO, it will probably outperform other solutions for speed. It will need enough free memory to cache all the keys from the SQL Server Table.
SSIS also has the advantage that it has lots of other transformation functions available if you need them later.
What you basically want to do is replication from Oracle to SQL Server.
You could do this in SSIS, A windows Service or indeed a multitude of platforms.
The real trick is using the correct design pattern.
There are two general design patterns
Snapshot Replication
You take all records from both systems and compare them somewhere (so far we have suggestions to compare in SSIS or compare on Oracle but not yet a suggestion to compare on SQL Server, although this is valid)
You are comparing 18 million records here so this is a lot of work
Differential replication
You record the changes in the publisher (i.e. Oracle) since the last replication then you apply those changes to the subscriber (i.e. SQL Server)
You can do this manually by implementing triggers and log tables on the Oracle side, then use a regular ETL process (SSIS, command line tools, text files, whatever), probably scheduled in SQL Agent to apply these to the SQL Server.
Or you could do this by using the out of the box replication capability to set up Oracle as a publisher and SQL as a subscriber: https://msdn.microsoft.com/en-us/library/ms151149(v=sql.105).aspx
You're going to have to try a few of these and see what works for you.
Given this objective:
I want to consume less resource to achieve this functionality which takes less time and less resource
transactional replication is far more efficient but complicated. For maintenance purposes, which platforms (.Net, SSIS, Python etc.) are you most comfortable with?
Other alternatives:
If you can use Oracle gateway for SQL Server then you do not need to transfer data and can make the query directly.
If you can't use Oracle gateway, you can use Pentaho data integration or another ETL tool to compare tables and get results. Is easy to use.
I think the best approach is using oracle gateway.Just follow the steps. I have similar type of experience.
Install and Configure Oracle Database Gateway for SQL Server.
https://docs.oracle.com/cd/B28359_01/gateways.111/b31042/installsql.htm
Now you can create a dblink from oracle to sql server.
Create a procedure which compare the missing records in oracle database and insert into sql server database.
For example, you can use this statement inside your procedure.
INSERT INTO "dbo"."sql_server_table"#dblink_name("column1","column2"...."column5")
VALUES
(
select column1,column2....column5 from oracle_table
minus
select "column1","column2"...."column5" from "dbo"."sql_server_table"#dblink_name
)
Create a scheduler which execute the procedure daily.
When both databases are online, missing records will be inserted to sql server. Otherwise the scheduler fail or you can execute the procedure manually.
It takes minimum resource.
I will suggest having a homemade ETL solution.
Schedule an oracle job to export source table data (on a daily
manner based on the application logic ) to plain CSV format.
Schedule a SQL-Server job (with acceptable delay from first oracle job) to read this CSV file and import it
to a medium table inside sql-servter using BULK INSERT.
Last part of the SQL-Server job will be reading medium table data
and do the logic(insert, update target table). I suggest having another table to store reports of this daily job result.
We have transaction replication between two server (production and staging) which is running on SQL Server 2008 R2. The distributor is running on the staging server. Now my management is asking to create a copy of database on the production environment and replication few tables to that database. Is it possible to replicate twice? Could you please help me on this. If not possible through replication, is there any other way to do it?
You sure can. Create another publication, add the subset of tables to that, and subscribe the other database to the new publication. NB, this will create another copy of changes for this table at the distributor so take that into account.
I have a host on a server and that contains an SQL Server Database.
I have another server in another country and i want have a backup from the database every 5 minutes or after each transaction only insert new row to another database.
After some research i found out i can use linkedservers for this goal.
Is this procedure works for me for doing this operation?
I don't know what the linkedserver will do for you.
You are connected from both server via a vpn?
You are in different network (domain) probably?
If you are using a linked server, it means you will probably create trigger or stored proc. You will have to configure msdtc (for trigger).
You can use :
Replication
Log shipping
Custom replication process
I had to configure 2 times a replications to move the data from a server to another, than manage these data with trigger. It was easier to work on the data localy
I'm trying to backup a live database to my computer and I can't find the option to do it. I'm connecting to it using Microsoft SQL Server Management Studio 2008 R2. I'm a MySQL monkey, so I'm used to being able to backup to .sql files and move them around.
Anyone have any idea how I can create a file backup of the database? I've found the backup option which only backs up on the server, or the export, which seems to only allow a single table, or code an SQL query, which I'm not too sure on, short of putting in something like SHOW TABLES;
Anyone have any ideas? I'm limited to readonly access for various reasons, nothing bad, promise!
You will only be able to backup the database to a location the service account for SQL has access to. If you have access to a central share on the server/network that you can access and the service can, you might backup to that location and then browse from your computer to pull it down.
If you are just wanting the database structure you could script the database out to a file. This would let you save it locally. If you also want the data though doing a full backup is the quickest way I know of.
EDIT
I would use the T-SQL BACKUP comand and include WITH COPY_ONLY to backup the database, since you stated this is a "live" database. If a scheduled job is performing backups against the database and you break in to do an additional one you will effect the backup recovery chain of the database. Using the COPY_ONLY will allow you to get a backup of the database without requiring it in the event of a recovery need.
You can also create sql dumps with Management Studio.
Right-click the database and select Tasks - Generate Scripts. This will open a wizard that allows you to select what the dump should include (e.g. tables, indices, views, ...).
Make sure you set "Script Data" to true if you want your dump to include inserts.
You can enter a valid UNC path in the Backup option.