I want to transfer multiple tables and their data from one SQL Server to another SQL Server on the local network automatically every 1 hour.
There is a tool built-in tool in SSMS to do this.
In SSMS, right click on the database name. Select Data > Import on the destination database. You will be prompted to provide connection information for the source database. This is internally using SSIS integration tool.
Create a SSIS Package USE Type two SCD if you want insert and
updates. you can use staging table from source to
destination is a good practice and it is industry standard. if you
are not having staging environment. You can use temp tables within
ssis package to achieve the same.
Schedule a job and run that ssis package in the job for every half an hour
Related
For my .NET Application I have an Excel file that is used as the data source. This Excel file is updated every hour. My current back-end database technology is MS Access. The Access file has a linked table to the mentioned Excel file. I would like to switch over my back-end to SQL Server.
My question is how can I create a linked table in SQL Server such that I always have the current data in my SQL Server database. Right now I have managed to import the Excel sheet through the import wizard in SQL Server Mgmt. Studio. But obviously this is static (not updated hourly like the excel sheet).
Can I create some sort of procedure that runs every hour, reads the Excel sheet and import to my SQL Server database? How would I go about doing that? Or any other method as long as my table in SQL Server gets updated on an hourly basis with data from the Excel sheet.
One, do not code read "in process" from the excel file. That is a disaster waiting to happen. Some user opens the file, locks it, and now your sql-server starts failing. :) That is a bad scenario.
You should create an ETL package to run (hourly)?
https://learn.microsoft.com/en-us/sql/integration-services/ssis-how-to-create-an-etl-package?view=sql-server-ver15
What is SQL Server Integration Services (SSIS)? MicrosoftSQL Server
Integration Services (SSIS) is a platform for building
high-performance data integration solutions, including extraction,
transformation, and load (ETL) packages for data warehousing. SSIS
includes graphical tools and wizards for building and debugging
packages; tasks for performing workflow functions such as FTP
operations, executing SQL statements, and sending e-mail messages;
data sources and destinations for extracting and loading data;
transformations for cleaning, aggregating, merging, and copying data;
a management database, SSISDB, for administering package execution and
storage; and application programming interfaces (APIs) for programming
the Integration Services object model.
data sources and destinations for extracting and loading data;
Deploy and Schedule:
https://www.mssqltips.com/sqlservertutorial/9069/deploy-and-schedule-an-sql-server-integration-services-ssis-package-step-by-step/
Scheduling the SSIS Package with SQL Server Agent
Manually executing packages is one thing, but normally you will schedule packages so your ETL can run in a specific time windows
(probably at night). The easiest option is SQL Server Agent. You can
right-click on the Jobs node to create a new job:
I have an automated SSRS report that runs off a SQL Server SP. A subscription then exports this to a folder each morning. I'm not sure if this next step is possible, but maybe someone knows:
Another Excel report from an outside source will be dropped into the same folder each morning. Is it possible to then automatically compare the two reports and delete records from my automated report that do not match?
I know I can manually import the new Excel file into SQL Server, do a join and then delete the records. But is it possible to do this automatically?
This sounds like a job for SSIS. SSIS is designed to handle ETL workflows like this. I would create a package that pulls in the outside Excel and either the output from your SSRS subscription or preferably calls the stored procedure itself. You can do all the data comparison you need and then output in whatever format you need or run T-SQL statements against your source database based on the results of your comparison as appropriate to your use case. You'd deploy your package to a SQL Server setup for SSIS and create a SQL Agent job to run it at the appropriate interval.
I have an Oracle database and a SQL Server database. There is one table say Inventory which contains millions of rows in both database tables and it keeps growing.
I want to compare the Oracle table data with the SQL Server data to find out which records are missing in the SQL Server table on daily basis.
Which is best approach for this?
Create SSIS package.
Create Windows service.
I want to consume less resource to achieve this functionality which takes less time and less resource.
Eg : 18 millions records in oracle and 16/17 millions in SQL Server
This situation of two different database arise because two different application online and offline
EDIT : How about connecting SQL server from oracle through Oracle Gateway to SQL server to
1) Direct query to SQL server from Oracle to update missing record in SQL server for 1st time.
2) Create a trigger on Oracle which gets executed when record is deleted from Oracle and it insert deleted record in new oracle table.
3) Create SSIS package to map newly created oracle table with SQL server to update SQL server record.This way only few records have to process daily through SSIS.
What do you think of this approach ?
I would create an SSIS package and load the data from the Oracle table use a Data Flow / OLE DB Data Source. If you have SQL Enterprise, the Attunity Connectors are a bit faster.
Then I would load key from the SQL Server table into a Lookup transformation, where I would match the 2 sources on the key, and direct unmatched rows into a separate output.
Finally I would direct the unmatched rows output to a OLE DB Command, to update the SQL Server table.
This SSIS package will require a lot of memory, but as the matching is done in memory with minimal IO, it will probably outperform other solutions for speed. It will need enough free memory to cache all the keys from the SQL Server Table.
SSIS also has the advantage that it has lots of other transformation functions available if you need them later.
What you basically want to do is replication from Oracle to SQL Server.
You could do this in SSIS, A windows Service or indeed a multitude of platforms.
The real trick is using the correct design pattern.
There are two general design patterns
Snapshot Replication
You take all records from both systems and compare them somewhere (so far we have suggestions to compare in SSIS or compare on Oracle but not yet a suggestion to compare on SQL Server, although this is valid)
You are comparing 18 million records here so this is a lot of work
Differential replication
You record the changes in the publisher (i.e. Oracle) since the last replication then you apply those changes to the subscriber (i.e. SQL Server)
You can do this manually by implementing triggers and log tables on the Oracle side, then use a regular ETL process (SSIS, command line tools, text files, whatever), probably scheduled in SQL Agent to apply these to the SQL Server.
Or you could do this by using the out of the box replication capability to set up Oracle as a publisher and SQL as a subscriber: https://msdn.microsoft.com/en-us/library/ms151149(v=sql.105).aspx
You're going to have to try a few of these and see what works for you.
Given this objective:
I want to consume less resource to achieve this functionality which takes less time and less resource
transactional replication is far more efficient but complicated. For maintenance purposes, which platforms (.Net, SSIS, Python etc.) are you most comfortable with?
Other alternatives:
If you can use Oracle gateway for SQL Server then you do not need to transfer data and can make the query directly.
If you can't use Oracle gateway, you can use Pentaho data integration or another ETL tool to compare tables and get results. Is easy to use.
I think the best approach is using oracle gateway.Just follow the steps. I have similar type of experience.
Install and Configure Oracle Database Gateway for SQL Server.
https://docs.oracle.com/cd/B28359_01/gateways.111/b31042/installsql.htm
Now you can create a dblink from oracle to sql server.
Create a procedure which compare the missing records in oracle database and insert into sql server database.
For example, you can use this statement inside your procedure.
INSERT INTO "dbo"."sql_server_table"#dblink_name("column1","column2"...."column5")
VALUES
(
select column1,column2....column5 from oracle_table
minus
select "column1","column2"...."column5" from "dbo"."sql_server_table"#dblink_name
)
Create a scheduler which execute the procedure daily.
When both databases are online, missing records will be inserted to sql server. Otherwise the scheduler fail or you can execute the procedure manually.
It takes minimum resource.
I will suggest having a homemade ETL solution.
Schedule an oracle job to export source table data (on a daily
manner based on the application logic ) to plain CSV format.
Schedule a SQL-Server job (with acceptable delay from first oracle job) to read this CSV file and import it
to a medium table inside sql-servter using BULK INSERT.
Last part of the SQL-Server job will be reading medium table data
and do the logic(insert, update target table). I suggest having another table to store reports of this daily job result.
I have database in MS Access in which new table is generated to store the records of each day. I want to transfer the data from the current day table to SQL Server Table.
In SQL server I have only one table and Data will append everyday.
What are the possible ways to achieve this functionality.
Thanks
There are many ways you can do this.The one is Import Export Wizard which is listed below:
1) Using Import-Export wizard: This will do is get the data from source (for you will be Access) and put it in Destination database ( which will be SQL Server)
You can call this package through SQL Agent ( you can make it run every day, weekly, monthly)
Import and Export wizard
http://www.packtpub.com/article/transferring-data-ms-access-2003-to-sql-server-2008
I would explore creating a Linked Server to Access, and create a stored procedure to load the data, which you could schedule with the Windows Task Scheduler and sqlcmd.
Taken from MSDN post
http://social.msdn.microsoft.com/forums/sqlserver/en-US/015d8295-a57a-4dc9-b3e2-6da1cfe8ab6d/convert-an-access-db-to-sqlexpress
Access has a tool called the Upsize Wizard built directly into the product to help you do this. A second alternative is the SQL Server Migration Assistant for Access, which you can learn about here http://technet.microsoft.com/en-us/library/hh313039.aspx.
Download link below
http://www.microsoft.com/en-in/download/details.aspx?id=28763
I need to copy some records from our SQLServer 2005 test server to our live server. It's a flat lookup table, so no foreign keys or other referential integrity to worry about.
I could key-in the records again on the live server, but this is tiresome. I could export the test server records and table data in its entirety into an SQL script and run that, but I don't want to overwrite the records present on the live system, only add to them.
How can I select just the records I want and get them transferred or otherwise into the live server? We don't have Sharepoint, which I understand would allow me to copy them directly between the two instances.
If your production SQL server and test SQL server can talk, you could just do in with a SQL insert statement.
first run the following on your test server:
Execute sp_addlinkedserver PRODUCTION_SERVER_NAME
Then just create the insert statement:
INSERT INTO [PRODUCTION_SERVER_NAME].DATABASE_NAME.dbo.TABLE_NAME (Names_of_Columns_to_be_inserted)
SELECT Names_of_Columns_to_be_inserted
FROM TABLE_NAME
I use SQL Server Management Studio and do an Export Task by right-clicking the database and going to Task>Export. I think it works across servers as well as databases but I'm not sure.
An SSIS package would be best suited to do the transfer, it would take literally seconds to setup!
I would just script to sql and run on the other server for quick and dirty transferring. If this is something that you will be doing often and you need to set up a mechanism, SQL Server Integration Services (SSIS) which is similar to the older Data Transformation Services (DTS) are designed for this sort of thing. You develop the solution in a mini-Visual Studio environment and can build very complex solutions for moving and transforming data.