I have this scenario which has to be solved by SSIS:
I have multiple instances of the same DB (they may be on different servers)
I have one Master DB on a different server
now I need to transfer data from these different instances to Master DB using SSIS
My question is - where exactly SSIS needs to be installed? On Master DB server? On each of the source DB servers? Both?
It must be installed on the server it will be executed from, it can be a seperate server from master or clients or anyone of them.
Note that if you want to perform Bulk insert operation, then you should install it on master server.
Check the following links to get some insights:
The Data Loading Performance Guide
SQL Sever Destination vs OLE DB Destination
Related
I have a question that might be very basic.
If I have a SSIS package where I have the Transfer DB task to transfer a database from a server with a SQL Server 2016 Instance to another server with an Instance SQL Server 2019,
Do I need to install SSIS for each instance on each server?
Do I need to create the catalog in both source and destination instances? or is it enough to create it only on the source instance? or it has to be on the destination?
Thanks!
SSIS is an ETL tool, so you'll install SSIS on the server that you want to designate as your ETL server. There are several common scenarios for where that server is located.
A stand-alone ETL server, which is neither the source nor the target of the current ETL job you're working on.
A separate ETL server that is an instance on the same server as either your current source or your current destination.
Your current source server
Your current destination server
You'll have to decide which scenario makes the most sense in your environment. At my current gig, we have a few dedicated ETL boxes, and we cluster the jobs on them by subject matter.
Previously, I worked in an environment where 10% of the ETL was importing data and 90% was manipulating it on one server. In that situation, it made sense to have the SSIS stuff all on the destination server so that we weren't moving data over the wire to transform it.
As your situation evolves, and you use more and more SSIS, you might change your mind. That's fine. It's not a trivial thing to change, but it's also not set in stone once you plunk SSIS down someplace.
This must be simple but I can't find the documentation. I have an ODBC connection to an Intersystems Cache' database that works fine in MS Access, but I can't see how to do the same in SQL Server (15.0.18930.0).
In Access, I did this: Blank Desktop Database, Create, External Data tab, ODBC Database, Link to the data source, (Select Data Source) --> Machine Data Source --> click on ODBC data source we already created. Then it gives me a list of tables to link to, and we're done.
The data source was System DSN, with various fields and a connection string like
DRIVER={InterSystems ODBC35};SERVER=xxxx;PORT=xxx;DATABASE=xxx;UID=xxx;PWD=xxx;
How is this done with SQL Server?
Thanks!
How is this done with SQL Server?
For SQL Server, write an SSIS package that uses the ODBC driver. Note that you may need to run the package in 32bit mode if the ODBC driver is 32bit. Then use that package to load a SQL Server table, or use an SSIS Data Streaming Destination to enable the SQL Server to dynamically run and read data from the SSIS package.
I want to create a job with a specific query to export some date from SQL Server to Oracle. How can I schedule this?
Thank you!
How you move forward with implementing this will depend on a variety of factors, but the following steps outline this process at a high level. Of course, you'll want to choose the configurations that best suite your specific environment and business needs.
Create an SSIS package with an OLE DB Connection Manager to SQL
Server
Add a Connection Manager for Oracle. Some options are the Microsoft
ODBC driver for Oracle or OLE DB Provider for Oracle. You can here
more about setting up connections here.
After the connections are configured, you'll want to create
components to export the data, likely using a Data Flow Task (or
multiple). How this is setup will vary greatly depending on the business needs. You can use the query mentioned in your question for the SQL command in the OLE DB source to SQL Server. I'd also recommend using stored procedures as opposed to static queries.
After the package is complete, you can add it as a job in SQL Agent
and schedule it to run at the desired frequency. Make sure that the
account the job runs under has the necessary permissions,
including on both the SQL Server and Oracle objects involved.
This is not required, but installing the SSIS catalog, deploying the projects there, and
running them from this instead of the file system will offer many
benefits. This link contains
more information on SSISDB (the catalog).
I need to build connectivity between Influx and a SQL Server database to transfer data from Influx to SQL Server periodically.
Ideally I would like to have a stored procedure in SQL Server which connects with Influx and gets the required data. This procedure will be scheduled using a SQL Server Agent Job.
Will appreciate any answers regarding the possibilities to achieve the requirement.
Start with the SSIS Data Streaming Destination. This will enable you to have a view in SQL Server that returns the results of running a package.
An SSIS package, in turn, can connect to data sources without loading drivers in the SQL Server process, or can run custom .NET code to interop, or orchestrate running external processes that generate files, and then return the file contents.
So whatever the mechanism of data extract from influxdb turns out to be, an SSIS package using the Data Streaming Destination can be an adapter for you to read that data from inside SQL Server.
What are the most reasonable ways to move table data from SQL Server to Oracle (on *nix) on a regular basis?
Using SQL Server Integration Services (SSIS) is likely your best bet. If you're not familiar with SSIS, the best way to try something out is to use the SQL Server Export Wizard and have it create an SSIS package for you. For example, if you go into SQL Server Management Studio and right-click on your database, then select Tasks->Export Data. From there click next until you get to the "Choose a Destination" step. Select the "Microsoft OLE DB Provider for Oracle" and click Properties to define your database connection. When you click through the wizard, on the Save and Execute page, make sure you check the checkbox labelled "Save SSIS Package", on the next screen specify where to save the SSIS package. Once you finish the Export Wizard, your data will have been exported and you will have an SSIS package that you can use as is, or go in and tweak it to do more specific things.
Once you have your SSIS package, you can schedule it by creating a SQL Server Agent Job.
Oracle Heterogeneous Connectivity / Database Gateways in conjunction with materialized view(s), PL/SQL, or Java
SSIS or DTS: both can be scheduled but require more than read-only access to SQL Server
Java (probably within Oracle but optionally at the OS) using ODBC or SQLJ to access SQL Server and, possibly, Oracle
SQL Server scheduled to export to CSV, Oracle scheduled to import from CSV
Any of the other ETL tools (e.g. Informatica, Cognos)
Any of the myriad languages that can access both databases (but would require maintaining a third environment to run the application within)
Scheduling:
Automatic / not reuqired with materialized views
Oracle DBMS_JOB / DBMS_SCHEDULER
OS-specific (cron, Windows Scheduled Tasks, etc)
In the case of SSIS, DTS, or CSV export, scheduled within SQL Server
Create a database link from Oracle to Sql Server (heterogeneous connectivity). You can use this link to retrieve the data from Sql Server with a simple select statement. If you want to schedule you can use a materialized view or dbms_scheduler.
An alternative is to put your data in a csv file, you can use an external table or sqlloader to load this data in the Oracle database.
Here is what I do:
Connect to SQL Server by Oracle SQL developer using this link:
https://kentgraziano.com/2013/01/14/tech-tip-connect-to-sql-server-using-oracle-sql-developer/
After you have added SQL jar, you will see SQL Server tab in the Connection window:
Then connect to the SQL Instance.
Then open SQL instance and choose database or table that you want to copy. Right click on any database/table, then click on "copy to oracle" there and choose the right user[database] at "Destination Connection Name " where you want to copy your tables.
You can also change some properties there.
Click "OK" and that's it.
Let me know in case of any issues.
You can have SQL Server interface with Oracle directly through SSIS (or DTS for 2k). It will provide ETL functionality and can be scheduled on a regular basis.
I had success creating a linked server (from within the Enterprise Manager, I think) to Oracle on the SQL Server side. Then I could use normal stored procedures on both sides to accomplish smaller data movement and updates in both directions. This approach can bypass the need to try and put something together outside the databases.
Try hard to use the latest possible Oracle client on the SQL Server side though. I recall some defects in the 10.2.0.2 client and getting the 10.2.0.4 client required your "official" Oracle registration or purchase number or something.
For big data moves (or maybe even moves/updates you want done on a daily or less frequent basis), definitely use one of the ETL tools. We had Informatica for our ETL processes, but if SSIS can pull off what you need, that's fine too.