I want to create a job with a specific query to export some date from SQL Server to Oracle. How can I schedule this?
Thank you!
How you move forward with implementing this will depend on a variety of factors, but the following steps outline this process at a high level. Of course, you'll want to choose the configurations that best suite your specific environment and business needs.
Create an SSIS package with an OLE DB Connection Manager to SQL
Server
Add a Connection Manager for Oracle. Some options are the Microsoft
ODBC driver for Oracle or OLE DB Provider for Oracle. You can here
more about setting up connections here.
After the connections are configured, you'll want to create
components to export the data, likely using a Data Flow Task (or
multiple). How this is setup will vary greatly depending on the business needs. You can use the query mentioned in your question for the SQL command in the OLE DB source to SQL Server. I'd also recommend using stored procedures as opposed to static queries.
After the package is complete, you can add it as a job in SQL Agent
and schedule it to run at the desired frequency. Make sure that the
account the job runs under has the necessary permissions,
including on both the SQL Server and Oracle objects involved.
This is not required, but installing the SSIS catalog, deploying the projects there, and
running them from this instead of the file system will offer many
benefits. This link contains
more information on SSISDB (the catalog).
Related
I have a question that might be very basic.
If I have a SSIS package where I have the Transfer DB task to transfer a database from a server with a SQL Server 2016 Instance to another server with an Instance SQL Server 2019,
Do I need to install SSIS for each instance on each server?
Do I need to create the catalog in both source and destination instances? or is it enough to create it only on the source instance? or it has to be on the destination?
Thanks!
SSIS is an ETL tool, so you'll install SSIS on the server that you want to designate as your ETL server. There are several common scenarios for where that server is located.
A stand-alone ETL server, which is neither the source nor the target of the current ETL job you're working on.
A separate ETL server that is an instance on the same server as either your current source or your current destination.
Your current source server
Your current destination server
You'll have to decide which scenario makes the most sense in your environment. At my current gig, we have a few dedicated ETL boxes, and we cluster the jobs on them by subject matter.
Previously, I worked in an environment where 10% of the ETL was importing data and 90% was manipulating it on one server. In that situation, it made sense to have the SSIS stuff all on the destination server so that we weren't moving data over the wire to transform it.
As your situation evolves, and you use more and more SSIS, you might change your mind. That's fine. It's not a trivial thing to change, but it's also not set in stone once you plunk SSIS down someplace.
How SQL Server maintenance plans work in relation with Jobs and DTExec?
I am specially interested in the maintenance plan's local connection which seems to be passed somehow to dtexec.
I have read some docs but none of them is clear enough. As I have understood it: the MP creates a SSIS package as well as a job... and the job in turn invokes the external application dtexec... which executes the SSIS package. This package has to access the SQL Server and to get that it uses the account provided by the "local connection" parameter of the maintenance plan.
Is my understanding right?
Some reference to MS official detailed documentation would be welcome.
Maintenance plans are saved in form of SSIS packages in the MSDB database. You can access them by connecting to the Integration Services from the SQL Server Management Studio. You can refer to the following link for more information:
Move SQL Server Maintenance Plan from One Server to Another
In addition you can retrieve information about these packages by querying the tables and views located in msdb database:
Querying SQL Server Agent Job Information
The system msdb database, introduction and tips
msdb Database official documentation
Ever SQL job execte the SSIS package by passing connections and other values as parameters.
in my development environment we support the application both on MSSQL Server as well as Oracle. The database schema of both of these RDBMS are same.
while development we found that the developer made a mistake and forgot to change the oracle database for the last 1 yr. therfore the oracle script is quite behind in term of schema from SQL Server schema script.
now the question is how i can compare the two RDBMS systems to find the difference and make the oracle script updated
If there are no track log from which it's possible to find and reproduce all changes applied to SQL Server since first detected inconsistency with Oracle version, or that changes was applied, but only partially, you really need to compare objects presented in both databases.
In this case setup a link between databases on any side and use system dictionary views to compare table structures and other objects to find differences and, possible, to generate script for Oracle scheme rollup.
If you want to act from MS SQL Server side:
Install and configure Oracle Instant Client
Install Oracle ODAC
Follow Microsoft recomendations (64-bit version)
Connect as any user with dba role (or use same Oracle schema where object resides) to Oracle from MS SQL database
If you want to act from Oracle Server side:
Install and configure Oracle Database Gateway for SQL Server.
Create database link to MS SQL Server.
After successful configuration you may join Information schema views on SQL Server side with Data dictionary views on Oracle side to find differences.
Of course there are many troubles at this way like different data types, but it gives a chance to automate at least part of work.
I'm managing a web app featuring countless ETL (Extract, Transform & Load) processes feeding a datawarehouse (using SSIS dtsx packages + *.sql files).
At the moment, everything is governed by several SQL Agent processes monitoring "scheduling queue" tables.
I'm trying to figure out a way to develop a scheduler or scheduling "framework" of some sort that needs to do a subset of what SQL Server Agent does (executing dtsx packages, executing SQL) but with a configuration UI based on ASP.NET, as users need to be able to create & modify schedules, manually launch processes and monitor custom logs.
Configuration info, activity, execution logs, schedules, jobs, job steps and everything else needed that I'm missing should be persisted to DB.
I'm guessing a Windows Service would be flexible but development costs might skyrocket.
I appreciate any sort of input, particularly some clues on SQL Server Agent's internal workflow. I DID try searching for info, but it being PROPRIETARY software, I found zilch.
(maybe I should specify I'm more than familiar with SQL Server and C# based .NET development, I just have no idea where to start from..)
The SQL server agent database is called 'msdb'. You can attach profiler to msdb and create some scheduled jobs to see how it works.
The gist of the answer is one should use SQL Server Service Broker, coupled with SQL Server Service Broker External Activator from the Feature Pack
What are the most reasonable ways to move table data from SQL Server to Oracle (on *nix) on a regular basis?
Using SQL Server Integration Services (SSIS) is likely your best bet. If you're not familiar with SSIS, the best way to try something out is to use the SQL Server Export Wizard and have it create an SSIS package for you. For example, if you go into SQL Server Management Studio and right-click on your database, then select Tasks->Export Data. From there click next until you get to the "Choose a Destination" step. Select the "Microsoft OLE DB Provider for Oracle" and click Properties to define your database connection. When you click through the wizard, on the Save and Execute page, make sure you check the checkbox labelled "Save SSIS Package", on the next screen specify where to save the SSIS package. Once you finish the Export Wizard, your data will have been exported and you will have an SSIS package that you can use as is, or go in and tweak it to do more specific things.
Once you have your SSIS package, you can schedule it by creating a SQL Server Agent Job.
Oracle Heterogeneous Connectivity / Database Gateways in conjunction with materialized view(s), PL/SQL, or Java
SSIS or DTS: both can be scheduled but require more than read-only access to SQL Server
Java (probably within Oracle but optionally at the OS) using ODBC or SQLJ to access SQL Server and, possibly, Oracle
SQL Server scheduled to export to CSV, Oracle scheduled to import from CSV
Any of the other ETL tools (e.g. Informatica, Cognos)
Any of the myriad languages that can access both databases (but would require maintaining a third environment to run the application within)
Scheduling:
Automatic / not reuqired with materialized views
Oracle DBMS_JOB / DBMS_SCHEDULER
OS-specific (cron, Windows Scheduled Tasks, etc)
In the case of SSIS, DTS, or CSV export, scheduled within SQL Server
Create a database link from Oracle to Sql Server (heterogeneous connectivity). You can use this link to retrieve the data from Sql Server with a simple select statement. If you want to schedule you can use a materialized view or dbms_scheduler.
An alternative is to put your data in a csv file, you can use an external table or sqlloader to load this data in the Oracle database.
Here is what I do:
Connect to SQL Server by Oracle SQL developer using this link:
https://kentgraziano.com/2013/01/14/tech-tip-connect-to-sql-server-using-oracle-sql-developer/
After you have added SQL jar, you will see SQL Server tab in the Connection window:
Then connect to the SQL Instance.
Then open SQL instance and choose database or table that you want to copy. Right click on any database/table, then click on "copy to oracle" there and choose the right user[database] at "Destination Connection Name " where you want to copy your tables.
You can also change some properties there.
Click "OK" and that's it.
Let me know in case of any issues.
You can have SQL Server interface with Oracle directly through SSIS (or DTS for 2k). It will provide ETL functionality and can be scheduled on a regular basis.
I had success creating a linked server (from within the Enterprise Manager, I think) to Oracle on the SQL Server side. Then I could use normal stored procedures on both sides to accomplish smaller data movement and updates in both directions. This approach can bypass the need to try and put something together outside the databases.
Try hard to use the latest possible Oracle client on the SQL Server side though. I recall some defects in the 10.2.0.2 client and getting the 10.2.0.4 client required your "official" Oracle registration or purchase number or something.
For big data moves (or maybe even moves/updates you want done on a daily or less frequent basis), definitely use one of the ETL tools. We had Informatica for our ETL processes, but if SSIS can pull off what you need, that's fine too.