I have a question that might be very basic.
If I have a SSIS package where I have the Transfer DB task to transfer a database from a server with a SQL Server 2016 Instance to another server with an Instance SQL Server 2019,
Do I need to install SSIS for each instance on each server?
Do I need to create the catalog in both source and destination instances? or is it enough to create it only on the source instance? or it has to be on the destination?
Thanks!
SSIS is an ETL tool, so you'll install SSIS on the server that you want to designate as your ETL server. There are several common scenarios for where that server is located.
A stand-alone ETL server, which is neither the source nor the target of the current ETL job you're working on.
A separate ETL server that is an instance on the same server as either your current source or your current destination.
Your current source server
Your current destination server
You'll have to decide which scenario makes the most sense in your environment. At my current gig, we have a few dedicated ETL boxes, and we cluster the jobs on them by subject matter.
Previously, I worked in an environment where 10% of the ETL was importing data and 90% was manipulating it on one server. In that situation, it made sense to have the SSIS stuff all on the destination server so that we weren't moving data over the wire to transform it.
As your situation evolves, and you use more and more SSIS, you might change your mind. That's fine. It's not a trivial thing to change, but it's also not set in stone once you plunk SSIS down someplace.
Related
Company is setting up new instances of SQL Server and because I have many SSIS jobs, the idea was to setup a dedicated SSIS server. It seems to be a relatively common practice and makes a lot of sense. You're removing the resources needed for SSIS off the main database server versus how we've done it historically which is to store the packages in a directory on the main database server and having SSIS compete with the same resources as the actual database. Here's where I am confused.
I was told by our DBA to place a copy of the SSIS package on a file share directory on the SSIS server. Then using the SQL Agent on the database server, create a job to run the package like normal where Package Source = "File System" and Package location being the file share directory and file name on the SSIS server. Problem is I don't see how that kind of setup accomplishes the goal of removing the execution of the package off the main database server. Won't the main database server still be executing the package thus impacting performance and all we're accomplishing is using the SSIS server as an external drive or place to store the actual .dtsx files?
The situation is that I have two DB servers with SQL Server, I need to transfer the data from Server A to Server B automatically every night.
So according to sql server data transfer from one server to another server, I created an SSIS project. And it manage to transfer data with good performance. But the question is, how do I export the project into SSIS package and run it every night?
I assume you are still running it from Visual Studio. To schedule it daily you will need to do the below,
Deploy your project to a SQL Server
Create an Agent job and schedule it.
Found a good article, sharing that with you here.
https://www.mssqltips.com/sqlservertutorial/9069/deploy-and-schedule-an-sql-server-integration-services-ssis-package-step-by-step/
You can either store the package as in the file system or in the SSIS catalog. I would say that storing it in the SSIS catalog is the best way to go here.
Before you deploy your package to the SSIS catalog you need to make sure that integration services is installed on your server as well as that the SSIS catalog has been created.
How to create ssis catalog
If all above is OK then proceed and deploy your package and create an agent job. Please notice that you need to set up permissions for the sql agent account so it can access your databases.
I want to create a job with a specific query to export some date from SQL Server to Oracle. How can I schedule this?
Thank you!
How you move forward with implementing this will depend on a variety of factors, but the following steps outline this process at a high level. Of course, you'll want to choose the configurations that best suite your specific environment and business needs.
Create an SSIS package with an OLE DB Connection Manager to SQL
Server
Add a Connection Manager for Oracle. Some options are the Microsoft
ODBC driver for Oracle or OLE DB Provider for Oracle. You can here
more about setting up connections here.
After the connections are configured, you'll want to create
components to export the data, likely using a Data Flow Task (or
multiple). How this is setup will vary greatly depending on the business needs. You can use the query mentioned in your question for the SQL command in the OLE DB source to SQL Server. I'd also recommend using stored procedures as opposed to static queries.
After the package is complete, you can add it as a job in SQL Agent
and schedule it to run at the desired frequency. Make sure that the
account the job runs under has the necessary permissions,
including on both the SQL Server and Oracle objects involved.
This is not required, but installing the SSIS catalog, deploying the projects there, and
running them from this instead of the file system will offer many
benefits. This link contains
more information on SSISDB (the catalog).
What are the most reasonable ways to move table data from SQL Server to Oracle (on *nix) on a regular basis?
Using SQL Server Integration Services (SSIS) is likely your best bet. If you're not familiar with SSIS, the best way to try something out is to use the SQL Server Export Wizard and have it create an SSIS package for you. For example, if you go into SQL Server Management Studio and right-click on your database, then select Tasks->Export Data. From there click next until you get to the "Choose a Destination" step. Select the "Microsoft OLE DB Provider for Oracle" and click Properties to define your database connection. When you click through the wizard, on the Save and Execute page, make sure you check the checkbox labelled "Save SSIS Package", on the next screen specify where to save the SSIS package. Once you finish the Export Wizard, your data will have been exported and you will have an SSIS package that you can use as is, or go in and tweak it to do more specific things.
Once you have your SSIS package, you can schedule it by creating a SQL Server Agent Job.
Oracle Heterogeneous Connectivity / Database Gateways in conjunction with materialized view(s), PL/SQL, or Java
SSIS or DTS: both can be scheduled but require more than read-only access to SQL Server
Java (probably within Oracle but optionally at the OS) using ODBC or SQLJ to access SQL Server and, possibly, Oracle
SQL Server scheduled to export to CSV, Oracle scheduled to import from CSV
Any of the other ETL tools (e.g. Informatica, Cognos)
Any of the myriad languages that can access both databases (but would require maintaining a third environment to run the application within)
Scheduling:
Automatic / not reuqired with materialized views
Oracle DBMS_JOB / DBMS_SCHEDULER
OS-specific (cron, Windows Scheduled Tasks, etc)
In the case of SSIS, DTS, or CSV export, scheduled within SQL Server
Create a database link from Oracle to Sql Server (heterogeneous connectivity). You can use this link to retrieve the data from Sql Server with a simple select statement. If you want to schedule you can use a materialized view or dbms_scheduler.
An alternative is to put your data in a csv file, you can use an external table or sqlloader to load this data in the Oracle database.
Here is what I do:
Connect to SQL Server by Oracle SQL developer using this link:
https://kentgraziano.com/2013/01/14/tech-tip-connect-to-sql-server-using-oracle-sql-developer/
After you have added SQL jar, you will see SQL Server tab in the Connection window:
Then connect to the SQL Instance.
Then open SQL instance and choose database or table that you want to copy. Right click on any database/table, then click on "copy to oracle" there and choose the right user[database] at "Destination Connection Name " where you want to copy your tables.
You can also change some properties there.
Click "OK" and that's it.
Let me know in case of any issues.
You can have SQL Server interface with Oracle directly through SSIS (or DTS for 2k). It will provide ETL functionality and can be scheduled on a regular basis.
I had success creating a linked server (from within the Enterprise Manager, I think) to Oracle on the SQL Server side. Then I could use normal stored procedures on both sides to accomplish smaller data movement and updates in both directions. This approach can bypass the need to try and put something together outside the databases.
Try hard to use the latest possible Oracle client on the SQL Server side though. I recall some defects in the 10.2.0.2 client and getting the 10.2.0.4 client required your "official" Oracle registration or purchase number or something.
For big data moves (or maybe even moves/updates you want done on a daily or less frequent basis), definitely use one of the ETL tools. We had Informatica for our ETL processes, but if SSIS can pull off what you need, that's fine too.
I need to copy an entire database from a SQL Server 2005 on my server over to my local SQL Express in order to run my application for a presentation. What is the fastest/easiest way to get this done?
EDIT: I have very limited access to my server so I don't think I can access the backup file that I could create so that is out.
If the database is not too big, you could use the Database Publishing Wizard.
This is a free tool from Microsoft which creates a complete SQL script of a database for you (not only the tables and stuff, but all data as well).
You can install the tool on your machine, connect to a remote server and let the tool create the script directly on your machine.
You can download the Database Publishing Wizard here.
Apparently the link above doesn't work anymore in 2019.
That's probably because in newer versions of SQL Server Management Studio, the functionality of the Database Publishing Wizard is included out-of-the-box, so there's no need to install it separately.
It's now called the Generate and Publish Scripts Wizard, but it does exactly the same.
You can right click the database -> Tasks -> Generate scripts. Here you can select one, multiple, or all objects. Then in the 'Set Scripting Options' step of the wizard, click Advanced. In here set the property 'Types of Data to script; to Schema and Data.
Having done these steps, make sure you publish to a file. Because only file can handle large amounts of data.
Now you should have all your objects, tables, and data scripted. Now start running the scripts and viola!
Back up the database on the server and then restore it locally in SQL Express.
EDIT: If this is part of your work, surely you can get someone in networks to get you a backup..?
If you can login to both servers (the Express and the 05 Server) using SQL Server Management Studio then you can do a DB Restore from one database to the other. No need for backup files at all.
You can use SSIS's database copy wizard, but it's not quick at all. Is there a DBA there that you can ask for the backup file? That will probably turn out to be the fastest way.
Depending which versions of SQL Server you are using, you might get some mileage out of the SQL Server Database Publishing Wizard from Microsoft. I've had mixed results with this tool in complex environments, but for most simple database scenarios it is a great tool.
An example of a scenario where I ran into difficulties was a DB with multiple distinct schemas, each with their own owner and extreme separation between the tables (don't ask...). As I said, though, most other scenarios have been fine.
Save your database as a sql script
EMS SQL Manager (for example) allows you to backup your database as a script in a .sql file. It is then possible to run this file against any other SQL server. Just update the first line of the script (CREATE DATABASE ....).
You can even fully parameter the script in order to include data from complete or filtered tables.