Company is setting up new instances of SQL Server and because I have many SSIS jobs, the idea was to setup a dedicated SSIS server. It seems to be a relatively common practice and makes a lot of sense. You're removing the resources needed for SSIS off the main database server versus how we've done it historically which is to store the packages in a directory on the main database server and having SSIS compete with the same resources as the actual database. Here's where I am confused.
I was told by our DBA to place a copy of the SSIS package on a file share directory on the SSIS server. Then using the SQL Agent on the database server, create a job to run the package like normal where Package Source = "File System" and Package location being the file share directory and file name on the SSIS server. Problem is I don't see how that kind of setup accomplishes the goal of removing the execution of the package off the main database server. Won't the main database server still be executing the package thus impacting performance and all we're accomplishing is using the SSIS server as an external drive or place to store the actual .dtsx files?
Related
I have a question that might be very basic.
If I have a SSIS package where I have the Transfer DB task to transfer a database from a server with a SQL Server 2016 Instance to another server with an Instance SQL Server 2019,
Do I need to install SSIS for each instance on each server?
Do I need to create the catalog in both source and destination instances? or is it enough to create it only on the source instance? or it has to be on the destination?
Thanks!
SSIS is an ETL tool, so you'll install SSIS on the server that you want to designate as your ETL server. There are several common scenarios for where that server is located.
A stand-alone ETL server, which is neither the source nor the target of the current ETL job you're working on.
A separate ETL server that is an instance on the same server as either your current source or your current destination.
Your current source server
Your current destination server
You'll have to decide which scenario makes the most sense in your environment. At my current gig, we have a few dedicated ETL boxes, and we cluster the jobs on them by subject matter.
Previously, I worked in an environment where 10% of the ETL was importing data and 90% was manipulating it on one server. In that situation, it made sense to have the SSIS stuff all on the destination server so that we weren't moving data over the wire to transform it.
As your situation evolves, and you use more and more SSIS, you might change your mind. That's fine. It's not a trivial thing to change, but it's also not set in stone once you plunk SSIS down someplace.
Having now added a like for like DB to do UAT on to our server I have run into a problem with our SSIS Packages. They are stored as files and as such we use File System in SQL Server Agent to run them on a schedule. I now need to run these packages on both databases (different data sources).
I have checked over documentation and forums and when paramterizing over environments every time it relates to the deployment of packages to the catalog. Which I think is because with files they are saved with package data sources not project data sources.
This left me wondering...
Is it possible to switch data sources when calling a package using File System in SQL Server Agent?
Unfortunately the answer on this is no.
Moving to the SSIS catalog allows you to switch between environments which you can set in SSMS.
This is a great link: https://blog.exsilio.com/all/configuring-environment-variables-ssis-package/
I did get an error when running my agent job after this but this was a need to set the runtime to 32bit.
In SSIS - Right click your solution and head to debugging to find the option.
In SQL Server Agent - Head to Config and advanced where you would set the environment variable in the SSIS step you have created.
Microsoft SQL Server 2016. I've done an export of a table (using the wizard) I saved the package and I see it in the msdb.sysssispackages table. But it doesn't show up in the Integration Services Catalogue section. So how can I make it show up so I can execute this package again?
If you're executing it through SQL Agent, the Package Source is SQL Server for packages stored in the msdb
A confusing bit of terminology that I started to respond with on your deleted question is that you can store a package in SQL Server and that usually means the msdb.dbo.sysssispackages table. But we also have the project deployment model, which deploys an .ispac file into the SSISDB.
If you want a package to be in the SSISDB that is not of the project deployment model, the 2016(?) release of SSIS allowed for an Incremental package deployment for a project deployment model. You could use that approach to uplift your msdb based SSIS package into a new "project" without ever opening Visual Studio to create an actual solution.
That's a technical answer, it can be done that way but I would advise against it as you generally want to source control any processes that you operationalize.
The situation is that I have two DB servers with SQL Server, I need to transfer the data from Server A to Server B automatically every night.
So according to sql server data transfer from one server to another server, I created an SSIS project. And it manage to transfer data with good performance. But the question is, how do I export the project into SSIS package and run it every night?
I assume you are still running it from Visual Studio. To schedule it daily you will need to do the below,
Deploy your project to a SQL Server
Create an Agent job and schedule it.
Found a good article, sharing that with you here.
https://www.mssqltips.com/sqlservertutorial/9069/deploy-and-schedule-an-sql-server-integration-services-ssis-package-step-by-step/
You can either store the package as in the file system or in the SSIS catalog. I would say that storing it in the SSIS catalog is the best way to go here.
Before you deploy your package to the SSIS catalog you need to make sure that integration services is installed on your server as well as that the SSIS catalog has been created.
How to create ssis catalog
If all above is OK then proceed and deploy your package and create an agent job. Please notice that you need to set up permissions for the sql agent account so it can access your databases.
I'm not new to SQL Server but am new to Integration Services, so I want to understand if the following requirement is fully achievable in SSIS, or if I will need to consider some level of C# development to supplement:
We have 25 Azure VM's running Windows Server 2016 Datacenter - on each VM we have thousands of log files in different folders. We need to have all these folders monitored and upon creation of any new file with a certain string present in the name, we want the contents of the file exported to a table in our Azure SQL Server 2017 database.
Is this kind of custom logic configurable in an SSIS project, or is SSIS more geared toward static definitions of file folders/filenames?
I don't think that you can implement a folder watcher within SSIS package. But you can achieve this in different ways:
(1) - using a folder watcher
You have to develop the SSIS packages you need to import data, then you must develop a folder watcher using C# to read the changes in folder and if the file name meet the requirementes you should run dtexec utility (using shell) to run the relevant SSIS package.
(2) - using SQL agent job
You can configure an SQL server agent job to check the file changes periodically and run the SSIS package when required.