I created an SSIS project with multiple packages. One of my package has a data flow task using Excel as data source. After deploying the project to SQL SSISDB, I can see the .dtsx files in SSMS. But where should the excel file reside? Should it still be in my local machine. The SQL resides on another server and not my local machine. Where is the deployed packages stored?
Can someone explain these?
Excel file
If the excel connection manager are pointing to a fixed path on your local machine. It will throw an error when executed from different server since the excel file path is not found.
You have to use dynamic excel connections to fix this issue, or to use an excel file on a shared network location (UNC path).
Package deployment
After deployment, the packages are stored within SSISDB database:
Deploying Packages to SQL Server Integration Services Catalog (SSISDB)
Ways to use and execute SQL Server Integration Services packages
SSISDB for SQL Server Integration Services
Related
I am attempting to deploy an SSIS package on a remote SQL Server by following the following tutorial. My issue is that the Integration Services Catalogs folder is empty and when I try to create a catalogue, I get the following error:
The path to the catalog backup file could not be determined.
Integration Services might not be installed on this server, or the user may not have the appropriate access permissions
Is it even possible to achieve my goal on a remote server or do I absolutely need access to the machine on which it is running on?
Once SSIS is installed on the server, you can create the SSIS catalog and deploy SSIS projects to the catalog remotely.
Having now added a like for like DB to do UAT on to our server I have run into a problem with our SSIS Packages. They are stored as files and as such we use File System in SQL Server Agent to run them on a schedule. I now need to run these packages on both databases (different data sources).
I have checked over documentation and forums and when paramterizing over environments every time it relates to the deployment of packages to the catalog. Which I think is because with files they are saved with package data sources not project data sources.
This left me wondering...
Is it possible to switch data sources when calling a package using File System in SQL Server Agent?
Unfortunately the answer on this is no.
Moving to the SSIS catalog allows you to switch between environments which you can set in SSMS.
This is a great link: https://blog.exsilio.com/all/configuring-environment-variables-ssis-package/
I did get an error when running my agent job after this but this was a need to set the runtime to 32bit.
In SSIS - Right click your solution and head to debugging to find the option.
In SQL Server Agent - Head to Config and advanced where you would set the environment variable in the SSIS step you have created.
Company is setting up new instances of SQL Server and because I have many SSIS jobs, the idea was to setup a dedicated SSIS server. It seems to be a relatively common practice and makes a lot of sense. You're removing the resources needed for SSIS off the main database server versus how we've done it historically which is to store the packages in a directory on the main database server and having SSIS compete with the same resources as the actual database. Here's where I am confused.
I was told by our DBA to place a copy of the SSIS package on a file share directory on the SSIS server. Then using the SQL Agent on the database server, create a job to run the package like normal where Package Source = "File System" and Package location being the file share directory and file name on the SSIS server. Problem is I don't see how that kind of setup accomplishes the goal of removing the execution of the package off the main database server. Won't the main database server still be executing the package thus impacting performance and all we're accomplishing is using the SSIS server as an external drive or place to store the actual .dtsx files?
I'm not new to SQL Server but am new to Integration Services, so I want to understand if the following requirement is fully achievable in SSIS, or if I will need to consider some level of C# development to supplement:
We have 25 Azure VM's running Windows Server 2016 Datacenter - on each VM we have thousands of log files in different folders. We need to have all these folders monitored and upon creation of any new file with a certain string present in the name, we want the contents of the file exported to a table in our Azure SQL Server 2017 database.
Is this kind of custom logic configurable in an SSIS project, or is SSIS more geared toward static definitions of file folders/filenames?
I don't think that you can implement a folder watcher within SSIS package. But you can achieve this in different ways:
(1) - using a folder watcher
You have to develop the SSIS packages you need to import data, then you must develop a folder watcher using C# to read the changes in folder and if the file name meet the requirementes you should run dtexec utility (using shell) to run the relevant SSIS package.
(2) - using SQL agent job
You can configure an SQL server agent job to check the file changes periodically and run the SSIS package when required.
Just started to work on a SSIS ETL package at work and I am sort of stuck thinking how to solve this problem and how others solved it. Since I am fair new to SSIS, I must ask the experts here. So I am creating a SSIS ETL package which I create source and destination connection (mostly relational databases) in a development environment, which means the source and destination connections are to development databases. Everything works out fine. The problem is when I complete the SSIS package, build it, create the manifest and deployed, then I realize the package will fail, because I have deployed the package in the production environment, which means the source and destination connection are to production databases, in production environment.
So my questions is how do I create a ssis ETL package in the development phase or development environment using development source and destination connections be able to deploy a package into a different environment where the source and destination connection are different?
I am using sql server 2012 with BIDS 2012, Data Tools 2012.
You can make the package connection configurable.
select "SSIS">> Pacakage Configuration >> navigate to connection >> select servername and database name
Click next
Save pacakge configuration to dtsconfig file.
During deployment modify this dtsconfig file and specify.
once you have deployed your package navigate to Integration Services Catalog -> SSISDB -> your SSIS Project.
Once there open up the desired package and under [Connections] tab you can change the connection.
Alternative to this (method 2) is to create Environments; where you can quickly change between dev, qa, prod by checking towards the bottom which environment you wish SSIS to pick up config from.