I am attempting to set up an SSIS Project Deployment Model with Environment specific variables. I am in part using this tutorial as a guideline, SSIS Parameters and Environments, as well as Deploy Integration Services (SSIS) Projects and Packages.
Do Environments not work like I would plausibly expect them to? I need different connection strings for different environments. So I created those project level parameters in the SSIS solution and set the connection managers to use these Expressions. I deployed the project to the database and created the Environments I wanted (Development, Production)
Now I would expect when I configure the SQL job and set the environment as I need to, the correct variables would be pulled for the connection strings I need. As you can see in the screen shot SSMS is complaining that the development parameters don't exist in the Production environment (duh!).
So I have to kind of wonder, have I gone down the wrong rabbit hole?
Production - Use these connection strings
Development - Use these other connection strings
You should have only one set of parameters in the SSIS project.
So instead of two parameters ODSDevelopment and ODSProduction you will have one parameter ODSConnectionString.
Then in the environments you will set up ODSConnectionString to point to Production or Development, depending on the environment.
In other words, word 'Production' may be used only in the environment name, but never in the parameter name.
Related
Ok, so I understand what they are, they're collections of parameter assignments that you can tell a package to use upon execution.
What I'm trying to understand is why or if I need to use them.
I'm migrated a bunch of old SSIS packages using the packages deployment model to some new servers using the project deployment model. I have one project containing about 35 packages. I've created parameters on all my packages, and have a couple of project level parameters for stuff like Server Name etc.
I'm developing on my PC and the packages will have their parameters set to my dev environment settings by default unless I change them.
I'm deploying my packages to 3 servers (Test, UAT, Prod). I deploy them from Visual Studio.
Each server runs an identically scripted SQL job to execute the package.
So now, I need to set my parameters for each environment/server.
Do I need to set up environments, or why can't I just right click - configure my Project in the SSIS Integration Services Catalog on each server, and set the parameters there for the project and each package?
If I create environments, I still need to enter all the parameter values for each server/environment, but then I need to set up the reference between the project and the environment, and set each SQL job to use the relevant environment when executing the job.
Are environments only useful if you have one server, one package catalogue, and one set of SQL Jobs, and you're just using different databases for each environment so you need the environments to toggle between each?
Aren't they overkill if you have your environments on different servers, or am I missing something?
A use case for Environments
As the DBA, I have access to the user names and passwords for systems. I can perform a one-time task of setting up environments that define all the things.
I can then empower the developers to create/deploy and configure their own SSIS projects without having to get involved or giving them a post-it with the associated credentials.
Environment vs Project configuration - I view this as automating what makes the most sense. I used to be a big advocate of Environments but I found migrating them difficult between servers (before I found the magic script) and the fact that you can only have 1 associated to a project a limitation I didn't like. At this point, I generally configure a project directly with SSMS and save the script off so I can replace values as I migrate up the environments.
In order to enable operational management of data integration processes developed in SSIS, I am seeking to be able to externally configure:
server (data source)
database (catalog)
schema
From what I have seen, all of these are typically hardcoded into SSIS packages through the Connection Manager and in SQL statements. This hardcoding limits the DBA from being able to allocate resources differently and, if there is ever a change, requires every package to be modified if Package Deployment is being used.
It appears that the Project Deployment would reduce this somewhat, but no eliminate it.
Target environment is SQL Server 2016 and VS 2017.
How can the server, database, and schema be externalized from the package?
SSIS has a robust facility for configuring packages per environment. You can configure any property in the package externally. This can be done in SQL Agent and even from the command line at runtime. Configurations can be stored in config files, system environment variables, a SQL table, etc. However, the modern way of configuring packages is through the project deployment model.
Here is the gist of how it works:
Add a parameter at the package or project level
reference that parameter in an expression which configures the property you want to set, i.e. the server name or initial catalog
Deploy the project to an instance of SSIS
In SSIS, add an environment and configure the variable. This can even be passwords which are securely stored
Add a reference to that environment from the project, and finally reference which environment you want to use at runtime.
The first link below shows a dialogue that was created for configuring connection managers with parameters. Please note that the package will store the default values, but when you create an environment as noted above, this allows you to easily set it at runtime.
As for configuring a schema, this is possible as well, by using parameters, but you would need to use expressions for your SQL queries and setting the destination. I would avoid making schemas variable across environments. This will present a lot of effort and complexity for very little flexibility that is offered in return. Please read up on these links and good luck!
How to parameterize connection managers
All about parameters in SSIS
It is not clear to me how I should use the new features of SSIS in SQL Server 2012/2014 in an enterprise environment. Specifically, I am referring to the project deployment model, project parameters, environments, etc. We use a three-tier environment workflow; developing in development, testing and staging in QA, and production in production. The developers only have access to the development environment. The DBA’s migrate code to the other environments. All source is kept in TFS.
What is the intended workflow using these new features? If a developer develops the project/package, does the developer deploy the project to the SSISDB or does the developer stop after checking in the source? Where does the DBA come into the picture? Which environment contains SSISDB? How does the project/package get deployed to the other environments?
There seems to be many “how-to’s” published on the Internet, but I am struggling to find one that deals with the business workflow best practices. Can anyone suggest a link to an article on this subject?
Thanks.
What is the intended workflow using these new features?
It is up to the enterprise to determine how they will use them.
If a developer develops the project/package, does the developer deploy the project to the SSISDB or does the developer stop after checking in the source?
Where does the DBA come into the picture? Which environment contains SSISDB? How does the project/package get deployed to the other environments?
It really does depend. I advocate that developers have sysadmin rights in the development tier of servers. If they break it, they fix it (or if they've really pooched it, we re-image the server). In that scenario, they develop the implementation process and use deployments to Development to simulate the actions the DBAs will take when deploying to all the other pre-production and production environments. This generally satisfies your favorite regulatory standard (SOX/SAS70/HIPPA/CPI/etc) as those creating the work are not the same ones that install it.
What is the deliverable unit of work for SSIS packages using the project deployment model? It is an .ispac file. That is a self contained zip file with a manifest, project level parameters, project level connection managers and the SSIS packages.
How you generate that is up to you. Maybe you check the ispac in and that is what is deployed to your environments. Maybe the DBAs open the solution from source control and build their own ispac. Maybe you have Continuous Integration, CI, running and you click a button and some automated process generates and deploys the ispac.
That's 1/3 of the equation. From the SSISDB side, you likely want to create an Environment and populate it with variable values. Things like Connection Strings and file paths and user names & passwords. When you start creating those things, CLICK THE CREATE SCRIPT TO NEW WINDOW button! Otherwise, you're going to have to re-enter all that data when you lift to a new environment. I would expect your developers to check those scripts into source control. For passwords, blank out the value and make notes in your deployment checklist that they need to fix that before mashing F5.
You also need SQL Scripts to create the structure (folder) within the SSISDB for the project to be deployed into. Once deployed, you'll want to apply the Environment values, created in the preceding step, to the newly deployed project. Save those out as well.
I would have each environment contain an SSISDB. I don't want a missed configuration allowing a process in the production tier to reach across to the development tier and pull data. I've seen that, it's not pretty. When code is deployed to the QA/Stage tier, we find out quickly whether we missed a connection string somewhere because the dev servers reject the connection from QA. This means our SQL Instances don't all run under the same server account. Each tier gets their own account: domain\SQLServer_DEV, domain\SQLServer_QA, domain\SQLServer_PROD Do what you can to prevent yourself from having a bad day. If you go with a single/shared SSISDB across all your tiers, it can work, but you're going to have to invest a lot more energy ensuring that packages always run with the correct configuration environment applied lest bad things happen.
I have a server hosting a number of different databases, all with the same structure, and a library of SSIS packages, some of which are common across the client databases and others are client-specific.
I'm aware that you can store packages in MSDB, but this is a shared store across the whole SQL instance - is it possible to attach a package to a specific database?
Actually, when you store packages into the msdb, they are stored in specific instance's msdb. Either run SELECT * FROM dbo.sysdtspackages90 (2005) or SELECT * FROM dbo.sysssispackages (2008) across all your instances and you'll determine which one is currently hosting your packages. If you are using folders, I have fancier version of these queries available.
What I believe you are observing is an artifact of the tools. There is only one instance of the SQL Server Integration Services Service. This service doesn't stop you from storing packages in specific instance, it just makes it a little more complex to do so. Or as I see it, by ditching the GUI (SSMS) you free yourself from the fetters of non-automated administration.
How do you push packages into the other named instances? You can either edit the service's .ini file as described in the above link and then reconnect to the Integration Services thing in SSMS or use a command line or query approach to managing your packages. We used the SSISDeployManifest in my previous shops with success to handle deployments. There is a GUI associated to the .ssisDeploymentManifest and you can use that to handle your deploys or you're welcome to work with the .NET object model to handle deployments. I was happy with my PowerShell SSIS deployment and maintenance but your mileage my vary.
Finally to give concrete examples for a command line deployment, the following would deploy a package named MyPackage.dtsx sitting in the root of C to named instances on the current machine and deploy them into the root of MSDB.
dtutil.exe /file "C:\MyPackage.dtsx" /DestServer .\Dev2008 /COPY SQL;"MyPackage"
dtutil.exe /file "C:\MyPackage.dtsx" /DestServer .\Test2008 /COPY SQL;"MyPackage"
I have an earlier version of my PowerShell script for generating calls to dtexec instead of using the object library directly.
Let me know if you have further questions
We have three environments - production, staging, and test. Each environment has multiple databases for our internal back office systems (ie, time collection, accounting, human resources, etc).
For our employee benefits, we create file feeds that get shipped out to our benefit providers to handle enrollment/termination. Prior to SSIS 2012, we had a set of custom SSIS components that centralized configurations - so depending on where we deployed the package, it would automatically pick up the right connection strings.
Now our old way worked, but it has its own difficulties (ie, new developers often forget to set up the package properly and spend time trying to figure it out), so we'd like to move forward with SSIS 2012 and the Project Deployment Model.
I really like the concept of the Environment configurations, but simply creating 3 environments for prod/stage/test doesn't seem like a good idea. A lot of our packages have custom SFTP credentials, file paths to various areas on the network, etc - so those environments would grow very quickly with things that don't really matter to most of our packages.
Is there a way to have one Environment configuration pull in data from another Environment configuration? Something like this would allow me to create a Production Environment config which has the common database servers, and then inherit that within a production config for one of our benefit file feed packages. The benefit file feed package prod config would have all of its other very specific variables.
Probably catalog stored procedures in SSISDB could be an option (e.g. catalog.create_environment, catalog.create_environment_reference, catalog.create_environment_variable)