I have a SSIS package that was originally developed to target SQL Server 2015 that does a variety of things in C# Scripts using SQL Server Management Objects (SMO) that now needs to be upgraded to target SQL Server 2019. Eventually, this package is deployed to the server to run in a SQL Agent Job.
Starting in SQL Server 2017, SMO was moved to a NuGet package and is no longer included in SQL Server Feature Packs as it was for previous versions.
The issue I have is that Integration Services Projects do not really support NuGet. As discussed in the below link:
SSIS Script Task cant find reference to assembly
As mentioned in the above ticket, I believe I could manually install the .dll's to the server if necessary but I am curious if there are other solutions to this issue.
Questions:
Is there a way to get Script Tasks in SSIS packages to utilize the SMO NuGet?
If "no" to the above, is there another common method of executing C# script via SQL server other than installing the .dll's to the server?
You would be better off installing SMO DLLs into target server Global Assembly Cache. SSIS still does not use local assemblies and you have to introduce some workarounds to defeat that.
You can do a simple MSI installation with SMO DLLs - for example see WIX deploy two assemblies to GAC or What's the 'correct' way of registering/installing an Assembly to the GAC?.
Related
I am trying to run a flatfile SSIS package via .Net code using the Microsoft.SqlServer.Dts.Runtime DLL version 14.0.0.0.
Upon executing I get "To run a SSIS package outside of SQL Server Data Tools you must install Standard edition of Integration Services or higher"
I have SSIS for Visual studio 2017 installed, along with with SQL server 2017 both have the SSIS extension installed.
There are a number of questions relating to this error when trying to run via the DTExe directly; but haven't found any questions regarding troubleshooting the Dts.Runtime DLL.
I presume that the DLL calls DTExe, but I don't know to tell which DTExe it will call. Using the "wrong" DTExe is the solution to many other questions regarding this error.
Whether you're using the managed object model or using the tooling, both are going to check to see whether the SSIS Service is installed on the machine. It is a requirement of licensing to run an SSIS package that the server running the package be a licensed SQL Server instance. So, the application you've written hopefully is targeted for running on a machine that already has a SQL Server server standard license or above.
For development, that runs you about 50 bucks last I checked. For production, we're talking tens to hundreds of thousands of dollars.
For completeness, SQL Server 2005 and 2008 SSIS packages would run on machines that had the data tooling installed (then called the Business Intelligence Design Studio, or BIDS) because the only way to acquire the installation media was to have the ISO images of SQL Server, Developer edition or above.
With 2012, you could now download the SQL Server Data Tools, SSDT without having the SQL Server product but you lost the ability to run packages outside of the Visual Studio debugger - which is a shame as the debugger can severely impact the throughput of a package.
If you check the licensing requirements - scaling out SSIS to different machines is not included in the Enterprise edition license.
SQL Server does not need SSDT installed to run an SSIS package. The option for installing the tools to run and interactive with it (dtutil, dtexec, ssisdeploymentwizard) .. might exist or not depending on version. But that's running packages. Development is where SSDT comes into play. SSDT designs and runs packages within the context of development (Visual Studio debugger). Anything else is going to get blocked by that licensing check.
I created a package that will run in SSDT, but it will not run through the execute package utility when saved to the filesystem. Perhaps there is an easy fix, but I am unaware of one, and googling didn't seem to yield anything useful.
I am running SQL Server Enterprise 2017, as well as Visual Studio 2017 with the most recent version of SSDT. I have tried installing SQL Server 2018 to no avail.
The point is scheduling the package, one solution would be for package to run through DTExec.exe, alternatively if there is a way to automate running the package through SSDT that would also work.
The error:
To run a SSIS Package outside of SQL Server Data Tools you must install Standard Edition of Integration Services or higher.
The catch:
afaik I have SQL server enterprise installed as I can save SSIS packages to the file system through the SQL Server Import Export Wizard (before I received enterprise edition it said originally it required SQL server standard edition or higher to save the packages)
Make sure that you have selected the Integration Services check box while installing SQL Server features. It looks like you have installed SQL Server Data Tools for Visual Studio which allows you to create SSIS packages but you didn't install SSIS feature which is required to execute packages.
You can refer to the following official documentation for additional information:
Install Integration Services
Make sure you have select the relevant TargetServerVersion property in the project configuration. You can refer to the following link for more information:
How to change TargetServerVersion of my SSIS Project
I'd like to know the difference between SSDT in Visual Studio and Integration Services in SQL Server. When developing SSIS package locally using SSDT in VS I could pass data to my local SQL Server without Integration Services being installed in SQL Server instance. I'm wondering if I need Integration Services to allow between servers communication. Let's say when SQL Server DB is on one server and .dtsx package is on another.
This is where Microsoft's marketing department has run amok. It is important to understand that Microsoft is not a company that writes code. Microsoft is a marketing company that happens to write code.
The simple answer is that SSDT is the package of project templates for what is intended to be Business Intelligence.
SSDT contains three template types:
SSIS: Integration Services
SSRS: Reporting Services
SSAS: Analysis Services
UPDATE
I failed to answer the question of running the packages. Basically, you can run any SSDT package against SQL Server within Visual Studio. However, if you want to deploy the SSDT packages to the SQL Server, then you must have those services installed. The services can be installed via the SQL Server instance installation wizard. You will need to be mindful of another hairy concept that is known as SQL Server Version Targeting: Click Here
For example, if you want to deploy and run SSIS packages to SQL Server, then you will need to install Integration Services (this will include DTExec.exe and ISDeploymentWizard.exe). Now, you will also need to install SSISDB to the SQL Server in order to be able to deploy SSIS packages to the SQL Server - this is performed via SQL Server Management Studio (SSMS). The actual packages are both deployed to, and managed from, a folder called Integration Services Catalog. The packages can then be automatically scheduled to run via the SQL Server Agent. It is extremely unlikely that you will ever work directly with the SSIDB, other than perhaps querying it for information: see here.
See Microsoft's instructions: click here
SSRS packages are managed through a separate Web-UI and I have not dealt with SSAS packages. Isn't this fun!?
A note on DTExec.exe
I have ran into purists who disdain SSIS, largely because they do not understand it. The general argument that I get is that it is slower than PowerShell or Stored Procedures. This may cause someone's head to explode.
Basically, PowerShell runs through the .NET Framework, which is in C# and has to run through a few layers in the OS in order to execute. While SSIS components are written in C#, the DTExec.exe application is written in C++, which can access system resources directly (C# cannot do this because it is managed code!). So, SSIS is going to blow PowerShell and Stored Procedures away in large tasks.
Stored Procedures are a different animal, but still slower because they lack the pipeline buffer (i.e. data flow tab). Another major limitation is how SQL Server executes Stored Procedures and that is sequentially. So, let us imagine that a comparable SSIS job is broken into multiple stored procedures and that those procedures are called by a main stored procedure - a super stored procedure so to speak. SQL Server will execute the stored procedures sequentially, one at a time - this is a huge performance bottleneck. SSIS's pipeline buffer obliterates that by processing a default of 10,000 rows (this is configurable btw) in each task and then passing them off to the next task. So, we can think of data flow tasks as their own stored procedure.
Additional Context
There is a long running confusion with what constitutes SSIS as it relates to Visual Studio, not necessarily SQL Server.
Pre 2005: It was Data Transformation Services (DTS)
2005 & 2008: In 2005, DTS was substantially overhauled and renamed to SSIS late in the development. That is why everything in SSIS still references DTS (i.e., *.dtsx files). It remains the case to this day. Odd, only a masochist likes DTS. BUT! The package of templates was renamed to Business Intelligence Development Studio (BIDS)
2012 & 2013: It was renamed to SSDT-BI. Apparently, there was already another product named SSDT
2015 and forward: It is now named SSDT
See Microsoft's attempt to explain SSDT: click here
Up through Visual Studio 2017 (VS 2017), SSDT and its various incarnations have been largely treated like the idiot step child to Visual Studio. I say this because VS was installed as a standalone product for these project types only. I don't know why it was done this way - my best guess is because SSDT is free. Anyways, if you wanted to use Visual Studio for other application development then you had to install a separate instance of Visual Studio. So, us developers quite literally have two standalone installations on our dev box and we have to use the specific install for whatever we are doing (i.e., SSDT or non-SSDT development).
Now, with VS 2019, Microsoft is doing away with this model and has finally integrated the SSDT package into the product. Though, the initial roll out of VS 2019 for SSDT was a comedy of errors right out of the box. See my explanation by clicking here. Basically, SSIS does not install with the package and has to be added separately. Though, you still have one instance of VS 2019. Additionally, the SQL11 data provider has been deprecated. And, that too apparently does not come with the installation package either and needs to be installed separately. So, any existing packages that use it will need to be upgraded and re-deployed (see Known Issue #1).
I am holding off on upgrading to VS 2019 for now. VS 2017 has been a pain to say the least. I personally still use VS 2013 Update 5. All VS instances are targeting SQL Server 2014.
Beside of #JWeezy good and detailed answer, i'd like to add a brief explanation:
SQL Server Data Tools for Visual Studio are the development environment for the SQL Server business intelligence suite (SQL Server Integration Services, Reporting Services, Analysis Services).
SQL Server Integration Services (installed from SQL Server installation), install all the files needed to run SSIS packages on the local machine.
Both products can run .dtsx packages but the first one is only for development and testing purposes while the second one is for production server.
References
Previous releases of SQL Server Data Tools (SSDT and SSDT-BI)
SQL Server Integration Services
Hi I'm looking for some guidance.
I've developed an SSIS project with a couple of script tasks and script components using visual studio 2015 for targeted version 2016.
I have two environment server A and server B both have SSIS 2016 SP1 installed on it.
I have done a project deployed of the packages to Sever A and tested the packages everything executed perfectly.
However when I deployed the packages in SSIS_DB from Server A to Server B I get these errors in my execution report.
The component metadata for "script component, clsid { }" could not be upgraded to the newer version of the component. The PerformUpgrade method failed
2
There was an exception while loading Script Task from XML: System.Exception: The Script Task "" uses version 15.0 script that is not supported in this release of Integration Services. To run the package, use the Script Task to create a new VSTA script. In most cases, scripts are converted automatically to use a supported version, when you open a SQL Server Integration Services package in %SQL_PRODUCT_SHORT_NAME% Integration Services.
at Microsoft.SqlServer.Dts.Tasks.ScriptTask.ScriptTask.LoadFromXML(XmlElement elemProj, IDTSInfoEvents events)
The interesting part is if I deploy my packages directly from visual studio to Server B the packages executes correctly with no issue.
So there seems to be inconsistencies between server A and Server B.
A small background to this is that when both servers were built Server A had Microsoft SQL Server 2016 (SP1-GDR) and Server Microsoft SQL Server 2016 (RTM)
but they both now have Microsoft SQL Server 2016 (SP1-GDR) after Server B was updated.
I'm not sure how the upgrade was done by my suspicion is that the SSIS on server are different but I can't seem to prove it.
when I check the version they both seem to return same or similar meta data
The right answer is here: https://andyleonard.blog/2017/09/ssis-version-15-0-script-error/
In my case the problem was the setting at the project level “TargetServerLevel” set to “SQL Server 2017”. The correct value must be “SQL Server 2016”. Once the value is changed and the project redeployed errors desappear.
I believe one of developers in my team used VS 2017 when added the project. All other tasks and components, excep script, were fine even with incorrect server version and the problem appeared only when I had to use script tasks and components.
To fix the problem open project parameters => Configuration Properties => TargetServerVersion
Select “SQL Server 2016”, then deploy to the server.
This was caused by SSMS. I don’t know what’s going on under the hood but During the deployment I was using SSMS 2017 which seems to be upgrading script components in my packages. I changed to use SSMS 2016 and this issue disappeared
Just summarizing some things because there are some reasons for facing this problem:
Your are deploying a package with Target Version Definition for another SQL Server
You are deploying a package which contains a script task with not same SSMS version as SQL Server Version
If your package doesn't contain a script task for example, then any version of SSMS can deploy the package to any version of SQL Server via SSISDB catalog. Not sure why is this happening. Seems that you can deploy and execute packages while making sure that the package target version is supported from SQL Version, but also if you have a ScriptTask inside you need to be super careful and make sure also that the integration services version of SSMS is the same with SQL Server version.
I will open a question to Microsoft just to be sure why the script task has this treatment. Maybe is because the script task is Framework dependent.
We are moving all of our SSIS packages from SQL Server 2008 R2 to SQL Server 2016. I suspect each SSIS package will need to be upgraded but I have been told otherwise.
Will all SSIS packages need to be upgraded to to work on the new SQL 2016 server? Should they be upgraded on the SQL 2008 R2 Server and then moved to the new Server? Or should we move the packages and then do the upgrades?
Also we have a large number of SSIS packages to upgrade. Will the SSIS Upgrade Wizard do these upgrades in bulk?
Here is a rough process that I am proposing we follow:
Use Data Migration Assistant to check for potential issues
Use SSIS Upgrade Wizard to update the packages
Manually resolve any issues
Test the packages
Are there any other things I should be concerned with?
I upgraded a set of SQL Server Integration Service packages recently by just importing them to a SQL Server Data Tools project.
The upgrade is done automatically and hands free by Visual Studio. Then you can attach them to 2016 SQL Server Integration Service .
I also recommend you check the following link from Microsoft that describes how to upgrade SSIS packages using the Package Upgrade Wizard :
Upgrade Integration Services Packages Using the SSIS Package Upgrade Wizard
The way we are upgrading is to create new projects/solutions in SSDT 2016 (Visual Studio), import the existing 2008 packages into the 2016 projects, and letting Visual Studio perform the upgrade while converting them to the Project Deployment model. Then we deploy them to the 2016 SSIS Server.
There are multiple ways to do it, and which way is "best" is a matter of opinion and may vary from case to case.
I agree that the best way to migrate is to create a new project in the VS 2016 solution, as the projects for VS 2008 and VS 2016 differ and you can not simply migrate them with any tool.
Here were my steps for migrating:
1) Open Command Prompt, run SSISUpgrade.exe from C:\Program Files\Microsoft SQL Server\130\DTS\Binn (Note path may vary depending on the version of SSDT you are using)
2) In the SSIS upgrade tool specify the path to VS 2008 solution and follow all the wizard steps up to finish.
3) Open the upgraded solution with VS 2016, remove the project from it(perhaps it didn't loaded correctly)
4) Create the new project of type Business Intelligence - Integration Services Project
5) Right click on the project -- add existing item, and then add dtsx package, config file for, all Powershell scripts and all other stuff than is used by this dtsx package
Keep mind, that VS 2016 will create a new folder for the project inside the solution folder, and place dtsx files, and all the stuff you've added in step 5.
And all of it should be kept in this separate folder, otherwise solution won't work.