I'd like to know the difference between SSDT in Visual Studio and Integration Services in SQL Server. When developing SSIS package locally using SSDT in VS I could pass data to my local SQL Server without Integration Services being installed in SQL Server instance. I'm wondering if I need Integration Services to allow between servers communication. Let's say when SQL Server DB is on one server and .dtsx package is on another.
This is where Microsoft's marketing department has run amok. It is important to understand that Microsoft is not a company that writes code. Microsoft is a marketing company that happens to write code.
The simple answer is that SSDT is the package of project templates for what is intended to be Business Intelligence.
SSDT contains three template types:
SSIS: Integration Services
SSRS: Reporting Services
SSAS: Analysis Services
UPDATE
I failed to answer the question of running the packages. Basically, you can run any SSDT package against SQL Server within Visual Studio. However, if you want to deploy the SSDT packages to the SQL Server, then you must have those services installed. The services can be installed via the SQL Server instance installation wizard. You will need to be mindful of another hairy concept that is known as SQL Server Version Targeting: Click Here
For example, if you want to deploy and run SSIS packages to SQL Server, then you will need to install Integration Services (this will include DTExec.exe and ISDeploymentWizard.exe). Now, you will also need to install SSISDB to the SQL Server in order to be able to deploy SSIS packages to the SQL Server - this is performed via SQL Server Management Studio (SSMS). The actual packages are both deployed to, and managed from, a folder called Integration Services Catalog. The packages can then be automatically scheduled to run via the SQL Server Agent. It is extremely unlikely that you will ever work directly with the SSIDB, other than perhaps querying it for information: see here.
See Microsoft's instructions: click here
SSRS packages are managed through a separate Web-UI and I have not dealt with SSAS packages. Isn't this fun!?
A note on DTExec.exe
I have ran into purists who disdain SSIS, largely because they do not understand it. The general argument that I get is that it is slower than PowerShell or Stored Procedures. This may cause someone's head to explode.
Basically, PowerShell runs through the .NET Framework, which is in C# and has to run through a few layers in the OS in order to execute. While SSIS components are written in C#, the DTExec.exe application is written in C++, which can access system resources directly (C# cannot do this because it is managed code!). So, SSIS is going to blow PowerShell and Stored Procedures away in large tasks.
Stored Procedures are a different animal, but still slower because they lack the pipeline buffer (i.e. data flow tab). Another major limitation is how SQL Server executes Stored Procedures and that is sequentially. So, let us imagine that a comparable SSIS job is broken into multiple stored procedures and that those procedures are called by a main stored procedure - a super stored procedure so to speak. SQL Server will execute the stored procedures sequentially, one at a time - this is a huge performance bottleneck. SSIS's pipeline buffer obliterates that by processing a default of 10,000 rows (this is configurable btw) in each task and then passing them off to the next task. So, we can think of data flow tasks as their own stored procedure.
Additional Context
There is a long running confusion with what constitutes SSIS as it relates to Visual Studio, not necessarily SQL Server.
Pre 2005: It was Data Transformation Services (DTS)
2005 & 2008: In 2005, DTS was substantially overhauled and renamed to SSIS late in the development. That is why everything in SSIS still references DTS (i.e., *.dtsx files). It remains the case to this day. Odd, only a masochist likes DTS. BUT! The package of templates was renamed to Business Intelligence Development Studio (BIDS)
2012 & 2013: It was renamed to SSDT-BI. Apparently, there was already another product named SSDT
2015 and forward: It is now named SSDT
See Microsoft's attempt to explain SSDT: click here
Up through Visual Studio 2017 (VS 2017), SSDT and its various incarnations have been largely treated like the idiot step child to Visual Studio. I say this because VS was installed as a standalone product for these project types only. I don't know why it was done this way - my best guess is because SSDT is free. Anyways, if you wanted to use Visual Studio for other application development then you had to install a separate instance of Visual Studio. So, us developers quite literally have two standalone installations on our dev box and we have to use the specific install for whatever we are doing (i.e., SSDT or non-SSDT development).
Now, with VS 2019, Microsoft is doing away with this model and has finally integrated the SSDT package into the product. Though, the initial roll out of VS 2019 for SSDT was a comedy of errors right out of the box. See my explanation by clicking here. Basically, SSIS does not install with the package and has to be added separately. Though, you still have one instance of VS 2019. Additionally, the SQL11 data provider has been deprecated. And, that too apparently does not come with the installation package either and needs to be installed separately. So, any existing packages that use it will need to be upgraded and re-deployed (see Known Issue #1).
I am holding off on upgrading to VS 2019 for now. VS 2017 has been a pain to say the least. I personally still use VS 2013 Update 5. All VS instances are targeting SQL Server 2014.
Beside of #JWeezy good and detailed answer, i'd like to add a brief explanation:
SQL Server Data Tools for Visual Studio are the development environment for the SQL Server business intelligence suite (SQL Server Integration Services, Reporting Services, Analysis Services).
SQL Server Integration Services (installed from SQL Server installation), install all the files needed to run SSIS packages on the local machine.
Both products can run .dtsx packages but the first one is only for development and testing purposes while the second one is for production server.
References
Previous releases of SQL Server Data Tools (SSDT and SSDT-BI)
SQL Server Integration Services
Related
I have a SSIS package that was originally developed to target SQL Server 2015 that does a variety of things in C# Scripts using SQL Server Management Objects (SMO) that now needs to be upgraded to target SQL Server 2019. Eventually, this package is deployed to the server to run in a SQL Agent Job.
Starting in SQL Server 2017, SMO was moved to a NuGet package and is no longer included in SQL Server Feature Packs as it was for previous versions.
The issue I have is that Integration Services Projects do not really support NuGet. As discussed in the below link:
SSIS Script Task cant find reference to assembly
As mentioned in the above ticket, I believe I could manually install the .dll's to the server if necessary but I am curious if there are other solutions to this issue.
Questions:
Is there a way to get Script Tasks in SSIS packages to utilize the SMO NuGet?
If "no" to the above, is there another common method of executing C# script via SQL server other than installing the .dll's to the server?
You would be better off installing SMO DLLs into target server Global Assembly Cache. SSIS still does not use local assemblies and you have to introduce some workarounds to defeat that.
You can do a simple MSI installation with SMO DLLs - for example see WIX deploy two assemblies to GAC or What's the 'correct' way of registering/installing an Assembly to the GAC?.
I need to understand what a certain job in SQL Server 2012 does. It's a job someone created and left the company before I started to work here, and nobody on my team knows what this job does also.
The job isn't SQL command based but SSIS Package (which I'm not familiar with), the package points to a Maintenance Plan with the same name as the job. As I read on the internet I connected via Integration Services type, then Stored Packages -> MSDB -> Maintenance Plans, right click on the Maintenance Plan and exported it to a .dtsx file.
I opened it in Visual Studio 2010 Shell, but I can't edit anything because of this error:
The task with the name "" and the creation name "" is not registered for use on this computer.
I also don't have a Solution Explorer for that package, and the icons of the tasks seem a bit faded compared to tasks icons of a new project if I create one.
Maybe the dated version of the VS can be the problem? Perhaps there is other way to see what this job does?
I never worked with SSIS before so maybe I'm missing something very basic but I've been on the Internet for days already and can't find any solution.
Please help,
Thanks in advance
By way of background, until Visual Studio 2019, VS, by itself, couldn't open an SSIS package.
Prior to VS 2010, you needed to use a different product, called Business Intelligence Design Studio (BIDS), which was based on VS, but was built for SQL Server functionality. With the release of SQL Server 2012, Microsoft created SQL Server Data Tools (SSDT) as a plugin for VS that required a separate installation. Until SSDT(VS) 2017, you needed to have stand-alone versions of BIDS or SSDT for each version of SSIS you were working with in your environment. The 2017 version, though, is backward compatible to SQL Server 2012, and forward compatible (!) to SQL Server 2019. Visual Studio 2019 has SQL Server Data Tools sort of built in, but you need to add extensions through the extension manager for SQL Server Integration Services (SSIS), Analysis Services (SSAS), and/or Reporting Services (SSRS).
That's kind of a lot to take in, and is sort of irrelevant to you, but it forms the context for Larnu's comments. Rather than messing around with antique versions of Visual Studio, you should just install either VS2017 and also SSDT, or VS2019 with the SSIS extension. Both are in current widespread use, so support is easy to find, and either will allow you to work with the 2012 package you're trying to open up.
Once you have the software installed, you should be able to create a "dummy" integration services solution that you can use to open random packages. I keep one around called MiscSolution. Right click on the solution name, select Add -> Existing Item. Then find your .dtsx package and import it. (Or right click SSIS Packages -> Add Existing Package. I always do it at the Solution level because that interface allows you to select multiple packages.)
Note that when you do this, you're making a new copy of the package in your local solution directory. You are NOT working on the copy of the package that sits in the folder where you picked it up. This matters if you're going to be making any changes, since it will need to be redeployed. It also matters if you accidentally hose up the package, because you've done no harm as long as you just delete that copy.
Once the package imports, which might take a minute or so depending on how complex it is, you should be able to open it up and see what's going on.
I am trying to run a flatfile SSIS package via .Net code using the Microsoft.SqlServer.Dts.Runtime DLL version 14.0.0.0.
Upon executing I get "To run a SSIS package outside of SQL Server Data Tools you must install Standard edition of Integration Services or higher"
I have SSIS for Visual studio 2017 installed, along with with SQL server 2017 both have the SSIS extension installed.
There are a number of questions relating to this error when trying to run via the DTExe directly; but haven't found any questions regarding troubleshooting the Dts.Runtime DLL.
I presume that the DLL calls DTExe, but I don't know to tell which DTExe it will call. Using the "wrong" DTExe is the solution to many other questions regarding this error.
Whether you're using the managed object model or using the tooling, both are going to check to see whether the SSIS Service is installed on the machine. It is a requirement of licensing to run an SSIS package that the server running the package be a licensed SQL Server instance. So, the application you've written hopefully is targeted for running on a machine that already has a SQL Server server standard license or above.
For development, that runs you about 50 bucks last I checked. For production, we're talking tens to hundreds of thousands of dollars.
For completeness, SQL Server 2005 and 2008 SSIS packages would run on machines that had the data tooling installed (then called the Business Intelligence Design Studio, or BIDS) because the only way to acquire the installation media was to have the ISO images of SQL Server, Developer edition or above.
With 2012, you could now download the SQL Server Data Tools, SSDT without having the SQL Server product but you lost the ability to run packages outside of the Visual Studio debugger - which is a shame as the debugger can severely impact the throughput of a package.
If you check the licensing requirements - scaling out SSIS to different machines is not included in the Enterprise edition license.
SQL Server does not need SSDT installed to run an SSIS package. The option for installing the tools to run and interactive with it (dtutil, dtexec, ssisdeploymentwizard) .. might exist or not depending on version. But that's running packages. Development is where SSDT comes into play. SSDT designs and runs packages within the context of development (Visual Studio debugger). Anything else is going to get blocked by that licensing check.
We are planning to upgrade our SQL Server 2104 to SQL Server 2016.
Our SQL Server is in a clustered setup, and infrastructure is planning to do a side-by-side upgrade (where they will spin up new windows server machines and move resources from old machines to new machines).
We have 99% SSIS packages stored in file system, and 2% in SSISDB (Integration Services Catalog).
We have SSRS reports too.
What should be our approach towards SSIS packages and SSRS reports, while SQL Server is getting upgraded.
Should we copy the SSIS packages to Visual Studio, and change TargetServerVersion, and paste back to the folder location?
Should we copy report server databases (2 of them) and database encryption key to new machine, and spin up new SSRS along with existing URL.
In your situation, the best is to add all the existing packages in one project and upgrade the project.
In the development studio for SQL Server 2016, this is really interesting feature that you can upgrade as well as downgrade existing projects.
Project Properties Dialog
Post upgrade tests your packages in lower environment, the only challenge could be any .net code tasks if used. sometimes code is not upgraded to the latest version.
For report server your strategy is fine.
List item
Scenario:
Production servers are running SQL Server 2008 in the domain myDomain.com
Dev/test/stage servers are running SQL Server 2008 in the domain dev-myDomain.com
Actual dev work done on local running SQL Server 2008 source controlled using SourceSafe
First of all, does this setup/environment make sense? And, what are some good ways of deploying projects including SSAS, SSIS, SSRS from local to the dev servers and finally into live?
Is including the output (compiled) files in SourceSafe and deploying from there good practice? If it is any ideas?
Or, is going straight from Visual Studio without having the output in SoureSafe better?
Thanks.
We rely heavily on the use of variables for the items we want to change in differnt configurations and then config files (one each for Dev, QA, Staging, Prod). All SSIS packages and config files are in Source Save.
Visual Studio has configurations you can set up in order to deploy objects to different servers.
You can also use utilities to copy some around:
DTUTIL for SSIS - delivered with SSIS
RSScripter for SSRS - third party free tool, which is very useful
I would think you would use the deploy tool in Visual Studio for SSAS, as it needs to populate the model. I am just getting started with SSAS, so there may be other tools.