I'm currently in the process of upgrading from SQL Server 2008R2 to 2014 (both Enterprise). There are a plethora of SSIS jobs that are in production, and which will need to be migrated. I'm trying to get a handle on how I should manage SSIS jobs going forward.
In 2008R2, I would always using BIDS to deploy packages to MSDB. All permissions were then controlled through SQL Server.
In 2014, I see that you can still save to the file system or MSDB, but that there is now the SSISDB that you create as an Integration Services Catalog. This method clearly allows a lot more flexibility with the addition of easy variable access and even environment variables.
Is deploying SSIS packages to SSISDB in 2014 now the best practice way of deploying and managing SSIS projects, rather than to deploy to MSDB? Am I still able to manage permissions? When I backup SSISDB, are all of my deployed projects backed up (like before with MSDB)? Lastly, when I schedule these packages via the SQL Agent, do they still behave the same, where the permissions of the SQL Agent service account and job owner determine the SSIS package permissions when it is run?
Many thanks to anyone who can help. I've been on Microsoft's site all day, and while the documentation is very helpful, it doesn't actually answer these questions specific questions.
I recently took the SSIS Exam (70-463) so i can tell you some things about the new deploy model.
Short Answer:
Yes SSISDB is Best Practice. Packages can be deployed to SSISDB. Packages keep a deployment history (like a very basic version control) so you can even rollback some revisions of your package.
Main Advantage of the new model is the configuration. You don't need XML or dedicated SQL tables to save your configurations. You can use input parameters and map them with enviroments defined on sql server.
You can manage security through SQL Server because now everything can be handled via SQL Server Security.
Another cool feature is the Integration Services Dashboard, a report automatically built with report services template. Just click Integration Service Catalog and right click your packages to view "All Executions".
You can see very detailled Information about your packages including execution time.
Long Answer:
In my opinion the main advantage are the project parameters. Imagine this like Parameters you can pass to the SSIS Package. You can parametrize your Connection Manager or just parts of it.
Example: You can parametrize the server name and in your ssisdb you can create two enviroments (or more) called "development" and "production". Then you can add variables to both of them and map them to the input parameter of your package.
The main advantage is that you can deploy a package to SSISDB and link to an environment and you don't have to handle the connection strings by yourself.
Related
Microsoft SQL Server 2016. I've done an export of a table (using the wizard) I saved the package and I see it in the msdb.sysssispackages table. But it doesn't show up in the Integration Services Catalogue section. So how can I make it show up so I can execute this package again?
If you're executing it through SQL Agent, the Package Source is SQL Server for packages stored in the msdb
A confusing bit of terminology that I started to respond with on your deleted question is that you can store a package in SQL Server and that usually means the msdb.dbo.sysssispackages table. But we also have the project deployment model, which deploys an .ispac file into the SSISDB.
If you want a package to be in the SSISDB that is not of the project deployment model, the 2016(?) release of SSIS allowed for an Incremental package deployment for a project deployment model. You could use that approach to uplift your msdb based SSIS package into a new "project" without ever opening Visual Studio to create an actual solution.
That's a technical answer, it can be done that way but I would advise against it as you generally want to source control any processes that you operationalize.
The situation is that I have two DB servers with SQL Server, I need to transfer the data from Server A to Server B automatically every night.
So according to sql server data transfer from one server to another server, I created an SSIS project. And it manage to transfer data with good performance. But the question is, how do I export the project into SSIS package and run it every night?
I assume you are still running it from Visual Studio. To schedule it daily you will need to do the below,
Deploy your project to a SQL Server
Create an Agent job and schedule it.
Found a good article, sharing that with you here.
https://www.mssqltips.com/sqlservertutorial/9069/deploy-and-schedule-an-sql-server-integration-services-ssis-package-step-by-step/
You can either store the package as in the file system or in the SSIS catalog. I would say that storing it in the SSIS catalog is the best way to go here.
Before you deploy your package to the SSIS catalog you need to make sure that integration services is installed on your server as well as that the SSIS catalog has been created.
How to create ssis catalog
If all above is OK then proceed and deploy your package and create an agent job. Please notice that you need to set up permissions for the sql agent account so it can access your databases.
When setting up a SQL Agent job, there are options for choosing, among others, "SQL Server" or "SSIS Package Store". If I choose "SSIS Package Store" and browse to MSDB instead of File System, I can choose SSIS packages stored in sysssispackages.
However, the documentation for the /DTS switch, which SSMS automatically chooses when selecting SSIS Package Store, appears to be only for packages stored on the file system.
I've also noticed that if the user executing the job doesn't have file system rights, the job will fail when choosing SSIS Package Store --> MSDB --> Package with an "Access is denied" error or "~Object doesn't exist". Since the package doesn't exist on the file system though, why does execution work when choosing SQL Server and fail when using SSIS Package Store? Is dtexec actually looking for a dtsx file on the file system even when MSDB is chosen? I have, likely a false, understanding that SQL Server and SSIS Package Store --> MSDB are the same thing when referencing the Server on which the agent is running.
I've done some research and I can't find a clear distinction of the differences, if any, between using SQL Server and SSIS Package Store when selecting from the MSDB folder.
Can anyone shed some light on the differences?
There are 2 locations to store SSIS packages: The file system or SQL Server. The rest is semantics.
File System
You can use the SSIS Package Store which is nothing but a well known location in the installation location.
%Program Files%\Microsoft SQL Server\{Version}\DTS\Packages
Or you can pick anywhere on the file system you like. If you go this route, then you'll need to ensure the SQL Agent account, or the credentialed proxies or, if you running packages from xp_cmdshell the SQL Server Service Account has access to that location.
The only advantage, if you want to call it that, of using the Package Store (i.e. the folder I mentioned) is you can use the Integration Services management tool that exists in SSMS (by connecting to Integration Services instead of database engine).
However that has a lot of pitfalls such as not being able to handle multiple instances, packages only run in 64 bit mode, no access to proxy accounts, etc. You shouldn't run packages from SSMS anyway.
SQL Server
If memory serves correct
2005 - stored in msdb.dbo.sysdtspackages90
2008 - stored in msdb.dbo.sysssispackages (I seem to recall 2008 RTM using a different table, or reused the 90 table but that got patched out)
2008 R2 - stored in msdb.dbo.sysssispackages
2012 (package deployment model) - stored in msdb.dbo.sysssispackages
2012 (project deployment model) - stored in SSISDB.catalog.packages*
2014 (package deployment model) - stored in msdb.dbo.sysssispackages
2014 (project deployment model) - stored in SSISDB.catalog.packages*
*With the project deployment model, packages are "compiled" (zipped with a manifest) into a .ispac which is stored into the bowels of the SSISDB.internals.* tables.
Wrapup
Ultimately, where you store your packages does not affect your ability to run them. You can run packages using DTEXEC, SQL Agent or custom .NET code. The choice of storing packages is primarily dependent upon your management style.
References
Package Management
I am running SQL Server 2012 and VS 2010 with SSDT (SQL Server Data Tools) installed. My dev DB uses stored procs, functions, CLR objects, etc. It has a snapshot of prod data of about 500GB.
I created SQL Server Database Project and then imported the database. This created all tables, views, procs and functions files under schema names. Great stuff -- now I can do a version control just like in other VS projects, create deployments, etc. So far, so good.
But, I am confused as to what my development process should be for changing/adding procs/tables under SQL Server Database Project. It appears that any changes I make are applied to some LocalDb/Projects database and NOT to my dev database.
Am I suppose to author all my objects in that LocalDb, then Build and deploy to my dev database via Publish? I am worried about my existing tables in the dev DB since if the publish process drops and recreates tables, I will loose my prod data snapshot.
What is the right development process to follow in SQL Server Database Project?
Think of the source database (in your case, your database project) as being the "to be" state after deployment. When a deployment is initiated, the executable (SqlPackage.exe) compares the source with the target and generates a difference/delta script to make the target look like the source. This is why we no longer have to specify CREATE or ALTER; the tool figures it out. To answer your question about ongoing development, you can develop either way. You can develop in the project files and publish them to a common Dev database (say, if you're on a team), or you can develop in the database with tools like SQL Server Management Studio (SSMS) and synchronize with the project files with a schema compare (I use the latter technique because I like SSMS).
For deployment, you'll have to have SSDT installed on the machine from which you execute the deployment (SSDT ships with SQL Server 2012 and later; I don't know about SQL Server 2008). You can create scripts to simplify deployment. You'll essentially call SqlPackage.exe (it lives in x:\Program Files (x86)\Microsoft SQL Server\nnn\DAC\bin) with an action and a source. I use Publish Profiles as well to take care of most command properties. So an example deployment might look like this:
SqlPackage.exe /Action:Publish /SourceFile:MyDatabase.dacpac /Profile:MyProfile.publish.xml
For more information:
SQL Server Data Tools Documentation
http://msdn.microsoft.com/en-us/library/hh272686(v=vs.103).aspx
SqlPackage.exe Documentation
http://msdn.microsoft.com/en-us/library/hh550080(v=vs.103).aspx
Make changes inside the VS DB project.
Deploy changes to localDB to test
Publish the database to your production server. I prefer to use Schema Compare to do this manually, but you can also publish the project via the right click --> publish menu (which will also create a publishing profile), or using command line arguments. The publish process won't drop and create tables (unless you tell it to drop & recreate the entire db).
Alternatively, in the project settings you can change the connection string to point to your production server (as pointed out in the comment). However, I recommend against this, as it will then attempt to publish to the production server every time you run a local build (F5).
I'm managing a web app featuring countless ETL (Extract, Transform & Load) processes feeding a datawarehouse (using SSIS dtsx packages + *.sql files).
At the moment, everything is governed by several SQL Agent processes monitoring "scheduling queue" tables.
I'm trying to figure out a way to develop a scheduler or scheduling "framework" of some sort that needs to do a subset of what SQL Server Agent does (executing dtsx packages, executing SQL) but with a configuration UI based on ASP.NET, as users need to be able to create & modify schedules, manually launch processes and monitor custom logs.
Configuration info, activity, execution logs, schedules, jobs, job steps and everything else needed that I'm missing should be persisted to DB.
I'm guessing a Windows Service would be flexible but development costs might skyrocket.
I appreciate any sort of input, particularly some clues on SQL Server Agent's internal workflow. I DID try searching for info, but it being PROPRIETARY software, I found zilch.
(maybe I should specify I'm more than familiar with SQL Server and C# based .NET development, I just have no idea where to start from..)
The SQL server agent database is called 'msdb'. You can attach profiler to msdb and create some scheduled jobs to see how it works.
The gist of the answer is one should use SQL Server Service Broker, coupled with SQL Server Service Broker External Activator from the Feature Pack