I've read about the use of Catalogs in 2012/14 SSIS as a replacement for Configurations in 2008. With that replacement, I haven't seen how people handled the scanario of a configuration that is used by all packages on the server such as a Server Connection or path location. With this scanario, all packages point to one configuration, and should something about that value change, all packages are updated. Is this possible with catalogs? It seems each project has their on catalog and if that is the case, everytime a server wide config / parameter changes, it needs to change in each project.
In the SSSIDB, a project lives under a folder. A folder may also contain an SSIS Environment.
When you right click on a project (or package) and select Configure, this is where you would apply configurations, much as you did in 2008. You can use an SSIS Environment that exists in the same folder as the projects, or you can reference one in a different folder. That is the approach I use and suggest to people.
In my Integration Services Catalog, I have a folder called "Configurations" (because it sorts higher than Settings). Within that, I create one Environment called "General". Many people like to make environments called Dev, Test, Prod but unless you have 1 SSIS server handling all of those, I find the complexity of getting my deployment scripts nice and generic to be much too painful.
I then deploy my projects to sanely named folders so the Sales folder contains projects like SalesLoadRaw, SalesLoadStaging, SalesLoadDW.
If I have created a new project, then I need to add a reference to Configurations.General collection and then associate the project item to the Environment item. For Connection Strings, you do not need to define a Variable to accept the string. You can directly assign to the properties of a connection manager (either project or package scoped).
The great thing about Configurations is that once you've assigned them, they persist through redeploys of the project.
The biggest thing that tends to bite people in the buttocks is that when you create an Environment and add those entries into them, DO NOT CLICK OK. Instead, click the Script button and script those to new window. Otherwise, you have to recreate all those entries for your dev/test/load/stage/production environments. I find it far cleaner to script once and then modify the values (SLSDEV to SLSPROD) versus trying to create them all by hand.
Related
I've worked a lot with Pentaho PDI so some obvious things jump out at me.
I'll call Connection Managers "CMs" from here on out.
Obvious, Project CMs > Package CMs, for extensability/ re-usability. Seems a rare case indeed where you need a Package-level CM.
But I'm wondering another best practice. Should each Project CM itself be composed of variables? (or parameters I guess).
Let's talk in concrete terms. There are specific database sources. Let's call two of them in use Finance2000 and ETL_Log_db. These have specific connection strings (password, source, etc).
Now if you have 50 packages pulling from Finance2000 and also using ETL_Log_db ... well ... what happens if the databases change? (host, name, user, password?)
Say it's now Finance3000.
Well I guess you can go into Finance2000 and change the source, specs, and even the name itself --- everything should work then, right?
Or should you simply build a project level database called "FinanceX" or whatever and make it comprised of parameters so the connectoin string is something like #Source + # credentials + # whatever?
Or is that simply redundant?
I can see one benefit of the parameter method is that you can change the "logging database" on the fly even within the package itself during execution, instead of passing parameters merely at runtime. I think. I don't know. I don't have a mountain of experience with SSIS yet.
SSIS, starting from version 2012, has SSIS Catalog DB. You can create all your 50 packages in one Project, and all these packages share the same Project Connection Managers.
Then you deploy this Project into the SSIS Catalog; the Project automatically exposes Connection Manager parameters with CM prefix. The CM parameters are parts of the Connection Manager definition.
In the SSIS Catalog you can create so called Environments. In the Environment you define variables with name and datatype, and store its value.
Then - the most interesting part - you can associate the Environment and the uploaded Project. This allows you to bind project parameter with environment variable.
At Package Execution - you have to specify which Environment to use when specifying Connection Strings. Yes, you can have several Environments in the Catalog, and choose when starting Package.
Cool, isn't it?
Moreover, passwords are stored encrypted, so none can copy it. Values of these Environment Variables can be configured by support engineers who has no knowledge of SSIS packages.
More Info on SSIS Catalog and Environments from MS Docs.
I'll give my fair share of experience.
I recently had a similar experience at work, our 2 main databases name's changed, and i had no issue, or downtime on the schedules.
The model we use is not the best, but for this, and for other reasons, it is quite confortable to work with. We use BAT files to pass named parameters into a "Master" Job, and basically depending on 2 parameters, the Job runs on an alternate Database/Host.
The model we use is, in every KTR/KJB we use a variable ${host} and ${dbname}, these parameters are passed with each BAT file. So when we had to change the names of the hosts and databases, it was a simple Replace All Text Match in NotePad++, and done, 2.000+ BAT Files fixed, and no downtime.
Having a variable for the Host/DB Name for both Client Connection and Logging Connection lets you have that flexibility when things change radically.
You can also use the kettle.properties file for the logging connection.
I have a database called CommonDB. I have created shared data sets from this database in one of my report projects. I now have a need to include the same shared data sets in another report project. Ideally it would be nice if I could just point it to the testing site in BIDS and develop my report based on a reference.
I was wondering if there is a way to do this without adding existing data set (as I was hoping to keep the code base the same so I wouldn't have to update it in different projects).I am aware you can also add existing data sets from url but that defeats the purpose as it just downloads a copy to my report solution and it's not synced.
Any ideas? Thanks :)
This scenario is not supported by BIDS/Visual Studio.
It gets worse: If you deploy multiple projects to the same server, using the same Shared Datasets folder, then at runtime they will try to reuse each-others Shared Dataset definitions. The latest deployed Shared Dataset definition will win.
Data Sources have similar issues - although they tend to be less volatile.
To avoid this, I prefer to keep all reports for a single SSRS server instance in a single BIDS/Visual Studio Project. I use Project Configurations to manage deployment of reports to disparate folders.
I am working on bringing an application comprised of several SqlServer databases into a source control system. I've come up with a solution using several project files, each representing one of the databases, but in order for it to compile, the database project references need to be defined for several database projects, or I get errors regarding missing dependencies and the like. With the references set up, the solution compiles ok.
However, with the solution publishing, I will need to publish the referenced database projects first if I don't want to get 'Invalid object name' script publishing errors on their referencing projects. I would like to have this configured and be able to publish by just clicking on the 'publish solution' button. Is there a way to define 'publication dependencies', similar to the compilation dependencies, that will allow me to do this?
I had something like this - had to turn off transactions (so I wouldn't roll back), then publish several times. In my instance, I only had one DB that required that, but I had to:
Publish DB A
Publish DBs B/C/D/E
Re-publish DB A to catch the rest of the objects.
I had a special Publish profile set up that did not use transactions just for this purpose. Other than that, you're probably going to have to keep some sort of track about dependencies if you create a lot of new objects that depend on other databases.
I have not come across any publication dependencies so far, but it would be helpful to have something like that to avoid these sorts of issues.
I am in the process of writing an offline-capable smartclient that will have syncing capability back to the main backend when a connection can be made. As a side note, I considered the Microsoft Sync Framework but since I'm really only going one-way I didn't feel it would buy me enough to justify it.
The question I have is related to SQLite vs. SQLCE and ClickOnce deployments. I've dealt with SQLite before (impressive little tool) and I've dealt with ClickOnce, but never together. If I setup an installer for my app via ClickOnce, how do I ensure during upgrades the local database doesn't get wiped out? Is it possible to upgrade the database (table structure, etc. if necessary) as part of the installer? Or is it better to use SQLCE for something like this? I definitely don't want to go the route of installing SQL Express or anything as the overhead would be far too high for what I am doing.
I can't speak about SQLLite, having never deployed it, but I do have some info about SQLCE.
First, you don't have to deploy it as a prerequisite. You can just include the dll's in your project. You can check this article which explains how. This gives you finite control over what version is being used, and you don't have to deal with installing it per se.
Second, I don't recommend that you deploy the database as a data file and let ClickOnce manage it. When you change that file, ClickOnce will publish it again and put it in the data directory. Then it will take the previous one and put it in the \pre subfolder, and if you have no code to handle that, your user will lose his data. So if you open the database file to look at the table structure, you might be unpleasantly surprised to get a phone call from your user about the data being gone.
If you need to retain the data between updates, I recommend you move the database to the [LocalApplicationData] folder the first time the application runs, and reference it there. Then if you need to do any updates to the structure, you can do them programmatically and control when they happen. This article explains how to do this and why.
The other advantage to putting the data in LocalApplicationData is that if the user has a problem and has to uninstall and reinstall the application, his data is retained.
Regardless of the embedded database you choose your database file (.sqlite or .sdf) will be a part of your project so you will be able to use "Build Action" and "Copy to Output Directory" properties of that file to control what happens with the file during the install/update.
If you choose "Do not copy" it will not copy the database file and if you choose "Copy if newer" it will only copy if you have a new version of your database file.
You will need to experiment a little but by using these two properties you can have full control of how and when your database file is deployed/updated...
Developers have expressed a desire to deploy a single database object from a Sql Server 2008 project, such as a stored procedure, without having to use the build/deploy or schema comparison functions.
To enable this, developers have created their database object scripts including 'if exists.. drop' checks at the top of the script and have included the grant statements for the objects in their scripts.
This results in build errors that then prevent the build/deploy or schema compare functions from operating. So then, developers mark the object as "not in build" but then the object can't be deployed at all using build/deploy or schema compare.
Has anyone found a way of quickly deploying a single database object from visual studio that does not involve schema compare or build/deploy which does not remove the object from the normal build process? Manual copy/paste is not an option but scripting/macros which effectively do the same would be viable.
SQL Server Data Tools (SSDT) now provides this functionality by way of comparison. Individual differences identified in the comparison may be deployed. We have found that during development, publishing tends to result in overlaying the simultaneous changes that other developers are making on the shared development database server. The comparison tool has been working fairly well, except for a very annoying crash issue that occurs when the comparison window is closed. We are using 32bit Vista with 3GB of RAM and VS 2010. This issue may not occur in other configurations.
First I'd like to comment that you seem to be fighting the intended paradigm with regards database projects.
If your project is deployed somewhere [1],
then there should be a corresponding branch / label [2] in your source repository.
If you change a single object [delta] from the above baseline [2], and build the new version [3],
then when you deploy [3] to [1], the deployment script should find only one difference, and the net effect of the change [delta] is all that will be applied.
So, in theory there's no reason not to just simply build and deploy.
TIP: When taking on a new paradigm, it pays to embrace it fully; partial adoption tends to cause its own set of problems.
However, that said, I may be able to help.
We have a similar need because actual deployment is out of our control. We control only part of the database, and have to provide our changes to another team for review.
We have to provide individual 'self-contained' scripts for each object with if exists..drop at the drop and grant permission at the bottom.
However, we want the other benefits of working with the database project, and then simply copy out the individual script files when we "deploy".
The solution we came up with was to place the extra "bits" in a comment block as follows:
/*SINGLE_OBJECT_DEPLOYMENT
if exists (...)
DROP ...
--*/
--/*SINGLE_OBJECT_DEPLOYMENT
if exists (...)
DROP ...
--*/
Note that a simple search and replace of /*SINGLE_OBJECT_DEPLOYMENT with --/*SINGLE_OBJECT_DEPLOYMENT enables the commented out code, so it can be easily put into a macro, or other semi-automated process.