I would need to migrate a SQL database from Sybase to MS SQL Server. Before doing the actual migration on the production server I first created an SSIS-package with SQL Server Management Studio's Import/Export Wizard for testing with other databases. The test migration was successful and I would now like to deploy my SSIS-package to the real servers.
However, it seems I cannot simply run the package in Management Studio choosing different data sources for it - it only runs on the same databases for which it was created. Now, it can be edited in something called SQL Server Business Intelligence Development Studio (or BIDS for short)(I am using the SQL Server 2008 version), but going through every data flow task changing the destination source manually for each of the ~ 150 tables I am moving is ineffective and also introduces a possibility for error.
I there a way to quickly change what data source is to be used for ALL destination sources in ALL the flow tasks of an SSIS-package? If not, what simple method is there for testing migration with test databases first and simply changing the data sources when deploying?
I am using ODBC data sources, but for some the package shows OLE-sources in BIDS instead.
I hope I was clear enough. If you have additional questions, please ask! Thank you!
I would use a variable for the ConnectionString property of the connection manager. A package level configuration can be very useful for accomplishing this task. Several ways to do this. I prefer to use a table in SQL Server that holds all the configurations for all packages. This can be especially effective if you have multiple packages and need to dynamically change a set of connection managers across those multiple packages.
The basic steps are:
Opposite click on your SSIS design surface and select "Package Configurations..."
Create a package level configuration of Configuration Type "SQL Server"
Store your connection in a Configuration table in SQL Server
Alter your Connection Manager to use a variable for the ConnectionString Property
Populate that variable from the Configuration table via your package level configuration
When it comes time to switch from Test to Production, simply update the connection string in your configuration table
These screenshots can help...
This is part of a larger package management framework that I implemented using this book:
Microsoft SQL Server 2008 Integration Services: Problem, Design, Solution
I highly recommend it. Should take less than a day to set it up. Book has step by step instructions.
This question and its associated answers also helpful.
Related
As a company we have grown and we are now moving a couple of SQL Server 2016 databases over to a new server. We have SSIS packages that run off the databases that we are moving from server 1 to server 2.
Is there a way to easily identify using SSMS which SSIS packages use the current server and databases we are moving? Some of the old SSIS packages don't have documentation so we are trying to avoid physically opening up all the SSIS packages. We would prefer to identify the SSIS packages that are impacted.
Thank you!
Here are my solutions on the top of my head. I'm not an expert by any means so don't be surprised if someone comes up with something better.
In SSMS, you can view the data sources being utilized by a
package by... Object Explorer > SQL Server Agent > Jobs > (Your Job) > Steps > Edit... > Data Sources (Tab)
Here, you can view the data sources of your package. This is
slightly faster than opening all your SSIS packages; but, it isn't a
great solution either.
Conversely, recognize that .dtsx files are simply plain text
files. You can scan keywords within all of them using a number of
different scripts (PowerShell, Python, SSIS package w/ a Script
Task, etc.)
What you can use depends on the tech stack that your organization
supports but I imagining Googling for such a program/script would
not be difficult.
If you are utilizing SQL Server configurations in your packages
and you consistently do so for every package, you can query the
[SSIS_Configurations].[dbo].[SSIS Configurations]
NOTE: Solution (1) and (2) do not take configurations into account.
Hopefully, some of these solutions are helpful to you. I would be interested in an efficient means to do this without delving into scripts as well.
In order to enable operational management of data integration processes developed in SSIS, I am seeking to be able to externally configure:
server (data source)
database (catalog)
schema
From what I have seen, all of these are typically hardcoded into SSIS packages through the Connection Manager and in SQL statements. This hardcoding limits the DBA from being able to allocate resources differently and, if there is ever a change, requires every package to be modified if Package Deployment is being used.
It appears that the Project Deployment would reduce this somewhat, but no eliminate it.
Target environment is SQL Server 2016 and VS 2017.
How can the server, database, and schema be externalized from the package?
SSIS has a robust facility for configuring packages per environment. You can configure any property in the package externally. This can be done in SQL Agent and even from the command line at runtime. Configurations can be stored in config files, system environment variables, a SQL table, etc. However, the modern way of configuring packages is through the project deployment model.
Here is the gist of how it works:
Add a parameter at the package or project level
reference that parameter in an expression which configures the property you want to set, i.e. the server name or initial catalog
Deploy the project to an instance of SSIS
In SSIS, add an environment and configure the variable. This can even be passwords which are securely stored
Add a reference to that environment from the project, and finally reference which environment you want to use at runtime.
The first link below shows a dialogue that was created for configuring connection managers with parameters. Please note that the package will store the default values, but when you create an environment as noted above, this allows you to easily set it at runtime.
As for configuring a schema, this is possible as well, by using parameters, but you would need to use expressions for your SQL queries and setting the destination. I would avoid making schemas variable across environments. This will present a lot of effort and complexity for very little flexibility that is offered in return. Please read up on these links and good luck!
How to parameterize connection managers
All about parameters in SSIS
I have developed 25 SSIS packages with SQL Server backend. Now my requirements have changed so that the SSIS package stays the same but the database which I use should shift from SQL server to Oracle Database,
How I can change the DB in my SSIS package from SQL Server to Oracle?
Well, I do not think, there is a way to do that. SSIS stands for SQL Server Integration Services for a reason :)
Oracle does have a similar framework called Golden Gate:
http://www.oracle.com/us/products/middleware/data-integration/goldengate/overview/index.html
I guess, you would have to start from scratch. But since you already have got the concepts - assuming your SSIS-Packages are working - to create your Data Integration Services, you will "only" have to get familiar with Golden Gate.
Are you kidding? This can be solved. All you have to do is edit the packages to use the source and destination as your oracle databases. The rest remains the same. It also depends on how the data is processed internally in packages.
If your packages are using any transformations in Data Flow and if the data flow is using any sql commands then you need to think of that too. Also, if your control flow is using any SQL task then also you need to think of that.
You can edit a single package and try converting it to Oracle. SSIS can treat Oracle data as well, this is not a limitation of SSIS.
I'm looking for the best approach (or a couple of good ones to choose from) for extracting from a Progress database (v10.2b). The eventual target will be SQL Server (v2008). I say "eventual target", because I don't necessarily have to connect directly to Progress from within SQL Server, i.e. I'm not averse to extracting from Progress to a text file, and then importing that into SQL Server.
My research on approaches came up with scenarios that don't match mine;
Migrating an entire Progress DB to SQL Server
Exporting entire tables from Progress to SQL Server
Using Progress-specific tools, something to which I do not have access
I am able to connect to Progress using ODBC, and have written some queries from within Visual Studio (v2010). I've also done a bit of custom programming against the Progress database, building a simple web interface to prove out a few things.
So, my requirement is to use ODBC, and build a routine that runs a specific query on a daily basis daily. The results of this query will then be imported into a SQL Server database. Thanks in advance for your help.
Update
After some additional research, I did find that a Linked Server is what I'm looking for. Some notes for others working with SQL Server Express;
If it's SQL Server Express that you are working with, you may not see a program on your desktop or in the Start Menu for DTS. I found DTSWizard.exe nested in my SQL Server Program Files (for me, C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn), and was able to simply create a shortcut.
Also, because I'm using the SQL Express version of SQL Server, I wasn't able to save the Package I'd created. So, after creating the Package and running it once, I simply re-ran the package, and saved off my SQL for use in teh future.
Bit of a late answer, but in case anyone else was looking to do this...
You can use linked server, but you will find that the performance won't be as good as directly connecting via the ODBC drivers, also the translation of the data types may mean that you cannot access some tables. The linked server might be handy though for exploring the data.
If you use SSIS with the ODBC drivers (you will have to use ADO.NET data sources) then this will perform the most efficiently, and as well you should get more accurate data types (remember that the data types within progress can change dynamically).
If you have to extract a lot of tables, I would look at BIML to help you achieve this. BIML (Business Intelligence Markup Language) can help you create dynamically many SSIS packages on the fly which can be called from a master package. This master package can then be scheduled or run ad-hoc and so can any of the child packages as needed.
Can you connect to the Progress DB using OLE? If so, you could use SQL Server Linked Server to bypass the need for extracting to a file which would then be loaded into SQL Server. Alternately, you could extract to Excel and then import from Excel to SQL Server.
In the SO podcast episode 54 Jeff talked about using Visual Studio to save all the database objects to individual files. This sounded like just what my team needed to better implement database schema changes into TFS and I told my lead about it. He thinks it's a great idea to.
Unfortunately, so far I've had no luck getting this to work for me. One of my problems is that I don't have SQL Server installed on my local box (dept policy). I'm obviously doing something wrong.
Can someone give me a rundown of the steps or provide a decent link?
Thanks!
1) Create a connection to the database in Server Explorer.
2) Right-click on the connection and select Publish to provider...
3) Next, Script to file, Next, Types of data to publish should be Schema (unless you need some initial data), finish.
4) Add script to the project and check into source control.
He was referring to a project type in VS that supports managing databases, including (if you set it up correctly), versioning your database publishes.
Check out this article on Database Projects in VS
You want to use the GDR 2. (Sometimes called Data Dude)
This allows for a completely offline solution. You don't need to have SQL Server installed on your machine for this to work. (In fact the GDR is the first version that does not care if you have SQL Server installed.)
I use the GDR for my db (a team of 3 devs and 2 testers) and it works GREAT!
Here is a link for the GDR 2 release:
http://blogs.msdn.com/vstsdb/archive/2009/04/21/microsoft-visual-studio-team-system-2008-database-edition-gdr-r2.aspx
And this is a link to the actual bits:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=bb3ad767-5f69-4db9-b1c9-8f55759846ed
The basic use is to import from an existing DB and server. (You will probably want a server project and 1 or more DB Projects)
You can then make your changes off line. When you are ready to send you changes back to the DB you can delploy (make sure you set up your deploy options first as I think the default is to drop the db and re-deploy). You can also do a Data->Schema Compare in Visual Studio and comapre your project to your database then get a script of changes from the diff output.
It takes a bit of work, but it really allows great source control and is easy once you get the hang of it. (I have my db auto deploy in my night time build twice a week.)
If you're importing an existing database schema, its important to get the right database project. You should likely be using "Database Projects\SQL Server 2005 Wizard".
Visual Studio requires a database connection it can use to create temporary copies of the databases it is working with. It sounds like this is the issue. Do you have SQLExpress on your local box? Use that as the deployment target (server name would be .\sqlexpress if . does not work).
As an alternative, grab update GDR 2 for Visual Studio. Allegedly, it allows one to work with database projects without using a local instance of sql server to deploy temporary working copies of the database.