My question is similar to Upgrading SQL Server 2000 to 2005 or 2008 - DTS to SSIS but I have a few restrictions that I'd like the community to consider.
We have two databases that currently run on SQL Server 2000 and we are being pushed to move off of this in favor of SQL server 2005 (We do not have 2008, it isn't an option).
We have approximately 150 DTS packages that will need to be converted. According to the stack overflow post above, we shouldn't use the conversion tool available. We also anticipate that any "backward compatibility" patches that Microsoft may have released for SQL Server 2005 won't be installed on our server.
Can anyone help me understand the learning curve involved in going from DTS to SSIS? For 150 packages, we've come up with a 900 hour estimate. Myself and a more senior developer will be working on this but we've never worked with SSIS. The 900 hours will be divided between us. As with any project, quality is the utmost importance. Is this estimate outrageous or low?
We will have access to other developers who have worked with SSIS, so we won't be completely alone in this. Also, I've learned that DTS package conversion is not an option; it is not supported by our DBA staff.
Some things we do with DTS include the following. This is by no means an exhaustive list:
Load XML, XLS, MDB, CSV, etc files in to SQL Server
Export XML, XLS, MDB, CSV, etc files from SQL Server
Run Workflow-type processing
The built in conversion tool will not
migrate transaction settings
migrate ActiveX scripts associated
with steps
Analysis Services task
complex Data Transformation tasks
custom tasks
Data Driven Query tasks
Data Mining Prediction Query task
Dynamic Properties tasks
Execute Packages tasks
Parallel Data Pump tasks
tasks that obtain their connection
information from Microsoft Data Link
(.udl) files
ActiveX Script code that accesses
the DTS object model through the
Parent property of the
GlobalVariables collection
There is a tool called dtsXchange available from Pragmatic works which does most of these conversions, but still it does not re-engineer the package.
Typically, a DTS package would use staging tables and most of the transformation / data manipulation would happen using SQL or Stored procedures. Automatic conversion would continue to use the same work flow.
The SSIS engine is quite powerful and has a lot of features to help do these tasks. Rewriting the packages in SSIS would be the way to go.
Having said that, considering that you do not have any experience with SSIS, I feel that 900 hours for 150 packages is a bit conservative. I think you should plan for 8 hours per package, but aim to do it in 6 hours.
I hope you are not using 64 bit. Neither XLS nor MDB files have a 64 bit data provider and you will be forced to run in 32 bit mode.
I have been working on a strong argument for rewriting against conversion. I will gladly share all the documentation I have with you. Email me at b.rengarajan#gmail.com
Raj
Related
As a company we have grown and we are now moving a couple of SQL Server 2016 databases over to a new server. We have SSIS packages that run off the databases that we are moving from server 1 to server 2.
Is there a way to easily identify using SSMS which SSIS packages use the current server and databases we are moving? Some of the old SSIS packages don't have documentation so we are trying to avoid physically opening up all the SSIS packages. We would prefer to identify the SSIS packages that are impacted.
Thank you!
Here are my solutions on the top of my head. I'm not an expert by any means so don't be surprised if someone comes up with something better.
In SSMS, you can view the data sources being utilized by a
package by... Object Explorer > SQL Server Agent > Jobs > (Your Job) > Steps > Edit... > Data Sources (Tab)
Here, you can view the data sources of your package. This is
slightly faster than opening all your SSIS packages; but, it isn't a
great solution either.
Conversely, recognize that .dtsx files are simply plain text
files. You can scan keywords within all of them using a number of
different scripts (PowerShell, Python, SSIS package w/ a Script
Task, etc.)
What you can use depends on the tech stack that your organization
supports but I imagining Googling for such a program/script would
not be difficult.
If you are utilizing SQL Server configurations in your packages
and you consistently do so for every package, you can query the
[SSIS_Configurations].[dbo].[SSIS Configurations]
NOTE: Solution (1) and (2) do not take configurations into account.
Hopefully, some of these solutions are helpful to you. I would be interested in an efficient means to do this without delving into scripts as well.
I'm a beginner, and trying to improve my knowledge on DB side.
I am learning SSIS with SQL Server 2008R2. Going by the tutorials from Web, I feel like this is somewhat similar to what I've read about Oracle Data Pump.
Can someone enlighten me, if there is similarity between two SSIS and Data Pump.
If they are totally different, please forgive me for this question. Else, let me know how they are similar.
Regards,
Justin
Data Pump is not a complete ETL tool,it is a feature in Oracle 11g.It transfers the data from a single file to a single destination.With SSIs you got all the extraction ,transformation and loading facilities.
Corresponding to SSIS ,oracle has oracle warehouse builder.
Oracle data pump is an alternative to EXPORT and Import utility in SQL SERVER.
I have never heard of Data Pump but initial googlings show it is more related to being a Data Flow Task within an SSIS package than being a substitute for a whole SSIS package. Data Pump is simply porting data from a single source to a single destination. An SSIS package can facilitate extracting, transforming, and loading any amount of sources to any amount of destinations within the same package. You also get the extensibility (if that is a word?) of writing .NET code or any other 3rd party assemblies that you would like to use to further do manipulation of data. You can also do file and DB maintenance with an SSIS package (clean up after processing of files, maintaining backups, etc.).
In the 2008R2 version I was using SSIS logging to a sysssislog table in a defined database. 2012 brings now the concept of Integration Services Catalogs that have their own SSISDB db. Is it still necessary to use the logging to sysssisslog tables or is the information that ends there now somewhere in SSIS DB (what i would expect, since all the reporting for SSIS execution is based on this db as well).
The logging you are familiar with remains available to you with the 2012 release of SQL Server. That said, database logging no longer has to be explicitly defined in your package if you are using the Project Deployment model (SSISDB).
Out of the box, you'll get Basic logging level when you run a package. The other options are none, performance and verbsose. Read more about how to set these and other execution parameters via MVP Phil Brammer. Matt Masson of the actual SSIS team points out what events those levels correspond to in his post on What Events are Included in the SSIS Catalog Log Levels.
Finally, SSIS Reporting Pack is an open source project from MVP Jamie Thomson that provides different insight into the basic data being captured in the new integration services catalog.
My thoughts: necessary no. But if you already have a framework built out culling data from that log (we use it for an alerting system), you are supported to keep using it. If you run integration services packages from multiple servers, there is no functionality to combine the logging from all those disparate SSISDB catalogs to provide insight into your entire universe. You can get that if you all the packages log to a centralized server using the classic technique.
I will have a Postgres database in production but want to use MS SQL (whatever edition) for reporting. So, I would like to have replication set up where MS SQL subscribes from postgres. Is this possible?
All heterogeneous replication scenarios are deprecated by Microsoft, and they now recommend building solutions using SSIS and CDC instead.
We load data from PostgreSQL into our SQL Server reporting database using SSIS and it works well, although we had to use a commercial OLE DB provider because of limitations (at that time) in the open-source one.
Actually copying the data is usually the easy part; most of the work comes in gathering requirements, understanding the data, transforming it, implementing logging and error handling etc. SSIS can do some things for you right away (e.g. logging) but my general advice would be to use it primarily as a workflow tool and for simple data copying with minimal transformation logic (e.g. data type conversion). If something seems seems too difficult or clumsy in SSIS then you can put it into a stored procedure or script and call that from SSIS instead.
I've been using and following PostgreSQL for several years and not aware of such a solution. If one exists, I'm concerned that might be complex or fragile. I would recommend regular export/imports via cron. In the between the export and import, you would need to take care of the translation step of the formats.
If you reporting actually happens in MS Excel or MS Access, I recommend looking into connecting them directly to PostgreSQL via ODBC.
Our office uses SQL Server 2000, and by and large it has served our needs well. I'm looking into the possibility of upgrading to SQL 2005 or 2008 because of the new features not found in 2000 (when I saw Index usage statistics which told me which Indexes were the most used and which were never used, that blew me away).
I'm not too concerned with how to move the databases from 2000 to 2005 or 2008 - I can just take a full backup and then restore the backup to the new sql installation. What concerns me are the existing DTS packages. We have dozens of them that we rely on every day. How difficult/easy is it to convert the existing DTS packages to SSIS? What are some of the 'gotchas' that I have to look out for?
Doing a backup & restore to do a database server upgrade has many gotcha's. It's best to do an in-place upgrade (you might want to take a full backup of the server first) & let the MS installer do the work.
2005 supports 2000 DTS packages as Legacy so there's no need to learn SSIS & convert all your packages unless you have the time & there isn't too many.
As for the backup & restore upgrade, don't forget about:
logins, you'll need transfer these,
see KB246133 for more details
Any user and role permissions given
to system databases will need to be
re-created or transferred
linked server or remote servers will need to be re-setup
SQL Server Agent Jobs need to be exported-imported
Any Extended stored procedures need
to be re-created
Replication will need to be re-setup
Log shipping will need to be
re-setup & the standby server also
upgraded
There is a migration tool for DTS packages, but I wouldn't use it. SSIS is so much better than DTS that you should:
Learn about SSIS. Get comfortable with it. Rejoice at having actual control flow.
Take the time to understand and document your existing DTS packges while you've still got the environment they're written for.
Use that understanding to write equivalent SSIS packages.
This idea will make no sense to you if you don't know SSIS. For instance, I was just thinking about how I used to have to do loops in DTS - that whole business with finding and modifying the precedence constraint you wanted to loop back to.
SSIS actually has a For Loop container that can loop over a set of numbers, over the rows in a rowset, over files in a folder, etc. No hacks with precedence constraints and little hidden scripts.
DTS packages can still be run as usual in 2008 as long as Backwards Compatibility is installed. Here's the link to upgrading which uses a wizard.
I'd recommend a tool from Pragmatic Works called DTS xChange. It handles most of the migration and helps you get on with your job. You'll have other opportunities to learn SSIS on new stuff but you need to probably wrap up your project. It can be downloaded at http://www.pragmaticworks.com