Hi I am using SSIS (MSSQL) to copy data between multiple tables. This has been working fine up until recently when the S.A.P. team keeps updating the schema of the tables without telling me.
I have multiple tables that they continue to add columns to; this in turn makes my SSIS job of copying the data across fail.
Is there a way in SSIS that I can look at the source table and adjust my destination table to reflect the changes on the fly?
I'm quite new at SSIS and don't mind running a script out of the GUI but wondered if this was an option within the GUI I'm already familiar with.
So in short, can I in SSIS allow for new columns being added to source tables and update my destination tables automatically to stop my jobs failing
(Oh and map source to destination tables automatically)?
You'll have to include the new columns in the data flow, i.e. source and destination (include and map them). So basically you CANNOT automate what you're looking for in SSIS. Hope it helps.
Look into BiML Script, which lets you create and execute SSIS packages dynamically based on the meta data available at run time.
Related
I am relatively new to SSIS and have to come up with a SSIS package for work such that certain tables must be dynamically moved from one SQL server database to another SQL server database. I have the following constraints that need to be met:
Source table names and destination table names may differ so direct copying of table does not work with transfer SQL server object task.
Only certain columns may be transferred from source table to destination table.
This package needs to run every 5 minutes so it has to be relatively fast.
The transfer must be dynamic such that if there are new source tables, the package need not be reconfigured with hard coded values.
I have the following ideas for now:
Use transfer SQL Server object task but I'm not sure if the above requirements can be met, especially selective transfer of tables and dynamic mapping of columns.
Use SQLBulkCopy in a script component to perform migration.
I would appreciate if anyone could give some direction as to how I can go about meeting the requirements and if my existing ideas are possible.
What I want to do is build a dynamic data pull from different SQL source servers (Server1,Server2,Server3) etc.
To pull down to dynamic locations on my SQL server (Dev,Prod) into databases (database1,database2,etc)
The tables will be dropped and recreated each time the package is run so that I am sure I match the source servers if they change anything on source (field names, datatypes, lengths, etc)
I will still get the data to extract. I want to pull this down using a single dataflow in a foreach loop.
I have a table that has all the server names and tables and databases in it and
I want to loop through that table and pull all the rows of tables inside down to my server (server1.database1.table_x,server5.database3.table_y,etc) So that I don't have to build a new data flow for each table.
In order to do this I have already built the foreach loop with a sql task that is dumping results into an object. Then the foreach loop takes that object that has 7 different fields (Source_Server_Name,Source_Server_Type_Driver,Source_Database,Source_Table,Source_Where_Clause,Source_Connection_String,then destination stuff) and it puts each of those fields into a different String variable for use inside the loop.
I can change the Connections dynamically using the variables but I can't figure out how to get the column mapping in the dataflow to function,
Is there some kind of script task I can use to edit the backend XML that will create the column mapping for me so the metadata does not error out? Any help would be greatly appreciated :-)
This is the best illustrated example I could find of what I am doing just remember I need to have a different metadata setup for each table I pull down to my server.
http://sql-bi-dev.blogspot.com/2010/07/dynamic-database-connection-using-ssis.html
The solution I ended up using is BIML which generates the package on the fly using dynamic sql and BIML. Not pretty but it works :-)
I have heard that it is possible to dynamically generate and publish the packages but I would never go this route. I have done something similar using c# code which can be run from an application via sql agent or from inside an SSIS package script task.
If you try this approach look into SqlConnection and SqlCommand. Then write code to build the sql statements dynamically.
For example create table statements using ExecuteNonQuery(), use datareader to pipe in input and pass that reader to SqlBulkCopy to write to the destination.
I would like to publish some data of a sql server 2k to msaccess databases.
I'd like to do that given a table supplying datatransformation info, for example :
tablenameOnServer | pathToPub
------------------------------------------------------
Clients | D:\Data\Pub1\ClientData1.mdb
Orders | D:\OtherData\Pub\Sales.mdb
The given mdb file should be created (or an empty one copied of course) and the table should be created each time the script runs.
I of course don't need a full blown solution, but some pointers as where to start are very welcome. I thought I'd use SSIS for this, but am new to it and I like to know where I start best in order to avoid too much loss of time :
Do I use SSIS with BIDS (vs2008), can I read data in a package and create tables on the fly?
Do I use C# and manipulate and create packages in code?
Or what do I do best? Is SSIS the obvious solotion anyway?
In any case : some pointers to get me started would be very welcome...
UPDATE : This question is about publishing data, so it can be shipped on CD for example. It's not about linking to an sql server.
The simplest solution is to simply link the sql server table in access. Then you can see the data in realtime.
You can create an Access database to do this fairly easily.
Here is a basic algorithm
Within access, create a linked table pointing to your table.
create an Access query to filter data according to your criteria called NewQuery
Use VBA to create new database NewDB
export your NewQuery with structure and columns to the NewDB
If you want to keep the SSIS packages, I would make a template SSIS package and then using the SSIS objects from C# create the packages. I've done this to start off packages which are later customized.
Alternatively, if you aren't going to keep the packages after they are generated and aren't going to use a lot of SSIS features (logging etc.), I would consider doing the whole thing from C# (or other language), because SSIS isn't a great tool for connection managers (source or destination) which change.
From C#, you can simply read the schema, create the table in Access with matching types and columns and populate it.
I'm using SSIS with SQL Server 2k5 to build a transfer task to copy all of the data from one database to another. This works quite well, except for one problem - the source database will periodically have schema changes (generally just additions like new columns) but the transfer task seems to choke if the two schemas don't match exactly. Is there some way that I can use SSIS to first bring the target DB up to date with the source DB's schema, and then do the transfer?
You can open the package programmatically and re-save it before executing. You can also programmatically build the package using the SSIS object model.
I am trying come up with a way to pull the tables out of an Access database, automate the creation of those same tables in a SQL 2008 DB, and move the data to the new tables. This process will happen on a regular basis and there may be different tables each time.
I would like to do this totally in SSIS.
C# SQL CLR objects are an option.
The main issue I have been running into is how to get the Access table's schema and then convert that to a SQL script that I can run via SSIS.
Any ideas?
TIA
J
SSIS cannot adapt to new tables at runtime. (You can change connections, move a source to a table with a different name, but the same schema) So, it's not really easy to do what I think you are saying: Upsize an arbitrary set of tables in an Access DB to SQL (mirroring their structure and data, naming, etc), so that I can then write some straight SQL to transform the data into another SQL database or the same part of the database.
You can access the SSIS object model from C# and build a package (or modify a template package) programmatically and then execute it. This might offer the best bang for your buck, but the SSIS object model is kind of deep. The SSIS Team blog have finally started putting up examples (a year after I had to figure a lot of this out for myself)
There is always the upsizing wizard, and I'm sure there are some third party tools.