I have created SSIS package for fuzzy lookup.
I just want to know how to make each following properties dynamic passed to execute package for any database table's column.
OLEDB_Source - Server, Database,Table and Column name.
FL_Large_Data - Server, Database,Table and Column name.
FL_Large_Data - Similarity threshold.
OLE DB Destination - Server, Database and Table name.
Since you are aiming to use this package for different tables and columns (Maybe you can if you have a fixed and unified table structure), it cannot be achieved using expressions. You must automate the package creation in order to do that, you have many choices:
Use BIML (Business Intelligence Markup Language )
Use SQL Server DTS libraries
Use some Wrapper libraries such as EzApi
For each one of the choices above, there are many tutorials found online, you can refer to them in order to create the package.
Related
I am relatively new to SSIS and have to come up with a SSIS package for work such that certain tables must be dynamically moved from one SQL server database to another SQL server database. I have the following constraints that need to be met:
Source table names and destination table names may differ so direct copying of table does not work with transfer SQL server object task.
Only certain columns may be transferred from source table to destination table.
This package needs to run every 5 minutes so it has to be relatively fast.
The transfer must be dynamic such that if there are new source tables, the package need not be reconfigured with hard coded values.
I have the following ideas for now:
Use transfer SQL Server object task but I'm not sure if the above requirements can be met, especially selective transfer of tables and dynamic mapping of columns.
Use SQLBulkCopy in a script component to perform migration.
I would appreciate if anyone could give some direction as to how I can go about meeting the requirements and if my existing ideas are possible.
I have more than 500s of tables in SQL server that I want to move to Dynamics 365. I am using SSIS so far. The problem with SSIS is the destination entity of dynamics CRM is to be specified along with mappings and hence it would be foolish to create separate data flows for entities for 100s of SQL server table sources. Is there any better way to accomplish this?
I am new to SSIS. I don't feel this is the correct approach. I am just simulating the import/export wizard of SQL server. Please let me know if there are better ways
It's amazing how often this gets asked!
SSIS cannot have dynamic dataflows because the buffer size (the pipeline) is calculated at design time (as opposed to execution time).
The only way you can re-use a dataflow is if all the source to target mappings are the same - Eg if you have 2 tables with exactly the same DDL structure.
One option (horrible IMO) is to concatenate all columns into a massive pipe-separated VARCHAR and then write this to your destination into a custom staging table with 2 columns eg (table_name, column_dump) & then "unpack" this in your target system via a post-Load SQL statement.
I'd bite the bullet, put on your headphones and start churning out the SSIS dataflows one by one - you'd be surprised how quick you can bang them out!
ETL works that way. You have to map source, destination & column mapping. If you want that to be dynamic that’s possible in Execute SQL task inside foreach loop container. Read more
But when we are using Kingswaysoft CRM destination connector - this is little tricky (may or may not be possible?) as this need very specific column mapping between source & destination.
That too when the source schema is from OLEDB, better to have separate Dataflow tasks for each table.
In MS SQL Server 2012/oracle, I need to create something like a.b.c
where c is the table/view name, b is the package name and a is the
schema name.
I wanted to create a table/view in two levels. I couldn't find any
proper document regarding this. Any suggestion to find the good document
will be helpful.
MS SQL Server and Oracle are very different. They're both relational DBs, but different vendors and don't function quite the same. There are similarities though.
At any rate, you don't create tables under a package. A table is an object much like a package is an object. A package is designed to hold a collection of procedures/functions within Oracle. SQL Server doesn't even implement packages.
So, as an Oracle example, the best you're going to get is Database > account (or schema..) > table.
I want to generate a typed DataSet for both Oracle and DB2 databases from a single XSD file.
Background:
When I drag a table onto the Visual Studio Dataset Designer, it generates a XSD file (e.g. DataSet1.xsd) with all appropriate information. Then VS invokes the built-in Tool (MSDataSetGenerator) to generate the Dataset classes into the CS file.
Somehow (I don't know how) CS determines the Data provider and generates Dataset classes that refer to the data provider (OracleDataProvider or IBM.Data.Provider).
Question:
Does anybody know how I can customize VS to compile both for DB2 and Oracle code generation ?
Best regards
Oliver
Hi the problem is not in the structure of the data (set,tables).
It's the automatically generated adapters that create a dependency on the database/connection.
The first way to circumvent this (a bit) is using ODBC connections for your adapters.
Then the connectionstring determines which database you use..
However.. the SQL syntax is always a little different.
Personally I just make a DataSet.xsd with only datatables (new datatable instead of new dataadapter).
Then I write my own repositories/adapters myself. So one for oracle and one for DB2.
These repositories/adapters fill the datatables..
So you can share the model between databases but you implement database specific adapters.
I would like to publish some data of a sql server 2k to msaccess databases.
I'd like to do that given a table supplying datatransformation info, for example :
tablenameOnServer | pathToPub
------------------------------------------------------
Clients | D:\Data\Pub1\ClientData1.mdb
Orders | D:\OtherData\Pub\Sales.mdb
The given mdb file should be created (or an empty one copied of course) and the table should be created each time the script runs.
I of course don't need a full blown solution, but some pointers as where to start are very welcome. I thought I'd use SSIS for this, but am new to it and I like to know where I start best in order to avoid too much loss of time :
Do I use SSIS with BIDS (vs2008), can I read data in a package and create tables on the fly?
Do I use C# and manipulate and create packages in code?
Or what do I do best? Is SSIS the obvious solotion anyway?
In any case : some pointers to get me started would be very welcome...
UPDATE : This question is about publishing data, so it can be shipped on CD for example. It's not about linking to an sql server.
The simplest solution is to simply link the sql server table in access. Then you can see the data in realtime.
You can create an Access database to do this fairly easily.
Here is a basic algorithm
Within access, create a linked table pointing to your table.
create an Access query to filter data according to your criteria called NewQuery
Use VBA to create new database NewDB
export your NewQuery with structure and columns to the NewDB
If you want to keep the SSIS packages, I would make a template SSIS package and then using the SSIS objects from C# create the packages. I've done this to start off packages which are later customized.
Alternatively, if you aren't going to keep the packages after they are generated and aren't going to use a lot of SSIS features (logging etc.), I would consider doing the whole thing from C# (or other language), because SSIS isn't a great tool for connection managers (source or destination) which change.
From C#, you can simply read the schema, create the table in Access with matching types and columns and populate it.