I have created a SSIS package. It'll load a XML file and store the data in the database.
I need to use a value (FILE_NAME) from FILE_INFO which is passed from the XML file into the OLE DB Destination in SQL Command mode. How can I use the FILE_NAME in the sql query.
This is my SQL query inside OLE DB Destination
Insert into DummyFile(DummyFileName, DummyFileStatusID)
VALUES ('I NEED TO INSERT THE FILE NAME HERE',
(Select FileStatusID from DummyFileStatus where StatusName='Created'));
Please advice.
I think you are looking for a Derived Column
If you want to insert the values into another OLEDB Destination DummyFileStatus, you can add a MutliCast transformation that allow you to insert data into multiple OLEDB Destination.
Or Just add another DataFlow Task that will be executed after this DataFlow task, to import Data from DummyFile to DummyFileStatus and use a Derived Column within it.
Additional Informations
Derived Column Transformation
Multicast Transformation
Related
I am filling a table using a package and after preparing the data to be saved I present it to an OLEDB destination for SQL server and setting the data acces mode to Table or View - Fastload or just Table or view (no fast load).
There is no error message but it does not write to the table.
I switch over to the normal Table or View so that each record is inserted with a separate INSERT command instead of a BULK insert.
When nothing happens I stop the execution of the package and do a select * from the destination table. I saw that he inserted 20 records. After investigation the data which is send to the OLE DB destination, I saw that record 21 results in a duplicate key.
Instead of getting an error message the package does not continue its execution flow.
What am I doing wrong.
Go through all the elements of the package, including the package itself, and set:
FailPackageOnFailure = True
Objective: automate creating a table along with all its columns and data types given a SSIS source
My guess is:
1) Pointing Sources to a Destination to a SQL command
2) Using Select * into ... Problem is I don't know what the from equivalent of a source is
Alternative) Store results in Recordset and pass on to Execute SQL task. Problem then is how to access that result from execute sql task
I think you should use a Recordset Destination to store data into an System.Object Variable, Then use a Script Task (starts after that Data Flow Task is executed) in which you will select the System.Object Variable as ReadOnly Variable. and you will write your own code to insert the Recordset to SQL using System.Data.SqlClient.SQLCommand Object
You can refer to one of these links
Issues with SSIS Script Task and SQL Bulk Insert - C#
Insert DataTable into SQL Table in C#
If you need just the structure of table use this trick
select top 0 * into NewTable from YourTable
I have created an SSIS package to extract and load data from access table (.mdb) named TmShipping to a SQL table say, TmShippingImport, this SQL table has Id and importedDate as additional columns. I have scheduled the package to run for every 30 minutes.
TmShipping.mds
----------------------------------------
OrderId CarrierId TotalCharge
TmShippingImport (SQL table)
-------------------------------------------------------
Id OrderId CarrierId TotalCharge importedDate
In the Data Flow Task:
I am getting the data from source using OLD EB connection and extracting all row data from the access table and the output of this is connected to a Recordset Destination so that I can extract each row.
In the Control flow task:
I have a loop container (connected to data flow task's o/p) which inserts each row into the sql table by a sql query and loads all the row data along with the current datetime.
Package execution
The SSIS package when executed for the first time loads each row into the SQL Table and add a DateTime to imporatDatetime Column. When new records are created in the Access table, the package now takes all the rows in the MS access (rows that were extracted previously and new rows) and loads them in the SQL table again. How to avoid duplicates? My primary key in SQL table is ID which is not present in the MS Access Table.
Tried using Lookup table in the Dataflow in between source and recordset destination but it failed saying I can't connect the available column to BLOB.
Should I be trying with Lookup Merge in the dataflow or should I make changes in the foreachloop container so that it checks for duplicates before inserting the rows into the sql table or...?
First of all, you're approach to inserting the data into the SQL table seems very expensive. Is there a reason you can't simply insert the data in to the SQL table in the data flow using a DataFlow destination, it will be much, much more performant.
If the reason you've not done this is because of the timestamp, you can achieve this be using a derived column transformation and GetDate().
Once you've done that you can implement the pattern used in the answer Tab linked to.
Ok so according to Microsoft docs the OLE DB Command Transformation in SSIS does this
The OLE DB Command transformation runs an SQL statement for each row in a data flow. For example, you can run an SQL statement that inserts, updates, or deletes rows in a database table.
So I want to write some SQL to Insert rows in one of my tables only IF the record doesn't exists
So I tried this but the controls keeps complaining of bad sintaxys
IF NOT EXISTS
(SELECT * FROM M_Employee_Login WHERE
Column1=?
AND Column2=?
AND Column3=?)
INSERT INTO [M_Employee_Login]
([Column1]
,[Column2]
,[Column3])
VALUES
(?,?,?)
However if I remove the IF NOT EXISTS section (leaving the insert only) the controls says may code is Ok, what am I doing wrong.
Is there an easier solution?
Update: BTW My source is a Flat File (csv file)
Update since answer: Just to let people know. I ended up using the OLE DB Command Transformation like I planned cause is better than the OLE DB Destination for this operation. The difference is that I did used the Lookup Component to filter all the already existent records (like the answer suggested). Then use the OLE DB Command Transformation with the Insert SQL that I had in the question and it worked as expected. Hope it helps
OLEDB Command object is not the same as the OLE DB Destination
Rather than doing it as you describe, instead use a Lookup Component. Your data flow becomes Flat File Source -> Lookup Component -> OLE DB Destination
In your lookup, you will write the query SELECT Column1, Column2, Column3 FROM M_Employee_Login and configure it such that it will redirect no match entities to the stream instead of failure (depending on your version 2005 vs not 2005) this will be the default.
After the lookup, the output of No Match will contain the values that didn't find a corresponding match in the target table.
Finally, configure your OLEDB Destination to perform the fast load option.
Though you can make use of Look up component in SSIS to avoid the duplicates which is the best possible approach, but if you are looking for some query to avoid the duplicates then, you can simply insert all the data in some temp/staging table in your database, and run the following query.
INSERT INTO M_Employee_Login(Column1, Column2, Column3)
SELECT vAL1,vAL2,vAL3 from Staging_Table
EXCEPT
SELECT Column1, Column2, Column3 FROM M_Employee_Login
I want to create a copy of a table, say TestTable, with a new name, say TestTableNew, in the same database with the use of an SSIS package. I've created a "Transfer SQL Server Objects Task" for this with the source database specified as both the SourceDatabase and the DestinationDatabase. When I run this task, the original table TestTable is overwritten with a new -empty- TestTable.
This might well be something really obvious that I've overlooked, but can I somehow specify another name for the destination table somewhere in this transfer task? Or should I solve this in another way?
You can't use the "Transfer SQL Server Objects Task" to copy a table to the same database because there isn't an option to specify the new table name. You would be copying table "TestTable" to table "TestTable", which will fail because they both have the same name.
You can set the "DropObjectsFirst" property to true, but that will make you lose your original table and its data, which I think you did on your test, otherwise you would have received a failure message.
The best option here is to use an "Execute SQL Task" to create the structure of your TestTableNew based on your TestTable and then do a simple OleDBSource -> OleDBDestination transformation to load all the data from one table to another.
My knowledge of SSIS is very limited but I assume you can run sql commands passing in
parameters and therefore generating something like the following dynamically
select *
insert into TestTableNew
from TestTable