I am using SSDT and working on a simple SSIS package.
The Control flow:
1. A Foreach Loop Container and seek a folder exist a "importdata{}.csv" file or not.
2. If found, a script task will set variables:
- User::FullPath = (e.g C:\importdata{}.csv)
- User::varFileNameNoExt = (importdata{}) without extension.
The {} is possible in "toy","game","food".
3. Go to dataflow
The Data Flow:
1. Flat File Source with a flat file connection, the connection string is varible and mapped connection string expression.
2.ADO.NET Destination , insert data.
My question is how can i set the ADO.NET Destination [TableOrViewName] Property in variable?
Assume the table : importdatatoy,importdatagame and importdatafood is created on SQL Server.
I try to set as "dbo"."[User::varFileNameNoExt]" ,but it cannot resolve the table name on runtime.
ADO.NET Destination [TableOrViewName] parametrization can be done at Data flow level. In data flow properties, you can specify "ADO.NET Destination [TableOrViewName]".
Also specify the quotes while assigning value to variable
Eg: varFileNameNoExt = "dbo"."tableName"
But first you will need to create mapping with an existent table.
Can you post your error message? I'm thinking you won't be able to combine static text and a variable like that inside of the TableOrViewName field. Instead do the combination in a new [User::varTableName] SSIS variable and use the Advanced Properties Expression editor to set the TableOrViewName to this new SSIS variable. Have a look here.
Related
I am new to SSIS. I am trying to create a separate excel file dynamically in data flow task for each iteration of the for-each loop? Please guide
You can utilize the following approach.
Create an excel file template on the folder where you want to drop the new files.
Connect your excel file destination to the template file created in the folder.
Create two variables:
variable: IterationCount Data Type Int default value 1.
Variable: FileName Data Type: string
Expression = "Mybasefilename_" + (DT_STR, 4,1252)[User::IterationCount] + ".xlsx"
On your excel file connection hit right click and hit properties go to expression and hit three ellipses and look for filename property.
Set the property value choosing #[User::Filename] variable. If the Name property is not available use the connection string property, however, you should add the folder path as part of your filename variable to create the entire file destination and name.
Last step in your FELC you need to update the IterationCount variable in each iteration.
So, we cannot catch the index of the iteration then you need to use an expression in the FELC, expression task, or a script task to update the IterationCount variable.
Expression task example:
#[User::IterationCount] = #[User::IterationCount] + 1
Helpful Links:
Microsoft - SSIS ForEach Loop Container
SSIS Expression Task
SSIS - Updating variables using Script Task
I want to export some SQL Server tables to Excel but the Excel file must already exist otherwise the SSIS would throw an error. Any way to tell SSIS to create the excel file with the correct columns if the Excel file doesn't exist?
When I try to click on mapping without the proper Excel file already made it throws error below:
Create three variables in your SSIS Package
FolderPath : provide the folder path in which you want to check the file
FileName : provide the file name
FileExistsFlg : we will use this to indicate file exists or not. 1 means exists and 0 means does not exists.
After creating variables, Drag Script Task to the Control Flow Pane and Provide Variables and Finally Click on Edit Script :
In the Task Script :
public void Main()
{
String Filepath=Dts.Variables["User::FolderPath"].ValueToString()+Dts.Variables["User::FileName"].Value.ToString();
if( File.Exists(FilePath))
{ Dts.Variables["User::FileExissFlg"].Value=1; }
Dts.TaskResult= (int) ScriptResults.Success;
}
Use the variable FileExistsFlg to run the next tasks if file exists (FileExistsFlg=1).
Bring Data Flow task in Control Flow Pane and Connect Script Task to it.
Double click on Green line ( Precedence Constraint) between two tasks.
Choose Expression and Constraint from Evaluation operation drop down and in Value choose "Success". In expression write #FileExistsFlg==1.
This is the method I ended up going with. Grab a new "Execute SQL Task".
ConnectionType = EXCEL,
Connection = [YOU'R EXCEL CONNECTION MANAGER],
SQLStatement = CREATE TABLE YourTable(COL1 VARCHAR(10))
Finally set the Excel Destination DelayValidation = True and put the Execute SQL Task in front of the Excel destination somewhere in Control Flow.
This will probably throw an error if the file already exist but for my use case it's good enough.
I have been trying to create an SSIS task which picks the MS Access file from a specific folder
and then export to SQL Server ( if that file/table found in server then skip else export).
I am new to SSIS, i have used script task to select the file names dynamically and then trying to move, but I end up getting unsatisfied results . Even I have googled and got few ideas, but still not able to get it the way I wanted. Any detailed help would be very helpful.
Note : Here, am not always sure about the filename from that folder(i.e dynamic)
There are many options for dynamically selecting files. Since you're unsure about the filename, I'm assuming this is a parameter or variable. The following is an example of checking a folder from a variable for the given file name and loading it to an SSIS object variable. These files are then loaded into a SQL Server table using the Foreach Loop. You mentioned files as opposed to a single file, so this example assumes that only part of the file name is passed in, such as would be the case if the date/UID was appended to the beginning or end of the file name.
Add a Script Task, with the parameters/variables holding the file and folder name as ReadOnlyVariables and the object variable which will store the file names during execution as a ReadWriteVariable. The code for this is at the end of this post.
The string.IndexOf method is used to check for files containing the given text, with the StringComparison.CurrentCultureIgnoreCase parameter used to make this search case-insensitive. This example uses a variable for the file path and a parameter for the file name (denoted by $Package in the parameter name).
Add a Foreach Loop of the Foreach From Variable Enumerator Enumerator type. Add the object variable that was populated in the Script Task as the Variable on the Collection page. On the Variable Mappings pane, add a string variable at index 0. This will need to be an empty string variable that will hold the name of each file.
Create a Flat File Connection Manager from an example data file. Make sure that the column names and data types are appropriately configured. To set the file name dynamically, choose the ConnectionString expression (click the ellipsis of the Expression property in the Properties window of the connection manager) and add the same string variable from the Mappings Pane of the Foreach Loop.
Inside the Foreach Loop, add a Data Flow Task with a Flat File Source using the same connection manager. Then add either an OLE DB or SQL Server Destination with your destination connection and connect the flat file source to this. I've found SQL Server Destinations to perform better, but you'll want to verify this in your own environment before making the choice. Choose the necessary table and map the columns from the flat file source accordingly.
List<string> fileList = new List<string>();
//get files from input directory
DirectoryInfo di = new DirectoryInfo(Dts.Variables["User::FilePathVariable"].Value.ToString());
foreach (FileInfo f in di.GetFiles())
{
//check for files with name containing text
if (f.Name.IndexOf(Dts.Variables["$Package::FileNameParameter"].Value.ToString(), 0, StringComparison.CurrentCultureIgnoreCase) >= 0)
{
fileList.Add(f.FullName);
}
}
//populate object variable
Dts.Variables["User::YourObjectVariable"].Value = fileList;
I have package variable (TableName), its value is set by for each loop container.
For instance TableName = sales set by for each loop after looping through informationschema.tables then I want to use TableName's value in dataflow for this I am using script component.
When I use script component to read TableName's value in script-file I cannot see variable any idea why??
I am using script component as source and TableName is in readonlyvariables.
Please help as I am new to SSIS.
On the Variable Mappings page of the Foreach Loop Editor, assign variables to each item of data that is returned by a single enumerated item. For example, a Foreach File enumerator returns only a file name at Index 0 and therefore requires only one variable mapping, whereas an enumerator that returns several columns of data in each row requires you to map a different variable to each column that you want to use in the Script task. +
After you have mapped enumerated items to variables, then you must add the mapped variables to the ReadOnlyVariables property on the Script page of the Script Task Editor to make them available to your script
For more details pls chk the below link
https://learn.microsoft.com/en-us/sql/integration-services/extending-packages-scripting/task/using-variables-in-the-script-task
https://social.technet.microsoft.com/wiki/contents/articles/22194.use-ssis-variables-and-parameters-in-a-script-task.aspx
Please use the following Microsoft Documentation link - it describes differences between Script Task and Script Component, and gives small code samples, specifically on using Variables.
as per this I need to use this.variables.myvariable
Thanks everyone.
I'm learning SSIS and this seems like an easy task but I'm stuck.
I have a CSV file Orders.csv with this data:
ProductId,Quantity,CustomerId
1,1,104
2,1,105
3,2,106
I also have a stored procedure ssis_createorder that takes as input parameters:
#productid int
#quantity int
#customerid int
What I want to do is create an SSIS package that takes the .csv file as input and calls ssis_createorder three times for each row in the .csv file (the first row contains column names).
Here is what I have done so far.
I have created an SSIS package (Visual Studio 2005 & SQL Server 2005).
In Control Flow I have a Data Flow Task.
The Data Flow has a Flat File source of my .csv file. All of of the columns are mapped.
I have created a variable named orders of type Object. I also have variables CustomerId, ProductId, & Quantity of type int32.
Next I have a Recordset Destination that is assigning the contents of the .csv file into the varialbe orders. I'm not sure about how to use this tool. I'm setting the VariableName (under Customer Properties) to User::orders. I think that now orders holds an ADO record set made up of the contents from the original .csv file.
Next I'm adding a ForEach Loop Container on the Control Flow tag and linking it to the Data Flow Task.
Inside of the ForEach Loop Container I'm setting the Enumerator to "ForEach ADO Enumerator". I'm setting "ADO object source variable" to User::orders". For Enumeration mode I'm selecting "Rows in the first table".
In the Variable Mapping tab I have User::ProductId index 0, User::Quantity index 1, User::CustomerId index 2. I'm not sure if this is correct.
Next I have a Script Task inside of the ForEach Loop Container.
I have ReadOnlyVariables set to ProductId.
In the Main method this is what I'm doing:
Dim sProductId As String = Dts.Variables("ProductId").Value.ToString
MsgBox("sProductId")
When I run the package my ForEach Loop Container turns Bright Red and I get the following error messages
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::ProductId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 1 to variable "User::ProductId" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::Quantity" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 2 to variable "User::Quantity" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::CustomerId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 3 to variable "User::CustomerId" cannot be applied.
Warning: 0x80019002 at MasterTest: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (12) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Dts.TaskResult = Dts.Results.Success
Any help would be appreciated
One of my coworkers just give me the answer.
You don't need the the ForEach Loop Container or the RecordSet Container.
All you need is the Flat File Source and an OLE DB Command. Connect to your database and inside the OLE DB Command select the appropriate connection.
In the Component Properties enter the following SQLCommand:
exec ssis_createorder ?, ?, ?
The "?" are place holders for the parameters.
Next under the Column Mappings tab map the .csv file columns to the stored procedure parameters.
You are finished go ahead and run the package.
Thanks Gary if you were on StackOverFlow I would give you an upvote and accept your answer.
If I understand correctly, what you want to do is execute a stored procedure 3 times for each row in the data source.
What if you just create a data flow with a flat file data source and pipe the data through 3 execute sql command tasks? Just map the columns in the data to the input params of your stored procedure.
Maybe I'm not seeing it correctly in your question and I'm thinking too simple, but in my experience you need to avoid using the foreach task in SSIS as much as possible.
I suspect that you need to look at your Data Flow task. It's likely that the values from the source CSV file are being interpreted as string values. You will probably need a Derived Column component or a Data Conversion component to convert your input values to the desired data type.
And, I think #StephaneT's solution would be good for executing the SP.
I'm not sure if this answers your question. But I was looking to do this and I achieved it using the BULK INSERT command. I created a staging table with all of the columns in the csv file, and instead of a stored procedure I used a INSTEAD OF INSERT trigger to handle the logic of inserting it into many tables.