In my SSIS package, I have a Script Component that creates a +1 new customer ID when the package runs.
I want to assign the numerical value of the ID to a variable. The variable will be used in a File System Task to create a directory with the Customer ID as the folder name.
Example - C:\Customer Orders\<CO_ID>
I have created a derived column (sequence) transformation in the data flow named (CO_ID). However, I am stuck on the proper syntax for the variable expression.
Any advice is appreciated.
Without the error message and more details, it is difficult to say what's wrong. But I think you're trying to convert a number to a string and then concatenate it to another string (the path). If that's the case, your derived column expression should be something like:
"c:\Customer Orders\" + (DT_STR, 10,1252) CO_ID
If the backslashes give you a problem (can't remember if they need to be escaped), then try doubling them up:
"c:\\Customer Orders\\" + (DT_STR, 10,1252) CO_ID
Derived column Transformation is used inside a Dataflow task, and Execute System Task is on the Control Flow level, so any variable update is not visible to the Control Flow level until the Dataflow Task execution is done. So it will only create a directory for the last CO_ID.
If you need to create a Directory for each customer, and assuming that you are creating ID inside the script Component. Just create the directory from it using the following code (Vb.Net)
If Not Io.Directory.Exists("c:\Customer Orders\" & COID.ToString()) Then
IO.Directory.CreateDirectory(c:\Customer Orders\" & COID.ToString())
End If
Else, if you only need to create columns and to loop over them later, i think that you only need to Cast the CO_ID column to DT_WSTR:
"C:\\Customer Orders\\" + (DT_WSTR, 50) [CO_ID]
Related
I am new to SSIS. I am trying to create a separate excel file dynamically in data flow task for each iteration of the for-each loop? Please guide
You can utilize the following approach.
Create an excel file template on the folder where you want to drop the new files.
Connect your excel file destination to the template file created in the folder.
Create two variables:
variable: IterationCount Data Type Int default value 1.
Variable: FileName Data Type: string
Expression = "Mybasefilename_" + (DT_STR, 4,1252)[User::IterationCount] + ".xlsx"
On your excel file connection hit right click and hit properties go to expression and hit three ellipses and look for filename property.
Set the property value choosing #[User::Filename] variable. If the Name property is not available use the connection string property, however, you should add the folder path as part of your filename variable to create the entire file destination and name.
Last step in your FELC you need to update the IterationCount variable in each iteration.
So, we cannot catch the index of the iteration then you need to use an expression in the FELC, expression task, or a script task to update the IterationCount variable.
Expression task example:
#[User::IterationCount] = #[User::IterationCount] + 1
Helpful Links:
Microsoft - SSIS ForEach Loop Container
SSIS Expression Task
SSIS - Updating variables using Script Task
I am using SSDT and working on a simple SSIS package.
The Control flow:
1. A Foreach Loop Container and seek a folder exist a "importdata{}.csv" file or not.
2. If found, a script task will set variables:
- User::FullPath = (e.g C:\importdata{}.csv)
- User::varFileNameNoExt = (importdata{}) without extension.
The {} is possible in "toy","game","food".
3. Go to dataflow
The Data Flow:
1. Flat File Source with a flat file connection, the connection string is varible and mapped connection string expression.
2.ADO.NET Destination , insert data.
My question is how can i set the ADO.NET Destination [TableOrViewName] Property in variable?
Assume the table : importdatatoy,importdatagame and importdatafood is created on SQL Server.
I try to set as "dbo"."[User::varFileNameNoExt]" ,but it cannot resolve the table name on runtime.
ADO.NET Destination [TableOrViewName] parametrization can be done at Data flow level. In data flow properties, you can specify "ADO.NET Destination [TableOrViewName]".
Also specify the quotes while assigning value to variable
Eg: varFileNameNoExt = "dbo"."tableName"
But first you will need to create mapping with an existent table.
Can you post your error message? I'm thinking you won't be able to combine static text and a variable like that inside of the TableOrViewName field. Instead do the combination in a new [User::varTableName] SSIS variable and use the Advanced Properties Expression editor to set the TableOrViewName to this new SSIS variable. Have a look here.
I am working on my first SSIS package. I have a view with data that looks something like:
Loc Data
1 asd
1 qwe
2 zxc
3 jkl
And I need all of the rows to go to different files based on the Loc value. So all of the data rows where Loc = 1 should end up in the file named Loc1.txt, and the same for each other Loc.
It seems like this can be accomplished with a conditional split to flat file, but that would require a destination for each Location. I have a lot of Locations, and they all will be handled the same way other than being split in to different files.
Is there a built in way to do this without creating a bunch of destination components? Or can I at least use the script component to act as a way?
You should be able to set an expression using a variable. Define your path up to the directory and then set the variable equal to that column.
You'll need an Execute SQL task to return a Single Row result set, and loop that in a container for every row in your original result set.
I don't have access at the moment to post screenshots, but this link should help outline the steps.
So when your package runs the expression will look like:
'C:\Documents\MyPath\location' + #User::LocationColumn + '.txt'
It should end up feeding your directory with files according to location.
Set the User::LocationColumn equal to the Location Column in your result set. Write your result set to group by Location, so all your records write to a single file per Location.
I spent some time try to complete this task using the method #Phoenix suggest, but stumbled upon this video along the way.
I ended up going with the method shown in the video. I was hoping I wouldn't have to separate it in to multiple select statements for each location and an extra one to grab the distinct locations, but I thought the SSIS implementation in the video was much cleaner than the alternative.
Change the connection manager's connection string, in which you have to use variable which should be changed.
By varying the variable, destination file also changes
and connection string is :
'C:\Documents\ABC\Files\' + #User::data + '.txt'
vote this if it helps you
I'm doing an Excel loop through fifty or more Excel files. The loop goes through each Excel file, grabs all the data and inputs it into the database without error. This is the typical process of setting delay validation to true, and making sure that the expression for the Excel Connection is a string variable called EFile that is set to nothing (in the loop).
What is not working: trying to input the name of the Excel file into the database.
What's been tried (edit; SO changed my 2 to 1 - don't know why):
Add a derived column between the Excel file and database input, and add a column using the EFile expression (so under Expression in the Derived Column it would be #[User::EFile]). and add the empty. However, this inputs nothing a blank (nothing).
One suggestion was to add ANOTHER string variable and set its properties EvaluateAsExpression to True and set the Expression to the EFile variable (#[User::EFile]). The funny thing is that this does the same thing - inputs a blank into the database.
Numerous people on blogs claim they can do this, yet I haven't seen one actually address this (I have a blog and I will definitely be showing people how to do this when I get an answer because, so far, these others have fallen short). How do I grab an Excel file's name and input it in a database during a loop?
Added: Forgot to add, no scripts; the claim is that it can be done without them, so I want to see the solution without them.
Note: I already have the ability to import the data from the Excel files - that's easy (see my GitHub account, as I have two different projects for importing all sorts of txt, csv, xls, xlsx data). I am trying to also get the actual name of the file being imported also into the database. So, if there are fifty Excel files, along with the data in each file, the database will have the fifty file names alongside that data (so if each file has 1000 rows of data, each 1000 rows would also have the name of the file they came from next to them as an additional column). This point seems to cause a lot of confusion, as people assume I'm having trouble importing data in files - NOPE, see my GitHub; again that's easy. It's the FILENAME that needs to also be imported.
Test package: https://github.com/tmmtsmith/SSISLoopWithFileName
Solution: #jaimet pointed out that the Derived Column needed to be the #[User::CurrentFile] (see the test package). When I first ran the package, I still got a blank value in my database. But when we originally set up the connection, we do point it to an actual file (I call this "fooling the package"), then change the expression on the connecting later to the #[User::CurrentFile], which is blank. The Derived Column, using the variable #[User::CurrentFile], showed a string of 0. So, I removed the Derived Column, put the full file path and name in the variable, then added the variable to the Derived Column (which made it think the string was 91 characters long), then went back and set the variable to nothing (English teacher would hate the THENs about right now). When I ran the package, it inputted the full file path. Maybe, like the connection, it needs to initially think that a file exists in order for it to input the full amount of characters?
Appreciate all the help.
The issue is because of blank value in the variable #[User::FileNameInput] and this caused the SSIS package to assume that the value of this variable will always be of zero length in the Derived Column transformation.
Change the expression on the Derived column transformation from #[User::FileNameInput] to (DT_STR, 2000, 1252)#[User::FileNameInput].
Type casting the derived column to 2000 sets the column length to that maximum value. The value 1252 represents the code page. I assumed that you are using ANSI code page. I took the value 2000 from your table definition because the FilePath column had variable VARCHAR(2000). If the column data type had been NVARCHAR(2000), then the expression would be (DT_WSTR, 2000)#[User::FileNameInput]
Tim,
You're using the wrong variable in your Derived Column component. You are storing the filename in #[User::CurrentFile] but the variable that you're using in your Derived Column component is #[User::FileNameInput]
Change your Derived Column component to use #[User::CurrentFile] and you'll be good.
Hope that helps.
JT
If you are using a ForEach loop to process the files in a folder then I have have used the technique described in SSIS Junkie's blog to get the filename in to an SSIS variable: SSIS: Enumerating files in a Foreach loop
You can use the variable later in your flow to write it to the database.
TO all intents and purposes your method #1 should work. That's exactly how I would attempt to do it. I am baffled as to why it is not working. Could you perhaps share your package?
Tony, thanks very much for the link. Much appreciated.
Regards
Jamie
I'm learning SSIS and this seems like an easy task but I'm stuck.
I have a CSV file Orders.csv with this data:
ProductId,Quantity,CustomerId
1,1,104
2,1,105
3,2,106
I also have a stored procedure ssis_createorder that takes as input parameters:
#productid int
#quantity int
#customerid int
What I want to do is create an SSIS package that takes the .csv file as input and calls ssis_createorder three times for each row in the .csv file (the first row contains column names).
Here is what I have done so far.
I have created an SSIS package (Visual Studio 2005 & SQL Server 2005).
In Control Flow I have a Data Flow Task.
The Data Flow has a Flat File source of my .csv file. All of of the columns are mapped.
I have created a variable named orders of type Object. I also have variables CustomerId, ProductId, & Quantity of type int32.
Next I have a Recordset Destination that is assigning the contents of the .csv file into the varialbe orders. I'm not sure about how to use this tool. I'm setting the VariableName (under Customer Properties) to User::orders. I think that now orders holds an ADO record set made up of the contents from the original .csv file.
Next I'm adding a ForEach Loop Container on the Control Flow tag and linking it to the Data Flow Task.
Inside of the ForEach Loop Container I'm setting the Enumerator to "ForEach ADO Enumerator". I'm setting "ADO object source variable" to User::orders". For Enumeration mode I'm selecting "Rows in the first table".
In the Variable Mapping tab I have User::ProductId index 0, User::Quantity index 1, User::CustomerId index 2. I'm not sure if this is correct.
Next I have a Script Task inside of the ForEach Loop Container.
I have ReadOnlyVariables set to ProductId.
In the Main method this is what I'm doing:
Dim sProductId As String = Dts.Variables("ProductId").Value.ToString
MsgBox("sProductId")
When I run the package my ForEach Loop Container turns Bright Red and I get the following error messages
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::ProductId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 1 to variable "User::ProductId" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::Quantity" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 2 to variable "User::Quantity" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::CustomerId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 3 to variable "User::CustomerId" cannot be applied.
Warning: 0x80019002 at MasterTest: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (12) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Dts.TaskResult = Dts.Results.Success
Any help would be appreciated
One of my coworkers just give me the answer.
You don't need the the ForEach Loop Container or the RecordSet Container.
All you need is the Flat File Source and an OLE DB Command. Connect to your database and inside the OLE DB Command select the appropriate connection.
In the Component Properties enter the following SQLCommand:
exec ssis_createorder ?, ?, ?
The "?" are place holders for the parameters.
Next under the Column Mappings tab map the .csv file columns to the stored procedure parameters.
You are finished go ahead and run the package.
Thanks Gary if you were on StackOverFlow I would give you an upvote and accept your answer.
If I understand correctly, what you want to do is execute a stored procedure 3 times for each row in the data source.
What if you just create a data flow with a flat file data source and pipe the data through 3 execute sql command tasks? Just map the columns in the data to the input params of your stored procedure.
Maybe I'm not seeing it correctly in your question and I'm thinking too simple, but in my experience you need to avoid using the foreach task in SSIS as much as possible.
I suspect that you need to look at your Data Flow task. It's likely that the values from the source CSV file are being interpreted as string values. You will probably need a Derived Column component or a Data Conversion component to convert your input values to the desired data type.
And, I think #StephaneT's solution would be good for executing the SP.
I'm not sure if this answers your question. But I was looking to do this and I achieved it using the BULK INSERT command. I created a staging table with all of the columns in the csv file, and instead of a stored procedure I used a INSTEAD OF INSERT trigger to handle the logic of inserting it into many tables.