I have two Excel files named 'First' and 'Second' in same location .
They have same schema.
I used foreach loop counter and put Data flow task into it.
The data flow diagram looks like this:-
Here, I selected first excel file as the source....
My For Each Loop Container Editor:-
After running the SSIS package successfully the output came like this:-
Which took data only from First excel file and three times,I must have done something wrong in there,But I cant figure it out.
Check your Foreach Loop Editor:
Collection>Folder
Collection>Files
Your file should not have a particular file name, for multiple excel use *.xlsx.
Edit:
Use a Script task to Debug. Map the value of ForEach to a variable and display it through Script task.
Edit the script task with below code.
MessageBox.Show(Dts.Variables["Variable"].Value.ToString());
Also, Please check your Source Excel connetion is configured correctly with values coming from foreach.
Related
I am new to SSIS. I am trying to create a separate excel file dynamically in data flow task for each iteration of the for-each loop? Please guide
You can utilize the following approach.
Create an excel file template on the folder where you want to drop the new files.
Connect your excel file destination to the template file created in the folder.
Create two variables:
variable: IterationCount Data Type Int default value 1.
Variable: FileName Data Type: string
Expression = "Mybasefilename_" + (DT_STR, 4,1252)[User::IterationCount] + ".xlsx"
On your excel file connection hit right click and hit properties go to expression and hit three ellipses and look for filename property.
Set the property value choosing #[User::Filename] variable. If the Name property is not available use the connection string property, however, you should add the folder path as part of your filename variable to create the entire file destination and name.
Last step in your FELC you need to update the IterationCount variable in each iteration.
So, we cannot catch the index of the iteration then you need to use an expression in the FELC, expression task, or a script task to update the IterationCount variable.
Expression task example:
#[User::IterationCount] = #[User::IterationCount] + 1
Helpful Links:
Microsoft - SSIS ForEach Loop Container
SSIS Expression Task
SSIS - Updating variables using Script Task
I need to execute a package once for every item in flat file. The package's parameters will be a few of the current item's attributes. What is the best way to proceed? I'm assuming the last thing I do is place an Execute Package Task component inside of a Foreach Loop Container like below:
Simple Solution
First of all you have to create a Package variable of type Object
Add a script task that loop over flat file and convert it to a DataTable or List(of T) and store it into the Object variable.
Connect the Script Task to the foreach loop you are using
In the foreach loop container select the enumeration type to Ado enumerator and select the Object variable as Source and map the columns you want to use in the execute package task to variables
Reading Flat File into DataTable
you can refer to one of the following links to learn more:
How to Read text file to DataTable
DataTable from TextFile?
Efficient function for reading a delimited file into DataTable
Side Note: If you need more details, give me a reply
Each item had three attributes: username, password, and exactAccountName
These were then used to execute an SSIS package with each login credential.
In order to achieve this, do the following:
Create a package variable of type Object in the master package.
Add a script task that loop over flat file and convert it to a DataTable or List(of T) and store it into the Object variable. (This step is from Hadi's answer)
Create a foreach loop to execute next.
In the foreach loop container select the enumeration type to Ado enumerator and select the Object variable as Source. (Also from Hadi's answer)
Put a Script Task and Execute Package Task in the foreach loop. Like so:
Create three string variables in the master package for username, password, and exactAccountName.
Populate the main function of the Script Task with code like this:
Set the parameters of the child package to be the variables specified by the Script Task and you're done!
for some reason, I can't seem to figure out why I'm missing some lines when I import a csv file that I exported using sqlplus. The ssis package gives no warnings or errors as well.
The reason I know this is because when I loaded the csv file into a separate analysis application it gets the correct totals for numeric columns and row counts.
But for some reason, SSIS doesn't seem to capture all of the lines...
there were 6313052 rows in the csv file and ssis imports 6308607...
any thoughts?
I've tried different code pages too (1250, 1252, and UTF8) but they didn't seem to have an affect
I checked out this link:
Why all the records are not being copied from CSV to SQL table in a SSIS package
and numbers I've checked on numbers 2, 3, and 4.
Although, for number 1, I'm using a for each loop container descirbed in this site:
http://help.pragmaticworks.com/dtsxchange/scr/FAQ%20-%20How%20to%20loop%20through%20files%20in%20a%20specified%20folder,%20load%20one%20by%20one%20and%20move%20to%20archive%20folder%20using%20SSIS.htm
to loop through files in a folder and import them.
I've also thought about missing delimiters, but I exported the files myself using sqlplus like this:
select
trim(to_char(t1.D_DTM, 'yyyy-mm-dd hh24:mm:ss'))||','||
trim(t1.TECHNOLOGY)||','||
trim(t1.VOICEDATA)||','||
trim(t2.MRKT_NM)||','||
trim(t2.REGION_NM)||','||
trim(t2.CLUSTER_NM)||','||
trim(t3.BSC_NM)||','||
trim(t1.BTS_ID)||','||
trim(t1.CSCD_NM)||','||
trim(t1.SECT_SEQ_ID)||','||
trim(t1.BND_ID)||','||
trim(t1.FA_ID)||','||
trim(t1.TCE_DTCT_CALL_SETUP_F_CNT)||','||
trim(t1.MS_ACQ_CALL_SETUP_F_CNT)||','||
trim(t1.SIGN_CALL_SETUP_F_CNT)||','||
trim(t1.BAD_FRM_CALL_SETUP_F_CNT)||','||
trim(t1.REORG_ATT_CNT)||','||
trim(t1.TCE_CALL_SETUP_F_CNT)||','||
trim(t1.WCD_CALL_SETUP_F_CNT)||','||
trim(t1.L_CALL_SETUP_F_CNT)||','||
trim(t1.TRAF_FRM_MISS_CNT)||','||
trim(t1.BCP_TMO_CALL_SETUP_F_CNT)
from table
I'm not sure how I could miss delimiters if I export a file like that...
Also, I tested importing these files into MySql using LOAD DATA INFILE and that seems to work fine and import all of the data...
Introduction : I have Multiple Excel files which loop through a Foreach Loop Container in SSIS Package.
The first Excel file Excel1.xlsx contains the old data (for example :I have a column named EffectiveDate populated with 2001-01-01 to 2013-04-01of
The second Excel file Excel2.xlsx contains the new entries with EffectiveDate from 2013-05-01 and also contains some old data from Excel1.xlsx.
These two files loop through Foreach Loop Container.
Problem : Once the first Excel file Excel1.xlsx is loaded , i want to compare it with second Excel file Excel2.xlsx and update the EffectiveDate of old data in Excel2.xlsx with EffectiveDate of matching rows in Excel1.xlsx
And all other rows( or new Entires) of Excel2.xlsx with GetDate().
Is it possible to get it done in a single Data Flow Task?
And also how do i compare two excel files in a single container?
You can have 2 Excel sources within one data flow task. You could use a merge join to compare the values, and feed that to an excel output.
If you want to loop through 10 excel files, comparing 1 to another, I would suggest that your merge join output be the 2nd excel source, and map your container variable to the first excel source. That way, Everything from Excel file 1 will be put into the output file, then for each subsequent file only the entries not listed already in the output file will be added.
If you get hung up on any of the steps individually I'm sure myself or others can help you push through the sticking points.
I'm tring to create an SSIS package to import some dataset files, however given that I seem to be hitting a brick
wall everytime I achieve a small part of the task I need to take a step back and perform a sanity check on what I'm
trying to achieve, and if you good people can advise whether SSIS is the way to go about this then I would
appreciate it.
These are my questions from this morning :-
debugging SSIS packages - debug.writeline
Changing an SSIS dts variables
What I'm trying to do is have a For..Each container enumerate over the files in a share on the SQL Server. For each
file it finds a script task runs to check various attributes of the filename, such as looking for a three letter
code, a date in CCYYMM, the name of the data contained therein, and optionally some comments. For example:-
ABC_201007_SalesData_[optional comment goes here].csv
I'm looking to parse the name using a regular expression and put the values of 'ABC', '201007', and
'SalesData' in variables.
I then want to move the file to an error folder if it doesn't meet certain criteria :-
Three character code
Six character date
Dataset name (e.g. SalesData, in this example)
CSV extension
I then want to lookup the Character code, the date (or part thereof), and the Dataset name against a lookup table
to mark off a 'checklist' of received files from each client.
Then, based on the entry in the checklist, I want to kick off another SSIS package.
So, for example I may have a table called 'Checklist' with these columns :-
Client code Dataset SSIS_Package
ABC SalesData NorthSalesData.dtsx
DEF SalesData SouthSalesData.dtsx
If anyone has a better way of achieving this I am interested in hearing about it.
Thanks in advance
That's an interesting scenario, and should be relatively easy to handle.
First, your choice of the Foreach Loop is a good one. You'll be using the Foreach File Enumerator. You can restrict the files you iterate over to be just CSVs so that you don't have to "filter" for those later.
The Foreach File Enumerator puts the filename (full path or just file name) into a variable - let's call that "FileName". There's (at least) two ways you can parse that - expressions or a Script Task. Depends which one you're more comfortable with. Either way, you'll need to create three variables to hold the "parts" of the filename - I'll call them "FileCode", "FileDate", and "FileDataset".
To do this with expressions, you need to set the EvaluateAsExpression property on FileCode, FileDate, and FileDataset to true. Then in the expressions, you need to use FINDSTRING and SUBSTRING to carve up FileName as you see fit. Expressions don't have Regex capability.
To do this in a Script Task, pass the FileName variable in as a ReadOnly variable, and the other three as ReadWrite. You can use the Regex capabilities of .Net, or just manually use IndexOf and Substring to get what you need.
Unfortunately, you have just missed the SQLLunch livemeeting on the ForEach loop: http://www.bidn.com/blogs/BradSchacht/ssis/812/sql-lunch-tomorrow
They are recording the session, however.