I am using foreach loop to dynamically load file in database but in my source file year and month wise data is stored.If 2020 will come then issue will generate.I want to know that if i want to do unpivot of all those columns suppose 2020 and 2021 data came then it should store sales values in different column and month and year in different column.
By foreach loop i can read all file but if i want to unpivot it dynamically and want to store it in database.
if anyone have any idea how to resolve it or which process should follow then please let me know...
I am using Foreach Loop in ssis to loop through all files inside the folder.But when i want to unpivot files at that time it is giving error because yearwise column will change.
And in my folder am having different yearwise files.How can i unpivot it dynamically??
This is My source files i want to unpivot all this files dynamically in ssis
Related
I have an Excel file, sheet name is sheet1. In this sheet1, I have a data where the first 5 rows have some information like date value and some other information.
I need to load the data from A5:Ad range at the same time I need fetch cell A3 value it has date value in it, I need to create one column in my table as date column and need to load that cell A3 value.
In one sheet I need to do 2 functionalities like I need to take range of values and cell value from A3 cell. Please give me some advice of yours. It would be very grateful. I googled but I didn't find anything for this. Till now I tried using openrowset option but it works for range of data if I want to fetch A3 cell value data I am failing.
Please provide your inputs.
With SSIS you can use the Excel process inside a dataflow task - but Excel must be installed on the system where you run the ssis package.
One option is to load the hole excel file into a temporary table (with an auto-id) and query this table by using SQL statements. Or, you can add a row number via script component to the ssis data flow and split the rows you need and save them into your destination table(s).
I have an excel file that I want to import using SSIS, I have done this process many more times without a problem with other excel files. however, this excel file has three spreadsheets on it. two of them works just fine importing correct data, BUT one of them is importing fixed data with 06-30-2019 for all date fields, and this not changing when it gets to table but it changed while it still at the source when I preview it, I see a fixed date already set up for date fields but other fields that are non-date fields are coming good. the dates in the spread field of that sheet have different dates per row for that field/column. how is this happening? I am using SSIS 2017 and excel plus loads to table SQL 2017. how can I fix this?
in my excel sheet1
col1 datefield
1 01-08-2019
2 05-06-2019
3 06-12-2019
4 07-25-2019
in my excel source SSIS on preview show this below and that what it loads as well to table.
col1 datefield
1 06-30-2019
2 06-30-2019
3 06-30-2019
4 06-30-2019
well, after I spent a great deal time, I have come to realize my excel file was in a shared file, which we can only access it VIA link/path given to me. instead, I was passing to my excel source the path from my local to that file(where I saved the file earlier), instead, I should have given the shared folder path, and happy to tell you all, the problem was solved.
However the Question Remains, why would Excel Plus Source do something like this and why choose a particular date 06/30/2019? I had that date on my file but not the only one and that date wasn't the first row in my data.
I have an Excel sheet that changes column names based on the year and current week of the year, so for example 201901 would be the first week of 2019.
The Excel sheet that is sent to us daily automatically adjusts the column names based on the current date (up to 6 months), so currently (31/07/2019) the year and week show 201931 - 202011:
So the N column next week will be 201932 (the columns shift left basically).
I have tried changing the Excel source columns to a different alias of just 1,2,3,4 etc in hopes to just get the data into SQL Server, and then script a trigger in SQL Server to change column names but doesn't work due to the mapping SSIS requires.
Works fine until the column changes to next week.
A simple method would be to drop the table and just dump the file in a new table named the same but can't see how to set up in SSIS as you need to map the column names (which unfortunately change).
Here is how the dataflow looks:
Ideally, for me, something like this would be perfect:
But not sure how to achieve this outcome in SSIS?
I would suggest transforming the data. Currently, you have a "cross-table"-format.
How about putting the Excel data in the form of (RAG_week; CalenderWeek; Value_of_CalenderWeek) ? For doing this you can use an Excel-Macro which fills a new Sheet in the Excel file. (Each cell is transformed in one dataset, being a row on its own.) Next, you create a similar table on the SQL Server. Then you can create a SSIS package with constant column assignment, simply appending the new data each week.
This impacts the further evaluation of your data, but seems to be a far more stable approach.
I want to load files in SQL server database on weekly basis. Each file name contains date on it. Currently, I am using Foreach Loop Container to get the file name and stored it in table. Table contains 3 columns FileName, Date and Week. After loading FileName using Execute SQL Task I extract Date and Week from the FileName and Populate Date and Week column. Then I use Execute SQL Task to SELECT all table date ORDER BY Date and Week and store it into object variable. Finally, I use Foreach Loop Container to load actual files in date order using ADO Enumerator and object variable. This works fine. However, I want to load files on Weekly basis. For an example all the files which has week 15 in the table should loaded first. Then it should load load all the files of week 16 and so on. The reason I want to load like this is after loading one week of files I want to process it using some stored procedure.
I think the problem can be solved by making two edits:
Loop over weeks
Add an Execute SQL Task that retrieve the Distinct Weeks from the table
Add a foreach loop container to loop over weeks
inside the foreach loop add an Execute SQL Task that retrieve the rows based on the current week
Use another foreach loop container to loop over result
Ordered results
You can simply add an ORDER BY clause inside the Execute SQL Task to get an ordered resultset.
This is a limitation of the ForEach loop enumerators - there is no way to load files in a sorted/ordered manner. If you want to load files in such a manner then there are two ways to do this:
Purchase an expensive package of components from third party vendors that provide a ForEach loop enumerator that can process files in a sorted/ordered manner
Do it yourself manually.
For option two, you will need to perform the following steps:
Create a ForEach File loop enumerator scan the folder for all files and insert the file names into a database table.
Create an Execute SQL Task that will SELECT all file names, ORDERED BY file name. You can add constraints in the WHERE clause to control the date range of files that you want to process.
Load the result set into a variable of type Object
Create a ForEach ADO loop enumerator to loop through each file name that is stored in the object.
Place a data flow in the loop and then process the files.
I have an SSIS package I am building to take data from a .CSV file and load it into a table in a SQL Server database. The .CSV file has more columns than my table and I'm looking to filter out the data based on some of these columns that are not being inserted into the table.
I have year, kind, type, dollars as my column names in the .CSV file but I'm only pulling type and dollars into the DB. However, I can only pull those rows where the kind= "L" and year is the current year (with one major caveat). If the process is running in the first quarter of a given year (so month <= 3) it needs to use the previous year as my qualifier for what rows it pulls in from the .CSV file. For instance, say it is February 2015 when this package is running, I need it to pull only rows with a year of 2014 and kind="L" from my .CSV file. If it is September 2015 then it needs to pull in rows with a year of 2015 and kind="L".
Any idea what the best way of doing this is? Right now I have a conditional split in my package but I can only get it to say year==YEAR(GETDATE()) and this will not work for the first quarter. I'd need some sort of variable logic to say something like IF(currentmonth<=3 THAN #year = currentyear-1) ELSE (#year = currentyear) and then use the #year variable in the conditional split. Is this possible?
Any help is much appreciated!
Normally for this kind of workflow, I will import the entire CSV into a temporary table, and then have a separate SQL script or view which reads from the temporary table and applies whatever business logic is needed for the final view.
If you want the logic to be in the SSIS package you can use a derived column component to declare a new boolean field for example IncludeRowInOutput and set it to be something like
((currentmonth <= 3 and year = year(getdate() - 1)) or (year = year(getdate))) and kind = 'L'
Then you can do the conditional split based on the IncludeRowInOutput field.
I'd normally be wary of using too much script components, I find that they are harder to debug and make the dataflow harder to understand.