I have a table with 3 columns. I need to import this table's data into an Excel sheet. I need to create the Excel sheet dynamically name and timestamp. Is there any reference for this procedure?
If you need to name the Excel file just as a date/time stamp, you can set this up in the SSIS variables.
Create a variable for the path, I've named it ExcelFilePath and set it to C:\Temp\ . Create another called ExcelFileName, and then go into the expression editor, and enter the following:
#[User::ExcelFilePath] + LEFT((DT_WSTR, 30)GETDATE(), 19)
Here's an image of how it should look:
You can then create an Excel Connection Manager, go into the Properties for it, and click the ... button on the Expressions property. A windows will open - under Property select ExcelFilePath, and then click the ... button in the Expression column next to it:
The expression editor will open again, this time enter #[User::ExcelFilePath], like this:
You may need to set up more dynamic parts in the Control Flow or Data Flow if you want to name the file with a dynamic name followed by the timestamp, but you can use the variables in exactly the same way as I've shown here. Likewise, if the path needs to be dynamic, you can set that variable via a script in the Control Flow or Data Flow.
Below link might help you:
Dynamically creating Excel File, ExcelSheets
Related
I need to import salary data from multiple excel files where filename of each file is a date.
I used SSIS and with success followed typical tutorials for importing multiple excel files. The thing is none of them show simple method how to add one extra column (with the name of the file) to the result. There are some tutorials with huge code scripts, that are too complicated for me.
What I did was to add 'Derived column' module between typical 'excel source' and 'OLE BD destination' where I added a new column 'date' with expression #[User::FileName] - a variable that is used for 'foreach loop container' but as a result I received corectly combined data from all files but the extra column contains the same data - the filename of the first imported file.
I wonder if there is any simple to make the variable I used to change with every loop ? So, as a result, I receive combined data plus one extra column containing the corresponding date, which is the name of each file. Many Thanks
If you are using a Foreach File Enumerator, select "Name only" in the Collection pane as in the image below (I'm assuming that "Name only" will give you the date that you are looking for).
This allows you to map this into a variable on each iteration. To do this, navigate to the Variable Mappings pane, and select the variable you want to use in your Data Flow Task, with 0 as the Index.
You can then add this variable as a Derived Column, and it will give you the name of the file you are importing.
I'm building a SSIS package (using VS 2017) to load a bunch of CSV files from a specific folder. It works great using the ForEach File loop. Data Flow task has a Flat File Source and OLE DB Destination. I want to be able to save the filename in the same table, along the data from the CSV file. How can I do this??
thanks a lot for your time and help
One other method if you want to load the entire file path of the flat files is
Right click on Flat File Source in Data Flow tab
Click "Show Advanced Editor"
Click "Component Properties"
Under Custom Properties you will find "FileNameColumnName".
If you give a name (e.g.: FlatFileName) then that will appear as one of the source output columns in mapping that contains the file name value(full path with file name). It can be mapped to any available column in destination or modified using Derived column task to get only file name.
In my development, I have stored mostly the entire path which helped me in better tracking.
One other method for anyone who comes across this question, instead of using ForEach loop you can also use a simpler method
Right click in Connection Manager
Click on "New Connection"
Select "MULTIFLATFILE" connection type and click Add.
In Connection manager editor, insert location and use wildcard * (E.g.: \\ABC\XYZ\file_*.txt) to pick all the flat files in that folder.
This automatically loops through all the flat files in that folder.
Hope this helps!
The ForEach File Enumerator can capture the file name and assign it to a variable, which can then be routed to the connection string variable of a connection manager for dynamic loading. This same methodology can be applied to write the file name to a database table.
In the data flow, add a Derived Column transformation and add a new column called 'FileName' (or whatever) and then set it's value to the variable value that the ForEach File Enumerator is setting for the file name.
I have a query,
I have 3 CTE's produced by a TSQL query, which are used as a list of union-ed demographics, we have a requirement to submit this file in a particular format with a header and footer row in each file saved as csv. Can someone suggest the best method to produce the following format from SSRS, ultimately the header and footer will be generated from a SQL table which will hold the available submission ID's, I am wondering if this format can be automated from SSRS, or if there is a better approach, as you can see the header and footer appear to be tab separated, without the additional columns, then the comma separated data block, followed by another tab separated footer.
Any suggestions would be appreciated - thanks very much
format required below (dummy data):
0001011RNL DBS 473520180116
10,6374635,19540714,,,7253647987,p1,,john,,2,3 MADE UP ST,,local,CUMBRIA,,ca12 0T1,,,,,,,,A82635,,
10,9283746,19650325,,,7536482965,p2,,peter,,2,38 MADE UP ST,,WIGTON,,,ca12 0T3,,,,,,,,A82045,,
9901011RNL DBS 473520180116
UPDATED based on user suggestions: SSIS appears to be the best approach - I have found the following resource which appears to be what is required, I have taken this from https://www.sqlservercentral.com/Forums/Topic815666-148-1.aspx
provided by John Dempsey
Place 2 Data Flow Tasks on the Control Flow tab.
Use the first Data Flow Task to generate your Header and Detail records (rows)
a. Obtain the data from your data source
b. Transform your data as needed to satisfy the results for your Detail Rows
c. Set up your Flat File Destination in the layout for your Detail Rows. (Don't worry about Header yet)
d. Upon opening your Flat File Destination task you will notice a item for the "Header:". If you were to enter something in the box here it would be hard-coded as the header for you within your file. But, instead of it being hard-coded you can make it dynamic. You can create a SSIS variable to store your dynamically built header information, then assign it to the Header property of the Flat File Destination in the 1st Data Flow task.
Set dynamic Header SSIS variable to Header property of Flat File Destination in 1st Data Flow Task.
a. On the Control Flow Tab select the first Data Flow Task then go to the properties window for task
b. Expand [+] Expressions property and click the [...] button
c. In the Property Expressions Editor window click in the left column to bring up a list of properties available within the Data Flow Task for you Header and Detail Rows.
d. Choose the property [Flat File Destination].[Header] (if you haven't renamed it yet) setting the expression to the name of your SSIS variable name you created for your dynamic header.
Your dynamic header should now show up when you run the first data flow task with your detail rows.
Open up the second data flow and create a source for your footer, I used a script component as an input data source, mapping the source row to an SSIS variable that contained my Footer Row.
Create your Flat File Destination for you footer making sure that you set the Overwrite property = False. Your Flat File Destination will be the same file name as your header and detail file name from you first Data Flow Task, but the layout will be different in order to line up with the footer.
Upon running you SSIS, the package will run through executing your first Data Flow Task which writes the data from the Header property, then each detail row. Then, it will run the second Data Flow Task writing the footer row to the end of the same file used in your Header and Detail Data Flow Task.
I have multiple flat files containing 126 columns , but each of them is without column names. How should i add column names to these files using SSIS. These files are needed to be imported using SSIS so that i can perform transformation on these files.
Do you want to create a new file which has column names or just assign field names to the columns for use in the rest of the package?
Whichever way, if the input file does not contain column names then set them up as follows…
Create a dataflow task and in the dataflow task create a flat file source.
Configure the flat file source and create a new Flat File Connection Manager
Browse to the input file you want and un-tick the Column Names In First Row
Select Advanced and change all of the default names (Coulmn 0, Coulmn 1 etc) into the field names (and types) you want.
Click OK
If you need to create a new file that has the column names in it, just create a flat file destination and this time have the Column Names In First Row turned on, wire it up to the input you created and save it to a new file
One way to do this (maybe not the quickest way) is by using the Advanced Editor.
Right click on the Excel Source component and select Show advanced editor.
In the new window, you need to go to the Input and Output Properties. You should have this by now:
Click on a column under Output Columns (F1,F2,...)
In Common properties, edit the Name to what you want.
I added a derived column component as my next step and this is what I see under the aviable columns:
As you can see, F1 (which I edited in step 2) has a new column name now.
Edit: I somehow assumed you needed this for Excel. Anyways, I hope it helps.
I am working with SSIS 2008. I have a select query name sqlquery1 that returns some rows:
aq
dr
tb
This query is not implemented on the SSIS at the moment.
I am calling a stored procedure from an OLE DB Source within a Data Flow Task. I would like to pass the data obtained from the query to the stored procedure parameter.
Example:
I would like to call the stored procedure by passing the first value aq
storedProdecure1 'aq'
then pass the second value dr
storedProdecure1 'dr'
I guess it would be something like a cycle. I need this because the data generated by the OLE DB Source through the stored procedure needs to be sent to another destination and this must be done for each record of the sqlquery1.
I would like to know how to call the query sqlquery1 and pass its output to call another stored procedure.
How do I need to do this in SSIS?
Conceptually, what your solution will look like is an execute your source query to generate your result set. Store that into a variable and then you'll need to do iterate through those results and for each row, you'll want to call your stored procedure with that row's value and send the results into a new Excel file.
I'd envision your package looking something like this
An Execute SQL Task, named "SQL Load Recordset", attached to a Foreach Loop Container, named "FELC Shred Recordset". Nested inside there I have a File System Task, named "FST Copy Template" which is a precedence for a Data Flow Task, named "DFT Generate Output".
Set up
As you're a beginner, I'm going to try and explain in detail. To save yourself some hassle, grab a copy of BIDSHelper. It's a free, open source tool that improves the design experience in BIDS/SSDT.
Variables
Click on the background of your Control Flow. With nothing selected, right-click and select Variables. In the new window that pops up, click the button that creates a New Variable 4 times. The reason for clicking on nothing is that until SQL Server 2012, the default behaviour of variable creation is to create them at the scope of the current object. This has resulted in many lost hairs for new and experienced developers alike. Variable names are case sensitive so be aware of that as well.
Rename Variable to RecordSet. Change the Data type from Int32 to Object
Rename Variable1 to ParameterValue. Change the data type from Int32 to String
Rename Variable2 to TemplateFile. Change the data type from Int32 to String. Set the value to the path of your output Excel File. I used C:\ssisdata\ShredRecordset.xlsx
Rename Variable 4 to OutputFileName. Change the data type from Int32 to String. Here we're going to do something slightly advanced. Click on the variable and hit F4 to bring up the Properties window. Change the value of EvaluateAsExpression to True. In Expression, set it to "C:\\ssisdata\\ShredRecordset." + #[User::ParameterValue] + ".xlsx" (or whatever your file and path are). What this does, is configures a variable to change as the value of ParameterValue changes. This helps ensure we get a unique file name. You're welcome to change naming convention as needed. Note that you need to escape the \ any time you are in an expression.
Connection Managers
I have made the assumption you are using an OLE DB connection manager. Mine is named FOO. If you are using ADO.NET the concepts will be similar but there will be nuances pertaining to parameters and such.
You will also need a second Connection Manager to handle Excel. If SSIS is temperamental about data types, Excel is flat out psychotic-stab-you-in-the-back-with-a-fork-while-you're-sleeping about data types. We're going to wait and let the data flow actually create this Connection Manager to ensure our types are good.
Source Query to Result Set
The SQL Load Recordset is an instance of the Execute SQL Task. Here I have a simple query to mimic your source.
SELECT 'aq' AS parameterValue
UNION ALL SELECT 'dr'
UNION ALL SELECT 'tb'
What's important to note on the General tab is that I have switched my ResultSet from None to Full result set. Doing this makes the Result Set tab go from being greyed out to usable.
You can observe that I have assigned the Variable Name to the variable we created above (User::RecordSet) and I the Result Name is 0. That is important as the default value, NewResultName doesn't work.
FELC Shred Recordset
Grab a Foreach Loop Container and we will use that to "shred" the results that were generated in the preceding step.
Configure the enumerator as a Foreach ADO Enumerator Use User::RecordSet as your ADO object source variable. Select rows in the first table as your Enumeration mode
On the Variable Mappings tab, you will need to select your variable User::ParameterValue and assign it the Index of 0. This will result in the zerotth element in your recordset object being assigned to the variable ParameterValue. It is important that you have data type agreement as SSIS won't do implicit conversions here.
FST Copy Template
This a File System Task. We are going to copy our template Excel File so that we have a well named output file (has the parameter name in it). Configure it as
IsDestinationPathVariable: True
DestinationVarible: User::OutputFileName
OverwriteDestination: True
Operation: Copy File
IsSourcePathVariable: True
SourceVariable: User::TemplateFile
DFT Generate Output
This is a Data Flow Task. I'm assuming you're just dumping results straight to a file so we'll just need an OLE DB Source and an Excel Destination
OLEDB dbo_storedProcedure1
This is where your data is pulled from your source system with the parameter we shredded in the Control Flow. I am going to write my query in here and use the ? to indicate it has a parameter.
Change your Data access mode to "SQL Command" and in the SQL command text that is available, put your query
EXECUTE dbo.storedProcedure1 ?
I click the Parameters... button and fill it out as shown
Parameters: #parameterValue
Variables: User::ParameterValue
Param direction: Input
Connect an Excel Destination to the OLE DB Source. Double click and in the Excel Connection Manager section, click New... Determine if you're needing 2003 or 2007 format (.xls vs .xlsx) and whether you want your file to have header rows. For you File Path, put in the same value you used for your #User::TemplatePath variable and click OK.
We now need to populate the name of the Excel Sheet. Click that New... button and it may bark that there is not sufficient information about mapping data types. Don't worry, that's semi-standard. It will then pop up a table definition something like
CREATE TABLE `Excel Destination` (
`name` NVARCHAR(35),
`number` INT,
`type` NVARCHAR(3),
`low` INT,
`high` INT,
`status` INT
)
The "table" name is going to be the worksheet name, or precisely, the named data set in the worksheet. I made mine Sheet1 and clicked OK. Now that the sheet exists, select it in the drop down. I went with the Sheet1$ as the target sheet name. Not sure if it makes a difference.
Click the Mappings tab and things should auto-map just fine so click OK.
Finally
At this point, if we ran the package it would overwrite the template file every time. The secret is we need to tell that Excel Connection Manager we just made that it needs to not have a hard coded name.
Click once on the Excel Connection Manager in the Connection Managers tab. In the Properties window, find the Expressions section and click the ellipses ... Here we will configure the Property ExcelFilePath and the Expression we will use is
#[User::OutputFileName]
If your icons and such look different, that's to be expected. This was documented using SSIS 2012. Your work flow will be the same in 2005 and 2008/2008R2 just the skin is different.
If you run this package and it doesn't even start and there is an error about the ACE 12 or Jet 4.0 something not available, then you are on a 64bit machine and need to tell BIDS/SSDT that you want to run in 32 bit mode.
Ensure the Run64BitRuntime value is False. This project setting can be found by right clicking on the project, expand the Configuration Properties and it will be an option under Debugging.
Further reading
A different example of shredding a recordset object can be found on How to automate the execution of a stored procedure with an SSIS package?