I am a newbie to SSIS and currently struggling with executing SQL task saving the result in a result set and exporting each table to a respective CSV using a data flow task.
There are 15 .sql files stored in a folder which I am creating a variable called FolderPath pointing towards them. Then I create a for each container that reads from a folder and create a variable in the variable mapping which is called SQLfile.
Inside the for-each container I have an execute SQL task which I changed its file connection variable and edited the expression to FolderPath + SQLfile.
Executing this loop works, when the result set value is set to none. Now I am trying to connect the tables created from this loop in a data flow task. I have no idea how to do this but I am guessing it has something to do with the result set. When I change the result set to full result set my loop breaks. I am assuming you cant have a result set inside a loop.
Now I am completely lost as I don't know how to save the result of those 15 tables and how to declare them as source in the data flow task.
Store the query in a variable. Then use the OLE DB Source component in the Data Flow Task and set the 'data access mode' to sql command from variable.
Related
I have an SSIS package which checks for the unprocessed file present in a tracking table and then processed it. Till date only one file would come in and we would process it and as such the process was designed accordingly.
However now multiple files can come in one go and we store those multiple files in the tracking table and we have a column which keeps a track of the unprocessed file.
I am trying to use the For Each loop to process all the unprocessed file. So I get the count of the unprocessed files and would like to simply tun the Point 1 by passing a parameter to the step 1 but I have not been successful in doing it using Foreach From Variable Enumerator. Am I missing something ?
You can do this using the following steps:
Add an Execute SQL Task to get unprocessed files and store the resultset inside a variable of type System.Object
Add a Foreach loop container, change the type to ADO enumerator and select the variable as source
In the variable mapping tab map the result (each file path) to a variable of type string
Inside the foreach loop container add a dataflow task that contains the Flat File source and implement the processing logic you need
Add a flat file connection manager define the columns
Click on the flat file connection manager, press F4 to show the property tab, go to expression.
Select the connectionstring property and use the variable that holds the filepath as expression
Detailed articles
Implementing Foreach Looping Logic in SSIS
Looping Through a Result Set with the ForEach Loop
Using SSIS to Loop Over Result Set and Dynamically Generate Output Files
How to loop through full result set using foreach container in SSIS
I am new to SSIS and would appreciate any help on my issue below -
I have a stored procedure which does not take any input and returns a temp table. This is used for data validation and will be run every day thus, I need to create an SSIS package for the same (the requirement is such). I created an Execute SQL task where I have the result set in a variable of type Object but I want to add a condition of checking rowcount >1 and if yes then write to an excel file.
Thanks,
Hiral
Answer depends on whether you are using SSIS 2012+ or 2008.
Assuming you are with SSIS 2012+:
Approach - declare variable type int for rowcount. Process received Object variable in Foreach loop, more details and screenshots are available in this article. Inside loop - increment variable. Then you can do a conditional execution based on rowcount variable value > 0.
Alternatives - valid for SSIS 2008 as well:
Object with result set is nothing more than a .NET dataset. Create a Script task which counts rows in the first table of the dataset, and stores this in a variable. Then use this variable in package as described above.
Instead of using Execute SQL Script task for SP run, use DataFlow with OLE DB Source and specify the stored procedure as the data source. Then you can count the resulting rows, store value in a variable etc.
Change property Retainsameconnection =true for connection manager
Doing so you can access temp table created in previous step .
You could use a global temp table instead if you want the creation done in a separate step. But this comes with its own issues like being accessible to everyone/every process.
We have an SSIS project that reads from a text file and inserts to a sql server table.
The Flat File Connection Manager for the Flat File Source uses a variable value as ConnectionString property. So essentially, it's expression ConnectionString = #[User::FileName]. This is working fine, and it's reading the file from the variable into the table.
Since the filename needs to be saved into the table, we need to also insert the filename into the table that's already storing the contents of the actual file. Currently, each line in our text file has 5 comma-separated value that we read into table [TableFile], which also has 5 columns.
The change would be that [TableFile] will now have an additional column [FileName]. Therefore, the Data Flow Task that runs and inserts the contents of the file will also insert the filename (already saved in variable #[User::FileName]).
Since the table will always have one filename, I was thinking of somehow using an Execute Sql Server Task item to update the table with this value. But I have no idea of how to include the value of #[User::FileName] in the SQL UPDATE statement.
Thanks.
Create another SSIS variable to build and hold your entire UPDATE sql string, using the FileName variable to build that part of the string.
Then in the Execute SQL task, set SQL Source Type to "variable", and choose your SQL String variable as the Source Variable property.
Here i am new in Developing the SSIS package
I need your support to come up with the solution.
I have 10 different set of stored procedures which I have to export into text file, all 10 procedures will return the same set of columns (only calling parameters are different).
I am not getting the solution how to do ?
Could you please help me to understand how to export the data from a stored procedure output to tab delimited text file?
Please let me know how to build the ssis package ?
Thanks
This is very hard to do without putting pictures in on each step. I do not seem to be able to put pictures in so I will try to describe in is much detail as I can.
You have to first set up a connection to the database where you are going to run the stored procedures from. This means creating a connection manager for "New OLE DB Connection". You will need vaild login to the database information to create this connection.
In the control area I would set up a "Execute SQL Task". I would set the result set to full result set and set the connection to the one you named in the prior step. To call a stored procedure from a SQL task use something like "exec ? = dbo.usp_check_load_table_all #JobCode = ?, #TransId = ? , #Status = ?, #TurnStatusOff = ?" The first ? is the return code from the stored procedure. The others are the parameters to run the stored procedure. Now you are running 10 different stored procedures and I only know how to run one but you could create ten packages, one to run each and concatenate the files when it is done. In the parameter mapping you set the values for the variables to run with. Make sure to create a USER::ReturnValue type long for the return code. The results set needs one entry a USER::Results of type object.
You now put in a foreach loop for a ADO enumerator putting in the USER::Results in as the variable. This allows you to read in each row one at a time. You must create user variables for the variable mapping to go into.
I would then do a data flow task and put a derived column task and set up each of the fields you want to write to the file from the USER:: fieids you created for the foreach loop.
I would create a flat file connection in the connection manager as a delimited file, tab delimited. You will need a file that looks like the output you desire as you will need to map each field in the file.
Add a flat file destination to under the deriried column task and map it to the flat file you just created. Now map each field to the output.
I hope this is helpful as I was once new SSIS myself.
How do I use SSIS to iterate the image files in a directory and using the filename run a query to insert the image into sql server?
I realise that with a Foreach File Enumerator I can loop the files and get the filename into a variable. How do I use this variable to run a query to find the record for that filename from hd in my table and then import the image into my sql server image type column?
Once I have the file in my database, I will delete the file from hd.
If I'm understanding the problem correctly, you would like to sweep all the files in some location into SQL Server using SSIS?
Data Flow Task
Your data flow task will be responsible for the actual import of files into the database. Your approach would be the same as outlined in Import varbinary data Pretty picture version at insert XML file in SQL via SSIS
Your source will be a Script Transformation Component operating as a source component. It's job will be to add all the file names into the Data Flow. Change the filter in the second link to *.png (or whatever your filter is) and it should work.
Use the Import Column Component on the generated file names. This will add the file pointer into the data flow so that it can get imported into the database. You will need to ensure your data type is DT_IMAGE. Even if you're using varbinary(max)/varchar(max)/nvarchar(max) it's all going to be DT_IMAGE within the context of the pipeline's metadata.
Route all of that data into your target table and you will have imported your file data.
File cleanup
At this point, you have imported all this data and now you want to remove the files from disk. Assuming you stored the file name in the database along with the image bits, I'd use an Execute SQL Task to retrieve the list of file names. Change the output type from None to Full Result Set and store that into a variable of type Object.
Connect a Foreach Enumerator to the output of the SQL Task and here you'll want to "shred" the results. Google that term and you'll find a variety of blog posts or previous SO questions on how to do this. The end result will be a file name will be pulled from the recordset object and assigned to a local variable.
Inside the Foreach Enumerator, use a File System Task and Delete the file which is referenced in the variable set from the Foreach Enumerator.