I have a SQL table that stores filename and ssis package name. Whenever the file gets dropped to a directory, the corresponding ssis package gets triggered referring the mapping table.
If I store the file name as say, a*.csv in database and the corresponding ssis package as sample-ssis.dtsx, Will I be able to trigger the same package for any csv file starting with "a"? Can someone please help me with this.
Sure, you can read the file name into a variable and use a script task to loop through your mapping table and see if any of the filename-with-wildcard entries in the mapping table match the file name in the variable.
Related
I'm working on an SSIS package that will be used to import data from an Excel file into SQL Server. My current struggle is figuring out how to make the SSIS package bring in exactly one excel file without knowing the name of it beforehand. I have a directory that will contain between 0 and n excel files at the same time. I want to pull in only the file with the oldest creation time. Is this possible?
I'm using Visual Studio 2015 to build the SSIS package. My DB is in SQL Server 2016.
To create a dynamic file connection:
Create a new Variable (Name Example: 'SourceFile') of datatype String.
In a 'For Each Loop Container' map that variable under the 'Variable Mapping' Tab and set the 'Enumerator Configuration' to the correct folder and file extension.
The 'For Each Loop Container' will read the file from the location and assign the name of the file to the variable.
In the Expressions Properties of your file connection set the ConnectionString property to #[User::SourceFile]
This should make your file source dynamic. It will pick up the file no matter what it is named, but the format of the file will have to be consistent.
Using just SSIS tasks, I am not aware of how to utilize the create date of the files to pick the oldest file, but if the file name contains the create date of the file you could substring the date out of #[User::SourceFile] variable and store it in another variable with each execution of the 'For Each Loop Container' to determine which file is oldest.
I keep running into issues creating a SSIS project that does the following:
inspects folder for .csv files -> for each csv file -> insert into [db].[each .csv files' name]
each csv and corresponding table in the database have their own unique columns
i've tried the foreach loop found in many write ups but the issue comes down to the flat file connection. it seems to expect each csv file has the same columns as the file before it and errors out when not presented with this column names.
anyone aware of a work around for this?
Every flat file format would have to have it's own connection because the connection is what tells SSIS how to interpret the data set contained within the file. If it didn't exist it would be the same as telling SQL server you want data out of a database but not specifying a table or its columns.
I guess the thing you have to consider is how are you going to tell a data flow task what column in a source component is going to map to a destination component? Will it always be the same column name? Without a Connection Manager there is no way to map the columns unless you do it dynamically.
There are still a few ways you can do what you want and you just need to search around because I know there are answers on this subject.
You could create a Script Task and do the import in .Net
You could create a SQL Script Task and use BULK INSERT or OPENROWSET into a temporary stagging table and then use dynamic sql to map and import the final table.
Try to keep a mapping table with below columns
FileLocation
FileName
TableName
Add all the details in the table.
Create user variables for all the columns names & one for result set.
Read the data from table using Execute SQL task & keep it in single result set variable.
In For each loop container variable mappings map all the columns to user variables.
Create two Connection Managers one for Excel & other for csv file.
Pass CSV file connection string as #[User::FileLocation]+#[User::FileName]
Inside for each loop conatiner use bulk insert & assign the source & destination connections as well as table name as User::TableName parameter.
if you need any details please post i will try to help you if it is useful.
You could look into BiML Script, which dynamically creates and executes a package, based on available meta data.
I got 2 options for you here.
1) Scrip component, to dynamically create table structures in sql server.
2) With for each loop container, use EXECUTE SQL TASK with OPENROWSET clause.
We have an SSIS project that reads from a text file and inserts to a sql server table.
The Flat File Connection Manager for the Flat File Source uses a variable value as ConnectionString property. So essentially, it's expression ConnectionString = #[User::FileName]. This is working fine, and it's reading the file from the variable into the table.
Since the filename needs to be saved into the table, we need to also insert the filename into the table that's already storing the contents of the actual file. Currently, each line in our text file has 5 comma-separated value that we read into table [TableFile], which also has 5 columns.
The change would be that [TableFile] will now have an additional column [FileName]. Therefore, the Data Flow Task that runs and inserts the contents of the file will also insert the filename (already saved in variable #[User::FileName]).
Since the table will always have one filename, I was thinking of somehow using an Execute Sql Server Task item to update the table with this value. But I have no idea of how to include the value of #[User::FileName] in the SQL UPDATE statement.
Thanks.
Create another SSIS variable to build and hold your entire UPDATE sql string, using the FileName variable to build that part of the string.
Then in the Execute SQL task, set SQL Source Type to "variable", and choose your SQL String variable as the Source Variable property.
I'm little new to SSIS and I have a need to import some flat files into SQL tables in the same structure.
(Assume the table is already exist in the same structure and table name and flat file name is same)
I thought to create a generic package (sql 2014) to import all those file by looping through a folder.
I try to create a data flow task in a foreach loop container in the data flow task I dropped a flat file source and ADO.Net destination .
I have set the file source to a variable so that every time it loops through it get the new file. similarly for the ADO.net table name I set it to a the variable so that each time it select a different table according to the file name.
since both source column names and destination column names are same I assume it will map the columns automatically.
but with a simple map it didn't let me to run the package so added a column on the source and selected a table and mapped it.
when I run the package I assumed it will automatically re map everything.
but for the first file it ran but second file it failed complaining with map issues.
can some one let me know whether this is achievable by doing some dynamic mapping?? or using any other way.
any help would be much appreciated.
thanks
Ned
How do I use SSIS to iterate the image files in a directory and using the filename run a query to insert the image into sql server?
I realise that with a Foreach File Enumerator I can loop the files and get the filename into a variable. How do I use this variable to run a query to find the record for that filename from hd in my table and then import the image into my sql server image type column?
Once I have the file in my database, I will delete the file from hd.
If I'm understanding the problem correctly, you would like to sweep all the files in some location into SQL Server using SSIS?
Data Flow Task
Your data flow task will be responsible for the actual import of files into the database. Your approach would be the same as outlined in Import varbinary data Pretty picture version at insert XML file in SQL via SSIS
Your source will be a Script Transformation Component operating as a source component. It's job will be to add all the file names into the Data Flow. Change the filter in the second link to *.png (or whatever your filter is) and it should work.
Use the Import Column Component on the generated file names. This will add the file pointer into the data flow so that it can get imported into the database. You will need to ensure your data type is DT_IMAGE. Even if you're using varbinary(max)/varchar(max)/nvarchar(max) it's all going to be DT_IMAGE within the context of the pipeline's metadata.
Route all of that data into your target table and you will have imported your file data.
File cleanup
At this point, you have imported all this data and now you want to remove the files from disk. Assuming you stored the file name in the database along with the image bits, I'd use an Execute SQL Task to retrieve the list of file names. Change the output type from None to Full Result Set and store that into a variable of type Object.
Connect a Foreach Enumerator to the output of the SQL Task and here you'll want to "shred" the results. Google that term and you'll find a variety of blog posts or previous SO questions on how to do this. The end result will be a file name will be pulled from the recordset object and assigned to a local variable.
Inside the Foreach Enumerator, use a File System Task and Delete the file which is referenced in the variable set from the Foreach Enumerator.