I have a connection manager (oledb) that points to a folder that has 20 dbf files. All the dbf files have the same schema. Just the data are for specific entities. I want to take all the data from the 20 dbf files and insert them into one table (in sql server). What tasks enables me to do this?
You should create "For each loop" container, to write each file path to variable f.e. #fp
After that, you put inside DataFlow task and configure the connection
After, you should create another variable like #table (substring your #fp,to only file name) and put this variable in DataFlow task in source table.
ready example
Related
I am trying to create some sort of automation whereby I can generate a series of pipe-delimited text extracts for about 100 different tables each month. Each extract would be based on a simple query like this:
SELECT *
FROM tablename
WHERE AsOfDate = 'currentmonth'
where both tablename and currentmonth would be variables. The tablename variable name would change for each of the tables but currentmonth would remain the same throughout the execution.
I have been attempting to build an SSIS package that uses a ForEach Loop container that runs through a list of all the table names and passes that variable into a SQL string, which is then used by the OLE DB Data source in the data flow.
However, all of these tables have different columns. Based on what I can tell, it would not be feasible to do a simple OLE DB Source to a Flat File Destination within that loop container since the Flat File Connection Manager must be configured to account for the different columns of each table.
Would there be any feasible way to do this outside of configuring the process manually for each of the 100+ tables?
You could look into BiML which programmatically creates your dataflows based on metadata.
Or you could use a Script task that loops through the tables, loops through their columns, and generates text files instead of using any dataflow at all.
I keep running into issues creating a SSIS project that does the following:
inspects folder for .csv files -> for each csv file -> insert into [db].[each .csv files' name]
each csv and corresponding table in the database have their own unique columns
i've tried the foreach loop found in many write ups but the issue comes down to the flat file connection. it seems to expect each csv file has the same columns as the file before it and errors out when not presented with this column names.
anyone aware of a work around for this?
Every flat file format would have to have it's own connection because the connection is what tells SSIS how to interpret the data set contained within the file. If it didn't exist it would be the same as telling SQL server you want data out of a database but not specifying a table or its columns.
I guess the thing you have to consider is how are you going to tell a data flow task what column in a source component is going to map to a destination component? Will it always be the same column name? Without a Connection Manager there is no way to map the columns unless you do it dynamically.
There are still a few ways you can do what you want and you just need to search around because I know there are answers on this subject.
You could create a Script Task and do the import in .Net
You could create a SQL Script Task and use BULK INSERT or OPENROWSET into a temporary stagging table and then use dynamic sql to map and import the final table.
Try to keep a mapping table with below columns
FileLocation
FileName
TableName
Add all the details in the table.
Create user variables for all the columns names & one for result set.
Read the data from table using Execute SQL task & keep it in single result set variable.
In For each loop container variable mappings map all the columns to user variables.
Create two Connection Managers one for Excel & other for csv file.
Pass CSV file connection string as #[User::FileLocation]+#[User::FileName]
Inside for each loop conatiner use bulk insert & assign the source & destination connections as well as table name as User::TableName parameter.
if you need any details please post i will try to help you if it is useful.
You could look into BiML Script, which dynamically creates and executes a package, based on available meta data.
I got 2 options for you here.
1) Scrip component, to dynamically create table structures in sql server.
2) With for each loop container, use EXECUTE SQL TASK with OPENROWSET clause.
I have been searching for about a week now and I was wondering if anyone may have a clue. I wrote a package to do the following:
loop through a parent folder and its subfolders for a csv with a particular naming structure (works)
Create a table for each .csv based on the enumeration of each file (works).
Import the data into sql server in their own tables with the file name that was created as the table name and not OLE DB Destination (which does not work). It works if it there is destination folder for everything, but when I use table variable that does not work.
What I did was add an Execute SQL task to the for each container to create a table with a variable for the file path that is mapped as an expression in the for each container in a create table query under property sqlstatementsource expression. The tables are created, but when I use the variable that was mapped for the for each loop as the table name or variable in OLE DB Destination I get an error asking for me to check if the table exists. The tables are created, but I cannot get the insertion of the data into their own tables. Even when I bypass the error of "Destination table has not been provided" and run the package. I set delayValidation as true and still nothing. SSIS from what I have seen so far does some cool things. However, I am stuck right now. What else am I doing wrong?
I forgot to mention that the data is going to sql server.
Thanks for everything.
You can't create an OLEDB Destination at design time with a variable for a table name. The OLEDB destination needs to know the table name, and the columns, so that it can pre-map the data flow to the table columns.
You have a couple of other options:
You can use BiML to dynamically create your dataflows and destinations.
You can use an ExecuteSQL Transformation as your dataflow destination, and write a dynamic SQL statement that inserts each row in the dataflow to the desired table.
i am trying to create a ForEachLoop container that extracts excel files within a source folder.
i have created an execute sql task within a ForEachLoop container that stores my excel files full paths in an sql server table
and now i can't figure how to make it go through that list and extract each file into an ole db destination table
ps: the excel files have different types of data, columns change almost from one file to another (28 files)
can you please help me ? thank you in advance.
It won't work within a for each loop because your destination for each spreadsheet has to be a table that matches the columns coming in. If it was 25 different spreadsheets with the same columns types and number of columns you could insert all the rows into one table but it sounds like you need to create separate data flows for each one. You can then combine the datasource--> transform--> Ole Destination onto one data flow (which could run in parallel) and you would have (for 26 imports) three steps for each spreadsheet.
As part of migration from traditional system to new technology, I need to rename N number of files[.txt, .pdf, .xl, etc] available in the particular folder using SSIS.
Move the file to destination
Parse the prefix of files which is used as ID for associating with the record in the table.
Ex: 1012BA12_Attach_Emp.doc [ID=1012BA12]
Then I need to go to database and lookup the new ID.
Ex: old ID=1012BA12 and equivalent new ID=512
Then replace the old ID with new one.
Ex: 512_Attach_Emp.doc
Insert one row to some table with respect new name & path.
I have been used the for each file enumerator, Execute sqltask and file system task
but it's consuming a day to do so.
Please guide me best approach.
The issue you are having is likely to be on the database side, not SSIS.
Do you have indexes on the tables you are accessing?
Are the files local to the SSIS instance, or does SSIS access the files remotely?