I have a table like:
ID, FileName, NumRec, Date
I have a requirement to export data to flat file using SSIS in such a way that I have to create a file each ID. But two files have to be created in separate folders. Eg. records related to I’d 1 & Id 2 should be created in separate files but in the same folder while those rated to Id 3 & id 4 have to be in two separate files but into same filed.
I was unable to loop a result set as a pair
Related
I am trying to automate a process where a team member is given a csv list with columns dealing with First names, last names, email addresses, addresses, phone numbers, and several other columns that are unique to each customer from a customer and then they have to upload the csv file into our system. For example, some customers could send a file with only first name, last name, and email addresses, while the next customer could send a file with those three along with certain ID columns that are directly valuable to them and them only.
The issue is each customer has different names for their columns than we have in our database, so the user has to manually map their columns to our columns in the database through a dropdown menu.
As you can see below, the name on their file upload could be "first_name" and it would match directly with our db column of "first_name". However some customers could have "f_name" or "first name" and we would have to map it directly in the drop down to the "first_name" column in our database.
Unfortunately these docs comes from the customer and we cannot automate it to have one csv document that everyone uses with the same exact columns. Is there a way to not have to map those manually and have that entered into the database automatically into the correct columns?
file upload
I'm using PostgreSQL 14 and pgAdmin4.
I want to create a consolidate table into the database by loading multiple csv files along with dynamic headers. We are going to load multiple csv files with different headers.
We need to create a consolidated table with all the columns without duplicates and dynamic columns, all the dynamic columns should be there in the consolidated table with the data.
Can you please assist this challenges and if there are any sample please share it with us. It'll be helpful for us.
Lets say if there is a CSV/TSV file with first line as headers
File 1 : 10 headers/column name
and we load that data into SQL table1
Next, File 2: 15 headers/column name
and we load that data into SQL table2
File1 + File2 = 16 headers (unique columns from both the tables and no duplicates)
Can we create consolidated table with 16 headers say table3 in an automated way?
Further,
Next, if File3 comes with 15 headers
Consolidated table + File3 = 18 unique headers , then we should be able to alter consolidated table and
insert 18 headers data into it.
Hi I have 2 type of data entry which needs to be stored in db so it can be used for calculations later. Each entry has a unique id for it. The data entry are -
1.
2.
So I have to save this data in DB. With my understanding I thought of the following -
Create 3 tables - Common, Entry1 and Entry2(multiple tables with unique id as name)
The Common table will have a unique entry of each data and which table refer to for the value (Entry1/Entry2).
The Entry1 data is a single line so it can be inserted. But the Entry2 data will require a complete table because of its structure. So whenever we add a type 2 entry then a new table has to be created, which will create a lot of tables.
Or I could save the type2 values in another database and fetch the values from there. So please suggest me a way which is better than this.
I believe that you have 2 entry types with identical structure, but one containing a single row and one containing many.
In this case, I would suggest a single table containing the data for all entries, wtih a second table grouping them together. Even if your input contains a single row, it should still gain an EntryID. Perhaps something like the below:
I have multiple flat files(.csv) as my source in a folder.Each file has varying number of columns which may or may not intersect with other files. However, all columns in any source file are always present in my destination table that contains the super set of all these columns.
My requirement is to loop through each of these files and dynamically map columns that are available in that file to the destination table(header names of csv file match column names in table).
Structure of File 1:
id, name, age, email
Structure of File 2:
id, name, age, address, country
Structure of File 3:
id, name, age, address
Structure of Destination Table:
id, name, age, address, country, email
I want to populate the table for all columns with data for what is available and NULL for what's not for every record. How can I achieve this using SSIS?
you can do this by adding one Flat File Connection Manager add only one column with Data type DT_WSTR and a length of 4000 (assuming it's name is Column0)
In the dataflow task add a Script Component after the Flat File Source
In mark Column0 as Input Column and Add 6 Output Columns (id, name, age, address, country, email)
In the Input0_ProcessInputRow method split this column and assign values to output column. (you can write the logic you want) you can read answers at the following question to get an example: Reading CSV file some missing columns
The Flat File Source does not support dynamic file format, you have to use multiple sources to load these files.
I have multiple flat files as my source. Each files are having some common columns. But there are also some extra columns in one of them. I need to move all the data from flat files to a table having the structure with just the common columns and skip the extra columns in the one of the files. Structure examples below:
Structure of File 1:
id, name, age
Structure of File 2:
id, name, age, address, country
Structure of File 3:
id, name, age, address
Structure of Table:
id, name, age
I want to populate the table for only the three common columns between the files. Rest I need to ignore. How can I achieve this using SSIS?
You will need to separate different type of files in different folder (you can do that in SSIS using file system task by matching file names etc, or in powershell). Files in each folder must have the same number of columns and name/type. Then you create for each loop for each folder and iterate through to get the data from required columns and load them into the destination table. You may need a final step (execute sql ) to remove duplication in the destination table.
Easy. Set up 3 DataFlow tasks, each one connecting to a different file. Each one writes to the same table (e.g. through an OLEDB Destination). For all except File1 (which matches the table structure), simply don't map the extra, unwanted columns (i.e. address,country) in the OLEDB Destination. The values in these columns will just get discarded, and not go into the table.
You need a for each loop to iterate through the folder where the flat files are.
As long as the columns you want are in the same position in each file then all should be good.
Have a look here.