i have multiple excel file along with multiple sheet in each excel file. i have odi install on UNIX operating system. With windows we can directly create ODBC connection & pass it on Physical schema. What if we have UNIX system. Problem statement : I have converted the excel file into csv with unix script but most of the columns present in excel have string values
This is how we can't differentiate comma which is used in string and column separation.
Any idea how to overcome this.
My approach has always been to convert the Excel file to CSV using a PERL script. This can be triggered in ODI using an ODI Tool OdiOsCommand.
I think I used this one which allows a lot of parameters : https://metacpan.org/pod/release/KEN/xls2csv-1.07/script/xls2csv
This one looks good as well : https://github.com/soh-i/Excel2CSV
Related
I want to import data on a weekly basis to an Oracle DB.
I'm receiving this data on specific location in a server in EDR format. For now I'm uploading them manually using Toad for Oracle uploader wizard. Is there any way to upload them automatically using Unix or any kind of scripting?
I would suggest to try out SQL loader through a shell script.
Code:
sqlldr username#server/password control=loader.ctl
two important files:
a. your data file to be uploaded.
b. Control file which states the table to be inserted and the delimiter character and the column fields, etc. basically describe how to load the data.
Oracle Reference
After creating a dynamic Excel connection manager with Visual Studio 2015 SSIS and iterating through multiple Excel files in a directory I have run into the problem of not being able to change the number of columns in the connection manager. The Excel files do not have the same number of columns (or heading names/locations). I'm passing the data from the connection manager straight into a script component in order to handle this.
I tried creating an Excel connection manager with more columns they I will ever use before switching it to a Package Connection and setting the Expressions ExcelFilePath to my For/Each loop variable but this doesn't seem to work. I've received the VS_NEEDSNEWMETADATA error after this and, after rebuilding, received a
"Column "F18" cannot be found at the datasource"
error when an Excel sheet with fewer than 18 columns was passed through the for/each loop.
Any suggestions or assistance would be appreciated. Thank you.
If the columns count are different between Excel files, you cannot use the same Excel source to import them. You will always get the×VS_NEEDSNEWMETADATA exception that you mentioned.
If you are handling Excel files with same structure but with different columns order you can refer to my detailed answer on the link below:
Importing excel files having variable headers
If you have the choice to convert Excel files to Flat files, there are many links that describe the full process on how to import files with different structure.
If you don't have this choice, you must think on automating packages creation which is more complex (using BIML or DTS wrappers)
Update 1
Some links about converting Excel to csv:
Script task for converting Excel to CSV
Convert XLS to CSV on command line
Auto convert Xls to CSV
Convert xlsx file to csv using batch
Batch convert xls-Files to csv
SSIS process for saving .xlsx file as .csv file
SSIS using Script task - Convert XLS to CSV
I have 40 excel sheets in a single folder. I want to load them all in different tables sql server database through SSIS package. The difficulty I am having is because of different number and name of columns in each excel sheet.
Can this task be achievable through a single package?
Another option, if you want to do it in one data flow, you can write custom C# source component with multiple outputs. In the script task you'll figure out the file type and send the data to the proper output.
NPOI library(https://npoi.codeplex.com/) is a good way to read excel files in C#.
But if you have fixed file formats I would prefer to create N Data Flows inside Foreach loop container. Use regular Excel source components and just ignore errors in each data flow. This will let you get a file and try to load it in each data flow one by one. On error you will not fail the package but just go to the next data flow until you find the proper file format.
It can only be done, by adding multiple sources or using a script component, white a flag on what sheet it is. Then you can use a conditional split and enter multiple destinations.
Should I reuse the same Flat File Connection Manager or I should place individual Flat File Connection Manager for each files to be imported?
Description:
I have 30 CSV files with different structure and I need to import these files into the SQL Server database.
Currently I am using separate Flat File Connection Manager for each Flat File Source. Then pushing the data into the SQL Server database using OLEDB Destination.
Should I reuse the same Flat File Connection Manager?
Can you guide me - how to do this? Any links will be helpful.
Because the structure differs between your files you should use separate connections. This allows you to properly define the column names, sizes and data types for each file type.
Instead of creating 30 Flat File Connections just use one inside a Foreach Loop container passing in an Expression for the file name.
To solve the problem of your CSV files being in different formats, when you create the Flat File Connection select Ragged right instead of delimited, this will treat each row in file as one very wide column instead of multiple fields (make sure you make the column wide enough to handle your files).
You can then send the output of your Flat File Source into a Script Component, in to which you can put all the logic to handle your files. Use a Regex or a split to convert each row back into fields and then you have the full power of C# to process each row. A Script Component can also have multiple outputs so you can even use it like a Conditional Split.
This might seem like a bit more work (depends or what your files are like and how you are processing them), but then end result is less moving parts.
I think you can use MultiFaltFile as Source connection Manager. Using this you can select multiple files at a time.
See the link below:
http://www.sqlservergeeks.com/sql-server-import-multiple-files-in-ssis-using-multi-flat-file
I'm writing an SSIS package to import the contents of several Excel files into a SQL Server database for my client. These files will be provided regularly and the system should be completely automated without user involvement.
The Excel files are provided by my client's business partners, so I don't have a lot of control over them.
One of the files seems to be in the Excel 2003 SpreadsheetML XML format. Note that this is different from Open XML. It seems from my research that SSIS cannot read this format. It can and does read "normal" Excel 2003 files just fine.
Does anyone know of a way (in code) to convert this file into either non-XML Excel 2003 or Excel 2007 so I can import it? It needs to be automated, so opening the file using Excel and "save as" another type is off the table.