I have a SSIS package created in VS2008 that dumps a SQL Server table into a flat file. I then try to copy that flat file into AWS Redshift and it bombs due to the flat file having a BOM in the first position. I changed the flat file manager's code page to 65001 (UTF-8) thinking this would solve the problem, but it didn't. How do I prevent a BOM from being added to the flat file?
Related
I am trying to restore a database using PgAdmin on a Mac but the tables do not get imported. Tables came in csv format and compressed in a zipped file but I made it into a tar file because PgAdmin only accepts tar format but I still don't get the tables.
I changed the tables from CSV to excel workbook (xlxs) format and compressed them into a tar file again but this doesn't work. I reinstalled the program and used an old file I used before (also a tar file, inner files are in dac format). The tables in this (old) file get restored so I know it's not my system. I have tried restoring the database by going the directory route and using the unzipped folder but the tables still don't come up. I then tried creating the tables on PgAmin and importing the data from the CSV file, does not work either. I am not sure what else to do.
I have 1000's of Excel files, and the data stored in them needs to be imported into SQL Server. The minimum size of 250 kb to 50mb.
Currently, I am storing the files in the server location and importing each file content to SQL Server. Once the data imported, the physical file still remains in the system for future reference.
But now the file occupies more than 25Gb of our server space. I don't want to delete the source files.
Can anyone help me sort out this problem?
I'm planning to convert the source file into bytes and store those bytes in SQL Server. But I don't know it is the right way of handling it.
CSV is the best way for keep your files .You should try convert your .xls to .csv ...That happend to me and I resolved with this method.
After creating a dynamic Excel connection manager with Visual Studio 2015 SSIS and iterating through multiple Excel files in a directory I have run into the problem of not being able to change the number of columns in the connection manager. The Excel files do not have the same number of columns (or heading names/locations). I'm passing the data from the connection manager straight into a script component in order to handle this.
I tried creating an Excel connection manager with more columns they I will ever use before switching it to a Package Connection and setting the Expressions ExcelFilePath to my For/Each loop variable but this doesn't seem to work. I've received the VS_NEEDSNEWMETADATA error after this and, after rebuilding, received a
"Column "F18" cannot be found at the datasource"
error when an Excel sheet with fewer than 18 columns was passed through the for/each loop.
Any suggestions or assistance would be appreciated. Thank you.
If the columns count are different between Excel files, you cannot use the same Excel source to import them. You will always get the×VS_NEEDSNEWMETADATA exception that you mentioned.
If you are handling Excel files with same structure but with different columns order you can refer to my detailed answer on the link below:
Importing excel files having variable headers
If you have the choice to convert Excel files to Flat files, there are many links that describe the full process on how to import files with different structure.
If you don't have this choice, you must think on automating packages creation which is more complex (using BIML or DTS wrappers)
Update 1
Some links about converting Excel to csv:
Script task for converting Excel to CSV
Convert XLS to CSV on command line
Auto convert Xls to CSV
Convert xlsx file to csv using batch
Batch convert xls-Files to csv
SSIS process for saving .xlsx file as .csv file
SSIS using Script task - Convert XLS to CSV
I'm very new to SSIS packages so please forgive me if this is a simple query.
I have 2 SSIS packages that have been set up;
The first picks up a csv file, formats the data slightly (cuts off a prefix in one of the columns) and places it in another folder with an updated filename and timestamp.
The Second package imports the formatted file into a SQL database table.
The issue that I have is that the incoming file names for the first package may differ, the structure of the data will remain the same.
Is there a way to configure the flatfile connection manager to pickup any files in the C:Incoming\ folder?
Thanks in advance.
You can use a foreach loop container, to get the files in a folder, and use expressions in flat file connection to get it.
For a detailed answer, you can refer to this article:
Loop through Flat Files in SQL Server Integration Services
I have a recurring issue with importing Excel spreadsheets into SQL Server. As a work around, I am trying to convert the worksheet to a text file, comma delimited. I can convert the excel file to text but cannot seem to then import the text file into SQL. When I try to create a path between the text file and the OLE DB destination. I do notice that the flat file is showing as a destination ("Flat File Destination"). Is there a way to get this to go from the destination text file to SQL after doing the transformation to text?
If you are trying to do this in one dataflow, turning your 1st destination into a source, you can't do it like that.
If you have a dataflow that moves the data from an Excel Source to a Flat File Destination, you then need to create a second dataflow that uses the Flat File as a SOURCE, and moves the data to an OLEDB Destination.
Just place the second dataflow after the first one on the Control Flow.