Using SSIS I am downloading data from an Excel file into a MS SQL database. In this Excel file, there's a column with data looking like 99,99, which goes into a column of type float in the database. After downloading to the database, the data in this column has ignored the comma and it has become 9999.
Any suggestions on how to correctly download the data into my database and keep the comma?
Make sure the locale settings on the SSIS server are consistent with those of the Excel. Specifically look at the thousand separator and the decimal separator. If SSIS thinks the comma is a thousand-separator, it is ignored.
You could also make a Derived Column Transformation, in which you import this column from Excel as text, change the ,into a .using a string replace function, and then cast it to float.
Related
I have multiple Excel files that have the same format. I need to import them into SQL Server.
The issue I currently have is that there are two text columns that I need to ignore completely as they are free text and the character length for some rows exceeds what the server allows me to import which results in a truncation error.
Because I don't need these columns for my analysis, the table I'm importing to doesn't include these columns but for some reason the SSIS packages still picks up those columns and cuts the import job halfway through.
I tried using max character length for those columns which still results in the truncation error.
I need to create an SSIS package that ignores the two columns completely without deleting the columns from Excel.
You can specify which columns you need to ignore from the Edit Mappings dialog.
I have added the image for your reference:
If you just create the SSIS package in SSDT the Excel file can be queried to return only the required columns. In the package, create an Excel Connection Manager using the Excel file. Then on the Control Flow of the package add a Data Flow Task that has an Excel Source component in it. On this source, change the data access mode to SQL command and the file can then be queried similar to SQL. In the following example TabName is the name of the Excel tab containing the data that will be returned. If either the tab or any column names contain spaces they will need to be enclosed in square brackets, i.e. TabName would be [Tab Name].
Import/Export Wizard
Since you mentioned in the comments that you are using SQL Server Import/Export Wizard. You can solve that if you have a fixed columns (range) that you are looking to import (example: first 10 columns).
In Import/Export wizard, after selecting destination options you will be asked if you want to read from tables or query:
Select the query option, then use a simple select query and specify the columns range after the sheet name. As example:
SELECT * FROM [Sheet1$A:C]
The query above will read from the first 3 columns in Sheet1 since A:C represent the range between first column A and third column C.
Now, you can check the columns from the Edit Mappings dialog:
SSIS
You can use the same logic within SSIS package, just write the same SQL command in the Excel Source after changing the Access Mode to SQL Command.
The solution is simple. I needed to write a query that will exclude the columns. So instead of selecting "Copy data from one or more tables" you select "write a query" and exclude the columns you don't need. This one worked 100%
Good day to you, Experts.
I'm stuck on a problem I'm having with an Excel 97-02 .xls file.
When adding it as a source in SSIS, I'm getting an External Columns Datatype of DT_IMAGE .
The column represents an ID and is numeric only. I can't extract and work with the data because of the DT_IMAGE datatype.
Setting IMEX=1 didn't help.
Thank you in advance.
Reading Excel files in SSIS is done using OLEDB provider which may not detect the appropriate Excel column type.
There are many other questions mentioning similar issues such as:
SSIS Excel Import Forcing Incorrect Column Type
SSIS Excel Data Source - Is it possible to override column data types?
SSIS keeps force changing excel source string to float
As you mentioned in the question, if you added ;Extended Properties="IMEX=1" to the connectionstring with no luck then i think there is 4 things you can try:
Sorting column data inside Excel
Change the entire column formatting manually
Go to the advanced editor on the Excel source >> into the output column list and set the type for each of the columns.
Adding IMEX=1; MAXROWSTOSCAN=0 to the connectionstring
If nothing of the above steps worked then you should save the Excel sheet as a text file and then you use Flat File Connection manager
I am trying to import "Financial data" from Excel files in to sql table. Problem I am facing is that My ssis package is incrementing decimal values. e.g -(175.20) from Excel is being loaded as "-175.20000000000005" in SQL.
I am using nVArChar (20) in destination SQL table. Images attached. What's the best data type in destination table. I have done a lot of reading and people seem to suggest decimal data but Package throws error for Decimal data type.Need help please.
Ended up changing the Data type to "Currency" in my SQL destination. Then added a data conversion task to change "DT_R8" data type from excel source to "currency[DT_CY]. This resolved the issue. could have used decimal or Numeric (16,2)data type in my destination as well but then i just went ahead with currency and it worked.
You could use a Derived Column Transformation in your Data Flow Task, with an expression like ROUND([GM],2) (you might need to replace GM with whatever your actual column name is).
You can then go to the Advanced Editor of the Derived Column Transformation and set the data type to decimal with a Scale of 2 on the 'Input and Output Properties' tab (look under 'Derived Column Output').
You'll then be able to use a decimal data type in your SQL Server table.
I am trying to export data from SQL server 2008 to Excel file using BIDS.
One of the fields 'DESCRIPTION' coming from SQL database is VARCHAR(4000).
I can export everything to excel but the 'DESCRIPTION' field size in excel is restricted to unicode 255 and no mater what I try it does not allow me to export the data over 255 characters (exports it as blank). I tried to change SQL field as varchar(max) or ntext but none of attempts worked. I used advanced editor in BIDS on excel destination to change 'DESCRIPTION' character length manually but as soon as I hit 'OK', it resets to unicode 255.
Could anybody please help me to resolve this issue?
Thanks,
Vishal
So, I did some testing. Excel data transformation is funky but I came up with a solution. I created an excel spreadsheet with fields as needed. I then created fake, dummy data in excel with character length far greater than 255 and hid the row. I then did the SSIS data transformation to the excel spreadsheet which worked. It's a weird and not preferable option but it works.
Problem: Excel only accepts 255 chars per cell when I attempt to use Excel Destination in SSIS (2008 R2) from a sql server table. SalesForce data loader would not accept CSV (with “” text qualifiers) created by
ssis flat file connection manager. SalesForce will only accept CSV (with “” text qualifiers). SalesForce will accept CSV as exported by Excel (2010).
Solution:
1. Create your excel connection manager, set name/path of the destination EXCEL file in your “Excel Destination Data Flow Competent” and map meta-data.
2. Open a new Excel file, remove all extra “sheets”, rename “sheet1” to that was created in step#1, above, select all cells and format to “text”, add all the column header names to the first row of your template sheet. In the columns that need to hold more data than 255 limit, paste in any characters that exceed your limit by 50% (just in case). These columns are now configured to hold your large data. Save the file, naming it something like TEMPLATE_Excel_forLargeCellValues.xlsx
3. Copy this template into your DESTINATION connection: Before your “Excel Destination Data Flow Competent” in the SSIS Control Flow, create a new “File System Task”. Create an ssis pkg level variable to hold the path/filename of your template excel file. In your “File System Task” set “IsSourcePathVariable” = TRUE, set “SourceVariable” to User::Template_Excel. Set “IsDestinationPathVariable” = FALSE, and set “DestinationConnection” = from step #1 above. Set “Operation” = Copy file. “OverwriteDestination”=TRUE. This will now copy your formatted Excel workbook/sheet into your destination folder with the file name you designated in step #1 above and because you put a larger amount of sample data in the columns that require more than 255 chars, all your data will fit.
Note: It is not necessary to delay validation on any components.
You're saying that the excel field is set to 255 right? Changing the SQL field won't have an effect on excel, you'd have to modify the excel file.
I don't believe you can modify the Excel output column to write more than 255 characters. Why not simply write your output to a csv, it can be opened and later modified in Excel anyway.
SSIS excel engine recognizes datatype of first 8 rows and assigns it to excel source or destination automatically. Even defining the excel column as memo wont work. I tried to resolve the error by changing registry value TypeGuessRows of excel engine but it did not work either. So I was not left with any other option but to create a dummy row(2nd row) with more than 255 characters and hide it.Excel source then identify the column with unicode text stream. You have to write some logic in SSIS package to exclude this row if you are trying to import the data from excel. I heard that this issue is resolved in excel versions on and after 2010. But BIDS 2008 does not have option to choose any version after 2007 so this is the only solution if you are working with BIDS 2008 and excel.
You have to select Microsoft Excel 97-2003 and use the xls as file extension in your file name for destination.
I got the same issue of the excel destination not allowing more than 255 characters. After spending almost a day, I tried adding more characters (to simplify, I added spaces more than 255) in the header of the column that has the issue with more than 255 characters. And it magically worked!
You can insert dummy data (260 characters) to under head column you want in your excel (Execute SQL Task)
Script Create and insert
CREATE TABLE `YourSheet` (`myColumn260char` LongText)
GO
INSERT INTO YourSheet(myColumn260char) Values('....................................................................................................................................................................................................................................................................')
And you can delete dummy row after imported.
Every time that I try to import an Excel file into SQL Server I'm getting a particular error. When I try to edit the mappings the default value for all numerical fields is float. None of the fields in my table have decimals in them and they aren't a money data type. They're only 8 digit numbers. However, since I don't want my primary key stored as a float when it's an int, how can I fix this? It gives me a truncation error of some sort, I'll post a screen cap if needed. Is this a common problem?
It should be noted that I cannot import Excel 2007 files (I think I've found the remedy to this), but even when I try to import .xls files every value that contains numerals is automatically imported as a float and when I try to change it I get an error.
http://imgur.com/4204g
SSIS doesn't implicitly convert data types, so you need to do it explicitly. The Excel connection manager can only handle a few data types and it tries to make a best guess based on the first few rows of the file. This is fully documented in the SSIS documentation.
You have several options:
Change your destination data type to float
Load to a 'staging' table with data type float using the Import Wizard and then INSERT into the real destination table using CAST or CONVERT to convert the data
Create an SSIS package and use the Data Conversion transformation to convert the data
You might also want to note the comments in the Import Wizard documentation about data type mappings.
Going off of what Derloopkat said, which still can fail on conversion (no offense Derloopkat) because Excel is terrible at this:
Paste from excel into Notepad and save as normal (.txt file).
From within excel, open said .txt file.
Select next as it is obviously tab delimited.
Select "none" for text qualifier, then next again.
Select the first row, hold shift, select the last row, and select the text radial button. Click Finish
It will open, check it to make sure it's accurate and then save as an excel file.
There is a workaround.
Import excel sheet with numbers as float (default).
After importing, Goto Table-Design
Change DataType of the column from Float to Int or Bigint
Save Changes
Change DataType of the column from Bigint to any Text Type (Varchar, nvarchar, text, ntext etc)
Save Changes.
That's it.
When Excel finds mixed data types in same column it guesses what is the right format for the column (the majority of the values determines the type of the column) and dismisses all other values by inserting NULLs. But Excel does it far badly (e.g. if a column is considered text and Excel finds a number then decides that the number is a mistake and insert a NULL instead, or if some cells containing numbers are "text" formatted, one may get NULL values into an integer column of the database).
Solution:
Create a new excel sheet with the name of the columns in the first row
Format the columns as text
Paste the rows without format (use CVS format or copy/paste in Notepad to get only text)
Note that formatting the columns on an existing Excel sheet is not enough.
There seems to be a really easy solution when dealing with data type issues.
Basically, at the end of Excel connection string, add ;IMEX=1;"
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\\YOURSERVER\shared\Client Projects\FOLDER\Data\FILE.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";
This will resolve data type issues such as columns where values are mixed with text and numbers.
To get to connection property, right click on Excel connection manager below control flow and hit properties. It'll be to the right under solution explorer. Hope that helps.
To avoid float type field in a simple way:
Open your excel sheet..
Insert blank row after header row and type (any text) in all cells.
Mouse Right-Click on the head of the columns that cause a float issue and select (Format Cells), then choose the category (Text) and press OK.
And then export the excel sheet to your SQL server.
This simple way worked with me.
A workaround to consider in a pinch:
save a copy of the excel file, modify the column to format type 'text'
copy the column values and paste to a text editor, save the file (call it tmp.txt).
modify the data in the text file to start and end with a character so that the SQL Server import mechanism will recognize as text. If you have a fancy editor, use included tools. I use awk in cygwin on my windows laptop. For example, I start end end the column value with a single quote, like "$ awk '{print "\x27"$1"\x27"}' ./tmp.txt > ./tmp2.txt"
copy and paste the data from tmp2.txt over top of the necessary column in the excel file, and save the excel file
run the sql server import for your modified excel file... be sure to double check the data type chosen by the importer is not numeric... if it is, repeat the above steps with a different set of characters
The data in the database will have the quotes once the import is done... you can update the data later on to remove the quotes, or use the "replace" function in your read query, such as "replace([dbo].[MyTable].[MyColumn], '''', '')"