I have an SSIS package that imports an XML file into SQL. The data of one particular field could be '112' or '039', for example. It is always three characters and gets padded with a leading 0 if only two.
The Destination field in SQL is varchar. For some or other reason SSIS is changing it to DT_UI2 and in the case of '039', only '39' comes.
I have added a data conversion that converts it to DT_WSTR but this does not help
Use a derived column with the following expression:
RIGHT("000" + (DT_WSTR,50)[Source Column],3)
The XSD that was originally generated defined this field as unsigned short. Changing it to string and redoing the flow solved the problem
Related
in SSIS I read a csv file with column format - for example 1.25 or 2.50.
In the datatransformation task I transform into decimal dt_decimal scala 2.
In the datatable the column has the format decimal(18,2).
The data will be stored with 125.00 or 25.00 instead of 1.25 and 2.50.
What do I have to adjust?
Possible Issue causes
(1) Data type mismatch
I think that a similar issue is caused by data type mismatch between source and destination, or data transformation output and Destination.
(2) Numeric separators
Another cause may be if the numeric values contains a comma , as formatting such as thousands commas 1,000,000 and decimal separator . like 1.02.
Possible solutions
(1) Specify data type in source
To prevent this issue to be caused by data transformations and if your source data is formatted well. Then there is no need for Data Transformation. Inside the Flat File Connection Manager editor. GoTo Advanced Tab, Select the column that contains decimal and change its data type (try DT_NUMERIC and DT_DECIMAL) and precision and scale property.
If the issue still happening, be sure that both input and output has same metadata (precision and scale).
(2) Derived Column
Or you should use a Derived Column Transformation with a similar expression:
(DT_NUMERIC,18,2)[COLUMN]
(3) Replace Separator using derived column
You can replace separator using a derived column
(DT_NUMERIC,18,2)REPLACE([COLUMN], ",", ".")
I am using SSIS and have a flat fixed width file I'm pulling into a SQL DB table that is coming over as a string. It comes over with 0000531164 and that needs to write to the SQL DB as 5311.64.
I'm using a derived column to remove the 0's but am not sure if I need to add a . to the expression or if there is a way to convert an assumed decimal to currency properly?
Current expression = (DT_WSTR,50)(DT_I8)(SUBSTRING(<Table Name>,21,16))
Another way, on the SQL Server side, would be to use STUFF and then just CAST it.
declare #var varchar(64) = '0000531164'
select cast(stuff(#var,len(#var) - 1,0,'.') as decimal(32,2))
In derived Column:
(DT_CY)([FieldName])/100
I don't think I explained it (or fully understood) what was needed at this point. I have a fixed width file that has client level and account level in same file. So those are separated in the package using conditional split and then I needed to separate the digits with a . to get the proper value. I ended up using this expression (DT_CY)(SUBSTRING(Filler,37,11) + "." + SUBSTRING(Filler,48,2)) where "Filler" is my fixed width values and the numerics are the start and length of the fixed width file.
Was overthinking it, this thread helped baseline what was needed. Thank you all.
I'm using SSIS to load a fixed length Flat File into SQL.
I have a weight field that has been giving me trouble all day.
It has a length of 8 with 6 DECIMAL POSITIONS IMPLIED (99V990099).
The problem i'm having is when it isn't populated and has 8 spaces.
Everything i try gets an error:
"Invalid character value for cast specification"."
OR
"Conversion failed because the data value overflowed the specified type.".
OR
Data conversion failed.
The data conversion for column "REL_WEIGHT" returned status value 2 and status text
"The value could not be converted because of a potential loss of data.".
I've tried declaring it as DT_String & DT_Numeric.
I've tried many variations of:
TRIM([REL_WEIGHT])=="" ? (DT_STR,8,1252)NULL(DT_STR,8,1252) : REL_WEIGHT
ISNULL([REL_WEIGHT]) || TRIM([REL_WEIGHT]) == "" ? (DT_NUMERIC,8,6)0 : (DT_NUMERIC,8,6)[REL_WEIGHT]
TRIM(REL_WEIGHT) == "" ? (DT_NUMERIC,8,6)0 : (DT_NUMERIC,8,6)REL_WEIGHT
But nothing seems to work.
Please someone out there have the fix for this!
I think you may be running afoul of the following point, explained nicely at http://vsteamsystemcentral.com/cs21/blogs/applied_business_intelligence/archive/2009/02/01/ssis-expression-language-and-the-derived-column-transformation.aspx:
You can add a DT_STR Cast statement to the expression for the MiddleName, but it doesn't change the Data Type. Why can't we change the data type for existing columns in the Derived Column transformation? We're replacing the values, not changing the data type. Is it impossible to change the data type of an existing column in the Derived Column? Let's put it this way: It is not possible to convert the data type of a column when you are merely replacing the value. You can, however, accomplish the same goal by creating a new column in the Data Flow.
I've solved this on past occasions by loading the data from the flat file as strings, and then deriving a new column in a Derived Column transformation which is of numeric type. You can then perform the appropriate trimming, validation, casting, etc. in the SSIS expression for that new column in the transformation.
Here, I found an example SSIS expression I used at one point to derive a time value from a 4-digit string:
(ISNULL(Last_Update_Time__orig) || TRIM(Last_Update_Time__orig) == "") ? NULL(DT_DBTIME2,0) : (DT_DBTIME2,0)(SUBSTRING(TRIM(Last_Update_Time__orig),1,2)+":"+SUBSTRING(TRIM(Last_Update_Time__orig),3,2)+":00")
There has to be a better way to do it, But i found a way that works.
Create a Derived Column Expression:
TRIM(REL_WEIGHT) == "" ? (DT_STR,9,1252)"0.0000000" : (DT_STR,9,1252)(LEFT(REL_WEIGHT,2) + "." + RIGHT(REL_WEIGHT,6))
THEN Create a Data Conversion Task to change it to Numeric and set scale to 6.
And then Map the [Copy of NewField] to my SQL table field set up as Decimal(8,6).
I don't know how the performance will be of that when loading a million records, probably not the best. If someone knows how to do this in a better way performance wise please let me know.
Thanks,
Jeff
I want to read data from text file (.csv), truncate one of the column to 1000 characters and push into SQL table using SSIS Package.
The input (DT_TEXT) is of length 11,000 characters but my Challenge is ...
SSIS can convert to (DT_STR) only if Max length is 8,000 characters.
String operations cannot be performed on Stream (DT_TEXT data type)
Got a workaround/solution now;
I truncate the text in Flat File Source and selected the option to Ignore the Error;
Please share if you find a better solution!
FYI:
To help anyone else that finds this, I applied a similar concept more generally in a data flow when consuming a text stream [DT_TEXT] in a Derived Column Transformation task to transform it to [DT_WSTR] type to my defined length. This more easily calls out the conversion taking place.
Expression: (DT_WSTR,1000)(DT_STR,1000,1252)myLargeTextColumn
Data Type: Unicode string [DT_WSTR]
Length: 1000
*I used 1252 codepage since my DT_TEXT is UTF-8 encoded.
For this Derived Column, I also set the TruncationRowDisposition to RD_IgnmoreFailure in the Advanced Editor (or can be done in the Configure Error Output, setting Truncation to "Ignore failure")
(I'd post images but apparently I need to boost my rep)
I have a csv file :
1|1.25
2|23.56
3|58.99
I want to put this value in a SQL Server table with SSIS.
I have created my table :
CREATE TABLE myTable( ID int, Value numeric(4,2));
My problem is that I have to create a Derived Column Transformation to specify my cast :
(DT_NUMERIC,4,2)(REPLACE(Value,".",","))
Otherwise, SSIS don't seem to be able to put my Value in my column, and fill my column with null value.
And I think it is tooooo ugly to do it this way. I want my Derived Column Transformation be here for real new derived column, and not some simple cast that I think SSIS have to detect.
So, what is the standard way to use SSIS to resolve this problem ?
BULK
INSERT myTable
FROM 'c:\csvtest1.txt'
WITH
(
FIELDTERMINATOR = '|',
ROWTERMINATOR = '\n'
)
csvtest1.txt
1|1.25
2|23.56
3|58.99
You're loading this up in international format (56,99 in lieu of 56.99). You need to load this as 56.99 for SQL Server to recognize it as such. Take out the REPLACE(Value, ".", ",") and just have the code be:
(DT_NUMERIC,4,2)(Value)
Handle the formatting on the application side, not on the data side. The comma is a reserved operator in SQL Server and you can't change that fact.
Haven't used SSIS a whole lot, but can't you set the regional settings on the File Source or at least set the decimal separator?
Can you change your SSIS source column to be in the correct datatype?
If you have control over the production of your file, I'd suggest you to format values without ANY decimal or thousand separation : in this case I'ld have a file with values:
1|125
2|2356
3|5899
and then apply a division by 100 when importing the data. While it has the advantage of being culture-independent, of course it has some drawbacks:
1) First of all, it may not be possible to impose this format of the file.
2) It presumes that all numeric values are formatted accordingly, in this case every value is multiplied by 100; this can be an issue if you have to mix values from countries with different decimal positions (many have two decimals, but some have zero decimals).
3) It may severely impact with other routines, maybe out of your control
Therefore, this can really be an option if you have total control on the csv file.