So in my Source Table from Excel I have a Column called real/min/max that counts population and I want to split this into 3 columns called ActualPop, MinPop, MaxPop.
So an example would be
real/min/max
33/1/50
And I would need this to populate in the new Columns as
ActualPop
33
MinPop
1
MaxPop
50
I tried the following Expressions:
ActualPop: TOKEN([real/min/max],"/",1)
MinPop: TOKEN([real/min/max],"/",2)
MaxPop: TOKEN([real/min/max],"/",3)
The issue is when I try to do my mapping to the SQL destination, I get an error about the Data Types. The destination has INT data types mean while in the Derived Column Editor I see the Data Types are Unicode String. I have tried to use the Data Conversion but that still does not work.
You can change the data type of the 3 derived columns using the Advanced Editor on the Derived Column.
then.... select four-byte signed integer [DT_I4] for each of your INTs.
See: Changing Datatype in SSIS Derived column
Related
I'm writing an SSIS package to load data from a .csv into a db.
There's a column in the csv file that is supposed to have a count, but the records sometimes have text, so I can't just load the data in as an integer. It looks something like this:
I want the data to land in the db destination as an integer instead of a string. I want the transformation to change any text to a 1, any blank value to a 1, and leave all the other numbers as-is.
My attempts have so far included using the Derived Column functionality, which I couldn't get the right expression(s) for it seems, and creating a temp table to run a sql query through the data, which kept breaking my data flow.
There are three approaches you can follow.
(1) Using a derived column
You should add a derived column with the following expression to check if the values are numeric or not:
(DT_I4)[count] == (DT_I4)[count] ? [count] : 1
Then in the derived column editor, go to the error output configuration and set the error handling event to Ignore failure.
Now add another derived column to replace null values with 1 :
REPLACENULL([count_derivedcolumn],1)
You can refer to the following article for a step-by-step guide:
Validate Numeric or Non-Numeric Data in SQL Server Integration Services without the Script Task
(2) Using a script component
If you know C# or Visual Basic.NET, you can add a script component to check if the value is numeric and replace nulls and string values with 1
(3) Update data in SQL
You can stage data in its initial form into the SQL database and use an update query to replace nulls and string values with 1 as follows:
UPDATE [staging_table]
SET [count] = 1
WHERE [count] IS NULL or ISNUMERIC([count]) = 0
I find this little challenging. I have source table and destination table.
Source table has one of the column with char datatype with length of 2.
Destination table has one of the column with int datatype.
Task is to migrate data from source to destination using SSIS package. And in that process I am using derived column transform to handle this conversion.
Column in Source table has value as "01" or "02" or "03". Whereas destination column having int datatype cannot be mapped with source column. As int only recognizes "1". I want 1 or 2 or 3 only and omit 0 in destination column.
I tried few things as below.
Try #1:
SUBSTRING(COLNAME, 2, 1)
This gives me "1" from "01" but when I do
(DT_R4) SUBSTRING(COLNAME, 2, 1 )
It fails me outright.
Try #2:
I try to cast like below.
(DT_DECIMAL, 2)(DT_R4) SUBSTRING(COLNAME, 2, 1 )
That did not work as well.
If there is a way out, I appreciate any direction here.
Since the destination column in of type Integer DT_I4 then you should use the following expression in the derived column:
(DT_I4)[SourceColumn]
Also check for Nulls and Empty strings:
ISNULL([SourceColumn]) ? NULL(DT_I4): ([SourceColumn] == "" ? NULL(DT_I4): (DT_I4)[SourceColumn])
Have you tried to directly connect source and destination columns without using the data conversion or derived column transformation? I think It should implicitly convert from char to Integer if all the data in the source table are numbers though the data type is char.
in SSIS I read a csv file with column format - for example 1.25 or 2.50.
In the datatransformation task I transform into decimal dt_decimal scala 2.
In the datatable the column has the format decimal(18,2).
The data will be stored with 125.00 or 25.00 instead of 1.25 and 2.50.
What do I have to adjust?
Possible Issue causes
(1) Data type mismatch
I think that a similar issue is caused by data type mismatch between source and destination, or data transformation output and Destination.
(2) Numeric separators
Another cause may be if the numeric values contains a comma , as formatting such as thousands commas 1,000,000 and decimal separator . like 1.02.
Possible solutions
(1) Specify data type in source
To prevent this issue to be caused by data transformations and if your source data is formatted well. Then there is no need for Data Transformation. Inside the Flat File Connection Manager editor. GoTo Advanced Tab, Select the column that contains decimal and change its data type (try DT_NUMERIC and DT_DECIMAL) and precision and scale property.
If the issue still happening, be sure that both input and output has same metadata (precision and scale).
(2) Derived Column
Or you should use a Derived Column Transformation with a similar expression:
(DT_NUMERIC,18,2)[COLUMN]
(3) Replace Separator using derived column
You can replace separator using a derived column
(DT_NUMERIC,18,2)REPLACE([COLUMN], ",", ".")
Can someone tell me how I code in SQL Server so that I am looking in a varchar text column to see if it contains a numerical range within the text?
For example, I'm looking for columns that contain anything between 100000 and 999999. The column may have a value like
this field contains a number `567391`
so I want to select that one, but not if it had
this field contains a number `5391`
For your given example, you can check the digits:
where col like '%[^0-9][1-9][0-9][0-9][0-9][0-9][0-9][^0-9]%'
This is not a generic solution, but it works for your example. In general, parsing strings in SQL Server is difficult. It is better to extract the values you are interested in when loading the data, so the relevant values are correctly in their own columns.
I'm using SSIS to load a fixed length Flat File into SQL.
I have a weight field that has been giving me trouble all day.
It has a length of 8 with 6 DECIMAL POSITIONS IMPLIED (99V990099).
The problem i'm having is when it isn't populated and has 8 spaces.
Everything i try gets an error:
"Invalid character value for cast specification"."
OR
"Conversion failed because the data value overflowed the specified type.".
OR
Data conversion failed.
The data conversion for column "REL_WEIGHT" returned status value 2 and status text
"The value could not be converted because of a potential loss of data.".
I've tried declaring it as DT_String & DT_Numeric.
I've tried many variations of:
TRIM([REL_WEIGHT])=="" ? (DT_STR,8,1252)NULL(DT_STR,8,1252) : REL_WEIGHT
ISNULL([REL_WEIGHT]) || TRIM([REL_WEIGHT]) == "" ? (DT_NUMERIC,8,6)0 : (DT_NUMERIC,8,6)[REL_WEIGHT]
TRIM(REL_WEIGHT) == "" ? (DT_NUMERIC,8,6)0 : (DT_NUMERIC,8,6)REL_WEIGHT
But nothing seems to work.
Please someone out there have the fix for this!
I think you may be running afoul of the following point, explained nicely at http://vsteamsystemcentral.com/cs21/blogs/applied_business_intelligence/archive/2009/02/01/ssis-expression-language-and-the-derived-column-transformation.aspx:
You can add a DT_STR Cast statement to the expression for the MiddleName, but it doesn't change the Data Type. Why can't we change the data type for existing columns in the Derived Column transformation? We're replacing the values, not changing the data type. Is it impossible to change the data type of an existing column in the Derived Column? Let's put it this way: It is not possible to convert the data type of a column when you are merely replacing the value. You can, however, accomplish the same goal by creating a new column in the Data Flow.
I've solved this on past occasions by loading the data from the flat file as strings, and then deriving a new column in a Derived Column transformation which is of numeric type. You can then perform the appropriate trimming, validation, casting, etc. in the SSIS expression for that new column in the transformation.
Here, I found an example SSIS expression I used at one point to derive a time value from a 4-digit string:
(ISNULL(Last_Update_Time__orig) || TRIM(Last_Update_Time__orig) == "") ? NULL(DT_DBTIME2,0) : (DT_DBTIME2,0)(SUBSTRING(TRIM(Last_Update_Time__orig),1,2)+":"+SUBSTRING(TRIM(Last_Update_Time__orig),3,2)+":00")
There has to be a better way to do it, But i found a way that works.
Create a Derived Column Expression:
TRIM(REL_WEIGHT) == "" ? (DT_STR,9,1252)"0.0000000" : (DT_STR,9,1252)(LEFT(REL_WEIGHT,2) + "." + RIGHT(REL_WEIGHT,6))
THEN Create a Data Conversion Task to change it to Numeric and set scale to 6.
And then Map the [Copy of NewField] to my SQL table field set up as Decimal(8,6).
I don't know how the performance will be of that when loading a million records, probably not the best. If someone knows how to do this in a better way performance wise please let me know.
Thanks,
Jeff