In SSIS I am querying the Active Directory for a list of domain users. The results (according to SSIS) are of data type 'Unicode text stream [DT_NTEXT]'.
I am using the Data Conversion item from the SSIS Toolbox to translate all the field before inserting them into an SQL table. I can go in and give each field an Output Alias and an output 'Data Type of String [DT_STR]' (I tried in the default options window and in the Advanced Properties window). I click OK to save my settings, but the red X is still there saying it can't convert to Unicode and if I reopen the properties window all the Data Types are set back to Unicode.
Has anyone had this happen to them before?
Unicode and non-unicode are incompatible types for conversion. You need to use a derived column transformation to add a new column. Use a data conversion in the expression:
(DT_STR, 20, 1252)[MyUnicodeCol]
Unicode and non-unicode are incompatible types for conversion. We need to convert that particular column as DT_WSTR from DT_NTEXT using Data conversion and then again we have convert it as DT_STR from DT_WSTR.
Conversion from DT_NTEXT to DT_STR
Related
I am facing a strange issue while using SSIS "Data Conversion component" to convert string to decimal datatype. I use SSIS 2016.
The source data input has values of mixed data types- string, integer, decimal and is defined as varchar in the flat file source. The target data type expected is numeric. When explicit type conversion happens from string to decimal, we expect the alphanumeric values to get rejected to error table and only the numeric values to pass through.
Instead, we are seeing some alphanumeric values shedding the characters in the value and passing through successfully with no error.
Examples: Value "3,5" converted to 35
Value "11+" converted to 11
We do not have control over source data and will not be able to replace char data before passing data into Data conversion component.
We have tried the below steps as a workaround and it has worked.
i.e,
First Data Conversion from DT_STR to DT_NUMERIC
Capture error rows that fail the above conversion
Second Data Conversion from DT_NUMERIC to DT_DECIMAL
But as the source data is not reliable, we may have to apply this workaround wherever there are numeric fields (int types & deicmals) which is not a friendly solution.
So checking with you all to understand if there is an easier and better solution tried out by anyone.
I did not expect this result, but I tried an expression task and it worked for DT_DECIMAL:
(DT_DECIMAL,1)"11+" -- evaluates to 11.0
But it does not work for DT_NUMERIC. SSIS won't allow a direct numeric result, but it can be nested inside a cast to DT_DECIMAL. Just to demonstrate that, in an expression task even this "numerically valid" cast would not be permitted, because the output simply can't be of type DT_NUMERIC:
(DT_NUMERIC, 3, 0)123
But this is permitted:
(DT_DECIMAL,0)((DT_NUMERIC, 3, 0)123)
So as long as you are happy to specify a precision and scale big enough to hold your data during the "validity" check done by DT_NUMERIC, and then cast it from there to DT_DECIMAL, all in a derived column transform, then DT_NUMERIC seems to enforce the strict semantics you want.
SSIS allows this:
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"11")
But not either of these:
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"11+")
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"3,5")
#billinkc Sorry for not responding to you earlier.
We are working under some restrictions:
(1) All we want to do is capture datatype issues in input data, so we wanted to harness the capability of SSIS Data Conversion Component in SSIS.
(2) DBA doesn't want us to use SQL for type conversions, so we are required to do these conversions between flat file source and flat file destination using SSIS.
(3) We are required to capture the type conversion errors at every step of conversion into an error output file with error column name and error description, to be used later. So we cannot remove char data in the field before passing it to Data Conversion component.
#allmhuran - We have used Derived column task before Data Conversion component to replace unnecessary characters in one of the other fields, but using the same for type conversion makes achieving (3) difficult. Because error output from Derived column task and Data Conversion component cannot be redirected to the same error output file.
We can completely ignore Data Conversion component and use only Derived column task to do all type conversions, whether single or nested. I am trying this and the error descriptions do not always look good, but the cons of the former method can be overcome. I will try this out!
in SSIS I read a csv file with column format - for example 1.25 or 2.50.
In the datatransformation task I transform into decimal dt_decimal scala 2.
In the datatable the column has the format decimal(18,2).
The data will be stored with 125.00 or 25.00 instead of 1.25 and 2.50.
What do I have to adjust?
Possible Issue causes
(1) Data type mismatch
I think that a similar issue is caused by data type mismatch between source and destination, or data transformation output and Destination.
(2) Numeric separators
Another cause may be if the numeric values contains a comma , as formatting such as thousands commas 1,000,000 and decimal separator . like 1.02.
Possible solutions
(1) Specify data type in source
To prevent this issue to be caused by data transformations and if your source data is formatted well. Then there is no need for Data Transformation. Inside the Flat File Connection Manager editor. GoTo Advanced Tab, Select the column that contains decimal and change its data type (try DT_NUMERIC and DT_DECIMAL) and precision and scale property.
If the issue still happening, be sure that both input and output has same metadata (precision and scale).
(2) Derived Column
Or you should use a Derived Column Transformation with a similar expression:
(DT_NUMERIC,18,2)[COLUMN]
(3) Replace Separator using derived column
You can replace separator using a derived column
(DT_NUMERIC,18,2)REPLACE([COLUMN], ",", ".")
I am working on SSIS Package to export the output data to Excel file. ( Excel Destination ).
I am running into conversion error.
Error Description : cannot convert between Unicode and non-Unicode string data types
Input Column Details
ColumnA ([DT_TEXT])
ColumnB ([DT_STR],200)
Data Conversion Output Column Details
ColumnA ([DT_TEXT])
ColumnB ([DT_WSTR],255)
How to convert Unicode datatype to excel column?
Using Derived Column Transformation
Add a derived column with the following expression
(DT_WSTR,255)[ColumnB]
When if fails you can use the Error Output to check the bad values causing the exception
Using Data Conversion Transformation
You can also achieve this using a Data Conversion transformation component. Just select the ColumnB as input and choose to convert to DT_WSTR data type with length = 255
Using Script Component
You just have to select ColumnB as Input column, add an Output column outColumnB of type DT_WSTR and length = 255. And just assign the input column to the output column inside the script.
Row.outColumnB = Row.ColumnB
Update 1 - Excel data types
Based on the following official documentation:
Import data from Excel or export data to Excel with SQL Server Integration Services (SSIS)
The Excel driver recognizes only a limited set of data types. For example, all numeric columns are interpreted as doubles (DT_R8), and all string columns (other than memo columns) are interpreted as 255-character Unicode strings (DT_WSTR). SSIS maps the Excel data types as follows:
Numeric - double-precision float (DT_R8)
Currency - currency (DT_CY)
Boolean - Boolean (DT_BOOL)
Date/time - datetime (DT_DATE)
String - Unicode string, length 255 (DT_WSTR)
Memo - Unicode text stream (DT_NTEXT)
Use data conversion tool and convert DT_WSTR to DT_STR.Check this:
Import Excel unicode data with SQL Server Integration Services
I'm getting the following error when I run my package:
[Data Conversion [2]] Error: Data conversion failed while converting column "FieldName" (373) to column "Copy of FieldName" (110).
The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
However, I don't understand why. I have double checked the inputs and outputs to validate that they make sense and are what I expect. I've also checked all the raw data in the column in my excel file.
My package setup:
Excel Datasource feeding Data Conversion then Derived Column and finally output to Ole DB Destination (sql)
What I've done:
I opened the advanced editor on the data conversion. I confirmed that the incoming data type is DT_STR which can be expected since the source datatype wasn't correctly identified. It is actually a date in my excel file. I confirmed that the data conversion output column is database timestamp [DT_DBTIMESTAMP] as I have set it to be. My destination table has a DateTime datatype for FieldName.
What am I missing?
I think this is a date format issue, check that column does not contains empty strings or NULL values.
Also check that values are similar to yyyy-MM-dd HH:mm:ss date format.
To read more about SSIS data types check the following article:
Integration Services Data Types
Also when converting string values to datetime, if values are well formated, just map the source column to the destination without Data conversion Transformation and they will be implicitly converted
Using Visual Studio 2015 Enterprise
I'm trying to change a few values inside the Script Transformation Editor but they are grayed out and I can't modify them.
Here I'm trying to change ScriptLanguage to Microsoft Visual Basic:
Here I would like to change the length of this HashValue column
I've tried restarting visual studio as well as removing the script and adding it back to no avail.
EDIT: I figured out the second one by changing the data type to DT_WSTR
First Issue
Note: Once you accessed the script editor window you cannot change the it's language.
But you can change your scripts default language from visual studio options. All you have to do is go to Tools and select Options.... Under the Business Intelligence Designers option, select Integration Services Designer and change the script language to whichever you prefer your default to be.
Second Issue
You cannot change the length column of type Integer:
DT_I1 is relative to Sql tinyInt data type (0 to 255)
DT_I2 is relative to Sql Smallint data type (-2^15 (-32,768) to 2^15-1 (32,767))
DT_I4 is relative to Sql Int data type (-2^31 (-2,147,483,648) to 2^31-1 (2,147,483,647))
DT_I8 is relative to Sql Big Int data type (-2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807))
Only length for DT_STR and DT_WSTR can be changed
MSDN articles about SSIS and Sql data types:
https://msdn.microsoft.com/en-us/library/ms141036(v=sql.120).aspx
https://msdn.microsoft.com/en-us/library/ms187752.aspx
https://msdn.microsoft.com/en-us/library/ms187745.aspx
Script Component Language
The Script Component ScriptLanguage property should generally be editable, UNTIL you have used the 'Edit Script...' dialog (since this builds up the backing project which can't be converted automatically). Try creating a new Script Component and editing this value first, but I was not able to replicate this being disabled at the start with my copy of VS2015.
Data Type Properties
Data type properties are controlled mainly by the selected DataType. In this case, you have a four-byte signed integer (DT_I4), which doesn't have any other settings. Other data types have different properties, i.e.:
DT_STR (string) can set Length and CodePage (character set),
DT_WSTR (Unicode string) can only set Length,
and DT_NUMERIC can set Scale and Precision.