I'm working on importing a CSV into my DB. The data should work, it's data from another application that we used to get through a dblink that we now have to get from CSV thanks to an upgrade.
Dates look like this: 4/30/2001
I tried to do a simple import like I do with numbers and strings, and got an error, so I did a derived column.
The derived column works on the dates that are not null (formula is (DT_DATE)DTE)
The derived column however failed on dates that can contain nulls. I even tried to update the formula to (ISNULL(EDTE) ? NULL(DT_DATE) : (DT_DATE)EDTE). No success, I still get the error:
[Flat File Source - O1 [6743]] Error: Data conversion failed. The data conversion for column
"EDTE" returned status value 2 and status text "The value could not be converted because of a
potential loss of data.".
Originally, in my flat file source, the dates were set to type date, and I got an error before it even got to the derived column. I've changed that to String, and it makes it to derived column, but both boxes are red. Here's the error:
[Derived Column 1 [17270]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.
The "component "Derived Column 1" (17270)" failed because error code 0xC0049063 occurred,
and the error row disposition on "input column "EDTE" (17809)" specifies failure on error.
An error occurred on the specified object of the specified component.
There may be error messages posted before this with more information about the failure.
Has anyone seen this? Am I importing my data in wrong?
I found the solution.
I import the date as a String. Then using derived column, I use this formula to convert it:
RTRIM([T-DTE]) == "" ? NULL(DT_DATE) : (DT_DATE)[T-DTE]
Related
I've been working on an SSIS package for which I've added derived columns (which I've posted earlier on here). The issue I'm running into is that this child package is running fine in one environment, but for some reason, is failing in my Dev environment.
The errors occurring are:
"SSISError Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "DerivedColumn" (9567)" failed because error code 0xC0049067 occurred, and the error row disposition on "output column "SchemaID"(9744)" specifies failure on error" and "SSISError Code DTS_E_PROCESSINPUTFAILED. TheProcessInput method on component "Derived Column" (9567) failed with error code 0xC0209029 while processing input "Derived Column Input"(9568). The identified component returned an error from the ProcessInputmethod".
I've compared the table's fields, particularly SchemaID in both environments and they're identical. I even tried recreating the particular table in Dev and re-tried running the package, to no avail.
This is the Derived Column Transformation Editor screen:
Can anyone please make any suggestions on this? Thank you in advance!
Most probably its bad data that is not being converted through your derived transformation. What would be ideal is to create a test table and change the data type of SchemaID column to varchar and get all the data in there. It will be easier for you to debug the issue. Alternatively, you may want to open the "Advanced Editor" tab for your derived column and change either the data type of input schemaID or the length of it (never mind if its int or something else that doesnt support lengths).
The SchemaId expression is the following:
SUBSTRING((DT_WSTR,15)PCMRetrievalCode,3,FINDSTRING(((DT_WSTR,15)PCMRetrievalCode),"C"C,1) - 3)
I think that the issue is caused by the FINDSTRING function. In case that PCMRetreievalCode column doesn't contains "C" or it will return an index < 3 then FINDSTRING(((DT_WSTR,15)PCMRetrievalCode),"C",1) - 3 will throw an exception because in the SUBSTRING function the length parameter should be positive.
You can add a conditional ? : to solve the issue:
FINDSTRING(((DT_WSTR,15)PCMRetrievalCode),"C",1) >= 3 ? SUBSTRING((DT_WSTR,15)PCMRetrievalCode,3,FINDSTRING(((DT_WSTR,15)PCMRetrievalCode),"C"C,1) - 3) : ""
Another suggestion is to increase the column length more than 13.
First of all, when the column isn't NULL, it is successfully able to do the conversion. The destination DB field accepts null values.
In my Data Conversion step I have set the Input Column's DataType to numeric[DT_Numeric], Precision 18, and Scale 2.
I have two rows in my CSV, the first row does not contain any NULLS.. and if I execute that it's a success. However when I add a 2nd row with a NULL value in that column it fails and I get these errors:
[Data Conversion [2]] Error: Data conversion failed while converting column "Column 24" (138) to column "DbColumn24" (22). The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
[Data Conversion [2]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "Data Conversion.Outputs[Data Conversion Output].Columns[DbColumn24]" failed because error code 0xC020907F occurred, and the error row disposition on "Data Conversion.Outputs[Data Conversion Output].Columns[DbColumn24]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Data Conversion" (2) failed with error code 0xC0209029 while processing input "Data Conversion Input" (3). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
I do not think I need a Derived Column step before data conversion step to handle any nulls because the DB column accepts nulls.
Since your source is a CSV, I doubt that you actually have any NULL values in your source at all. More likely you have empty string values, which are not the same as NULL, and empty strings cannot be implicitly converted to Decimal. You will probably have to explicitly convert the empty strings to NULL in an expression.
The work around would be
1) First convert the column to varchar/nvarchar
2) Use derived column to convert to Decimal
3) Map the converted column to destination
I have a CSV file which contains this data:
EmployeeCode,EmployeeName,EmployeeSalary,Date
101,raju,1000,2/2/2003
102,krish,100,3/4/2005
103,rishbh,320,12/9/2007
104,rani,4690,12/8/2008
105,olesia,2000,17/4/2009
106,olga,2000,12/6/2010
107,mellisa,3330,12/4/2011
And I have table called employees:
EmployeeCode nvarchar(50)
EmployeeName nvarchar(50)
EmployeeSalary money
Date datetime
When I try to load this CSV file into my table using the SSIS package it gives me an error
1) [ADO NET Destination [2]] Error: An exception has occurred during
data insertion, the message returned from the provider is: The given
value of type String from the data source cannot be converted to type
date of the specified target column.
2) [SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.
The ProcessInput method on component "ADO NET Destination" (2) failed
with error code 0xC020844B while processing input "ADO NET Destination
Input" (9). The identified component returned an error from the
ProcessInput method. The error is specific to the component, but the
error is fatal and will cause the Data Flow task to stop running.
There may be error messages posted before this with more information
about the failure.
However If I remove date column from the CSV and try to insert everything works well. What is wrong with date column, why is it not taking the values?
Use OLE DB Destination rather than ADO NET Destination
This issue is due to default datatype of flat file is picked "string [DT_STR] " which is equal to varchar in database , but you have made your destination table data type nvarchar is equal to Unicode string [DT_WSTR] .
Either using derived column Change data type string [DT_STR] to Unicode string [DT_WSTR]
or
Go to show advance editor (source ) >> Input and output Properties output columns >> DAta type Property >> change data type to Unicode string [DT_WSTR] for EmployeeCode and EmployeeName.
First of all you should use OLE DB destination instead of ADO.Net Destination if your table is huge and now come to issue ,in your table data type of date is as 'datetime' so either you have to change your input data to date time by using Right click on component and go to advance editor.
Otherwise you can data conversion between source and destination and convert date as "DB_Datetime".I hope you will not face any issue when you do accordingly.
I had the same issue and my answer is going to sound silly. I struggled with it for 2 days and ended up realizing that the date format was wrong in my CSV file.
for example:
105,olesia,2000,17/4/2009 -- this is invalid because it ends up taking 17 as a month and
its invalid
blew my mind when I realized it! Hope this helps anyone else going forward!
I have a column named Paid_Date in my table that has a kind of improper date value with a colon in between date and the time values (e.g. '04MAY2015:00:00:00').
In order to exclude that colon, I have used an expression
((DT_DBTIMESTAMP)SUBSTRING(PAID_DATE,1,9))
in my Derived Column transformation.
When I try running this, I'm getting the following error.
Expression used in my trasformation: (DT_DBTIMESTAMP)SUBSTRING(PAID_DATE,1,9)
Source column data type : varchar
Source column Value: 04MAY2015:00:00:00
Error: [Derived Column [1613]] Error: An error occurred while attempting to perform a type cast.
Detailed Error:[Derived Column [1613]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.
The "component "Derived Column" (1613)" failed because error code 0xC0049064 occurred,
and the error row disposition on "output column "PaidDate" (1954)" specifies failure on error.
An error occurred on the specified object of the specified component.
There may be error messages posted before this with more information about the failure.
I have even tried to use the following expression:
(DT_DBTIMESTAMP)((DT_STR,50,1252)SUBSTRING(PAID_DATE,1,9))
Is it not possible to convert a string after a SUBSTRING operation?
Casting a string to a DT_DBDATETIME must use a specific string format:
yyyy-mm-dd hh:mm:ss[.fff]
You'll need to transform the string to the standard format before the cast will work.
Extract the month, use a lookup transformation or nested ternary operators to change it to two digit numeric string, then format the results into a string you can cast into a DT_DBDATETIME.
I have an SSIS package I am using to load a Fixed Width flat file. I have put in all the column lengths and have two packages against similar files working correctly. The third however keeps throwing the following error:
[Source 1 [16860]] Error: Data conversion failed. The data conversion for column "Line Number"
returned status value 2 and status text "The value could not be converted because of a
potential loss of data.".
[Source 1 [16860]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.
The "output column "Line Number" (16957)" failed because error code 0xC0209084
occurred, and the error row disposition on "output column "Line Number" (16957)"
specifies failure on error. An error occurred on the specified object of the specified
component. There may be error messages posted before this with more information about
the failure.
After doing some testing this happens for any column I have the is using the DT_I4 Data Type and has a blank in the column. I was going to try using a derived column, but this seems to fail for some of the columns even if I change them to a string data type to handle the blank as a NULL and then do a conversion to an INT later in the data flow.
In the source and the destination task I have the Retain NULL values checkbox ticked however this hasn't changed anything.
Any suggestions on handling this error where INT seems to fail at converting a blank to a NULL?
DT_I4 maps to a four byte signed integer in SSIS.
You were on the right track with your derived column. You just need to add the right expression.
You can try this expression:
ISNULL([Line Number]) ? "0":[Line Number]
This link may also be of use - see the postcode column in the example
http://www.bidn.com/blogs/DonnyJohns/ssas/1919/handling-null-or-implied-null-values-in-an-ssis-derived-column
I ended up using the approach from this blog post:
http://www.proactivespeaks.com/2012/04/02/ssis-transform-all-string-columns-in-a-data-flow-stream/
to handle all of the null and blank columns via a script task and data conversion.