SSIS - Derived Column - sql-server

I am having an issue with using the Derived Column Transformation Editor and was wondering if anyone on here could help me out.
I have a column called WorkitemNumber that holds information like INC0000001234
I want to change the WorkitemNumber to a more unique number to prevent overlapping entries in between months. So I came up with this...
I have a Derived Column:
Derived Column Name: ConvertID
Derived Column: <add as new column>
Expression: REPLACE(WorkitemNumber,"INC","") + "913"
Data Type: Unicode string [DT_WSTR]
Length: 258
The expression above changes INC0000001234 to 0000001234913. What I would like to do is cut out all of the extra 0s, but I want to refrain from using LTRIM because it will break once the incident report numbers go to the next set of thousands.
I am pretty new with using SSIS and it would be really helpful if someone would be able to tell me how I would go about getting rid of the extra 0s on the end regardless of the amount of 0s in front of the incident number.
I would want the final output to be 1234913 recardless if the incident number was INC0000001234 or INC001234.
EDIT
I was able to drop the 0s but I am getting the following error and the information is not being written into my other tables...
[SQL03 [459]] Error: SSIS Error Code DTS_E_OLEDBERROR.
An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0"
Hresult: 0x80004005 Description: "Invalid character value for cast specification".
[SQL03 [459]]
Error: There was an error with input column "ConvertIncidentID" (1015)
on input "OLE DB Destination Input" (472).
The column status returned was: "Conversion failed because the data value
overflowed the specified type.".
[SQL03 [459]]
Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.
The "input "OLE DB Destination Input" (472)"
failed because error code 0xC020907A occurred, and the error row disposition on
"input "OLE DB Destination Input" (472)"
specifies failure on error. An error occurred on the specified object of the
specified component. There may be error messages posted before this with more
information about the failure.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.
The ProcessInput method on component "SQL03" (459)
failed with error code 0xC0209029 while processing input "OLE DB Destination Input"
(472). The identified component returned an error from the ProcessInput method.
The error is specific to the component, but the error is fatal and will cause the
Data Flow task to stop running. There may be error messages posted before this with
more information about the failure.

If you're trying to strip the leading zeros, I would go with this approach
This expression removes the INC from your string
REPLACE(WorkitemNumber,"INC","")
Once INC is removed, you can cast to a numeric type to drop the leading zeros and then back to a string to allow you to do concatenation or whatever else you may need to do
(DT_WSTR,10)((DT_I8)REPLACE(WorkitemNumber,"INC",""))
Personally, I find it more user friendly to chain Derived Columns together so that if it breaks, I can spot the error more easily than something like the above. In your case, I used the original replace to generate a column and then use
(DT_WSTR,10)((DT_I8)WorkItemNoINC)
in my second Derived column
Edit
Taking another bite at the apple
OLE_SRC Query
I have defined the following source query
SELECT
'INC0000001234' AS WorkitemNumber
, CAST(MONTH(D.foo) AS varchar(2)) + RIGHT('0' + CAST(DAY(D.foo) AS varchar(2)), 2) AS fudge
FROM
(
-- Generate a year's worth of dates
SELECT
DATEADD(d, NUMS.n, '2012-12-31') AS SourceDate
FROM
(
SELECT TOP 366 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS n
FROM sys.all_columns AS SC
) NUMS
) D(foo);
This query generates 366 rows of data. A constant column named WorkItemNumber and one called fudge that will be all the possible month + day combinations. I left pad the number with a 0 so that a value of 121 is clearly January 21 and not December 1.
DER Drop INC
Here I create a new column called WorkItemNoINC and use the expression
REPLACE(WorkitemNumber,"INC","")
The result of this operation is that the text INC is removed from the source column and an empty string substituted in its stead.
DER WorkItemConcat
In this step, I concatenate the WorkItemNoINC column with our fudge value generated from the query. String + String yields string named WorkItemConcate (because I didn't proofread)
WorkItemNoINC + fudge
DER Strip leading zeros
In this operation, I will cast the value of WorkItemConcate to a bigint and back to string to strip the leading zeros.
(DT_WSTR,10)((DT_I8)WorkItemConcate)
Run
I added a data viewer after the last transformation and you can see that for all the possible values of day and month in a year, these expressions do not present a problem.
Making it fail
If I update my source query to use a NULL for WorkItemNumber, this package will not fail as you are experiencing.
If I change the casing in WorkItemNumber to inc, then I can generate this error.
Error: 0xC0049064 at DFT Whatevs, DER Strip leading zeros [18]: An error occurred while attempting to perform a type cast.
Error: 0xC0209029 at DFT Whatevs, DER Strip leading zeros [18]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "DER Strip leading zeros" failed because error code 0xC0049064 occurred, and the error row disposition on "DER Strip leading zeros.Outputs[Derived Column Output].Columns[Derived Column 1]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047022 at DFT Whatevs, SSIS.Pipeline: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "DER Strip leading zeros" (18) failed with error code 0xC0209029 while processing input "Derived Column Input" (19). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
Similar but not the same as you. I don't have a 2008 instance handy to test against but try breaking your operations out across multiple derived columns transformations and add data viewers as needed.

Related

Error at Data Conversion Transformation when Trying to Convert String from Excel Source to DB_TIMEZONEOFFSET to SQL Server Destination

Been several years since I had to write an SSIS package and struggling with converting a datetimezone string from Excel to Data Conversion transformation converting to DB_TIMEZONEOFFSET to a SQL Server Destination.
The following is the Timestamp in all 7 rows of Excel source:
The date column from the Excel source is a Unicode String [DT_WSTR] as seen below:
The SQL Server destination field is setup as DATETIMEOFFSET(7) as seen below:
I am receiving the following errors on the Data Conversion transformation when run the data flow:
[Data Conversion 2] Error: Data conversion failed while converting
column "Column1.createdAt" (82) to column "Copy of Column1.createdAt"
(38). The conversion returned status value 2 and status text "The
value could not be converted because of a potential loss of data.".
[Data Conversion 2] Error: SSIS Error Code
DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "Data
Conversion.Outputs[Data Conversion Output].Columns[Copy of
Column1.createdAt]" failed because error code 0xC020907F occurred, and
the error row disposition on "Data Conversion.Outputs[Data Conversion
Output].Columns[Copy of Column1.createdAt]" specifies failure on
error. An error occurred on the specified object of the specified
component. There may be error messages posted before this with more
information about the failure.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The
ProcessInput method on component "Data Conversion" (2) failed with
error code 0xC0209029 while processing input "Data Conversion Input"
(3). The identified component returned an error from the ProcessInput
method. The error is specific to the component, but the error is fatal
and will cause the Data Flow task to stop running. There may be error
messages posted before this with more information about the failure.
It is also important to note that I did also try to use a Derived Column transformation before the Data Conversion transformation as well to cast as DATETIMEOFFSET(7) to match that of SQL Server destination and received errors there as well similar to above.
Been fighting this for a while now and any extra eyes will help.
Thanks.
ssis cannot do the conversion if there are T and Z letters in the field. Use the REPLACE command to replace the letter T with space and the letter Z without characters. Like This : (DT_DBTIMESTAMPOFFSET,7)REPLACE(REPLACE(createdAt,"T"," "),"Z","")

SSIS data conversion from CSV source column NULL to DB destination Decimal(18,2)

First of all, when the column isn't NULL, it is successfully able to do the conversion. The destination DB field accepts null values.
In my Data Conversion step I have set the Input Column's DataType to numeric[DT_Numeric], Precision 18, and Scale 2.
I have two rows in my CSV, the first row does not contain any NULLS.. and if I execute that it's a success. However when I add a 2nd row with a NULL value in that column it fails and I get these errors:
[Data Conversion [2]] Error: Data conversion failed while converting column "Column 24" (138) to column "DbColumn24" (22). The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
[Data Conversion [2]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "Data Conversion.Outputs[Data Conversion Output].Columns[DbColumn24]" failed because error code 0xC020907F occurred, and the error row disposition on "Data Conversion.Outputs[Data Conversion Output].Columns[DbColumn24]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Data Conversion" (2) failed with error code 0xC0209029 while processing input "Data Conversion Input" (3). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
I do not think I need a Derived Column step before data conversion step to handle any nulls because the DB column accepts nulls.
Since your source is a CSV, I doubt that you actually have any NULL values in your source at all. More likely you have empty string values, which are not the same as NULL, and empty strings cannot be implicitly converted to Decimal. You will probably have to explicitly convert the empty strings to NULL in an expression.
The work around would be
1) First convert the column to varchar/nvarchar
2) Use derived column to convert to Decimal
3) Map the converted column to destination

SSIS how to convert object to varBinary for SQL

I have a csv file from csvde which has the objectGUID and some other information from AD.
I want to use SSIS to copy this info into the following table in SQL
[DN] [varchar](255) NULL,
[ObjectGuid] [varbinary](50) NULL,
[pwdExpiry] [bigint] NULL
Sample csv file from csvde file looks like this:
DN,objectGUID,msDS-UserPasswordExpiryTimeComputed
"CN=DEFRAVMDROOTCA1,OU=Test,DC=company,DC=net",X'c7eb536bc818f842bde11d8152755e7e',9223372036854775807
"CN=Sajeed Temp,OU=Test,DC=company,DC=net",X'417979aad0a9cc49818d1960ec95fb80',130834271604215531
"CN=IAMToolset.2,OU=Test,DC=company,DC=net",X'5a936d02842275458c45c1eb932a0f67',130849689154565841
"CN=IAMToolset.3,OU=Test,DC=company,DC=net",X'c03d2d11f8d8fe40a0eac70d41f3e1fd',130849689796313340
"CN=IAMToolset.4,OU=Test,DC=company,DC=net",X'e8b084e9d8c4a8428284ecb40b628da8',130849690015854425
"CN=IAMToolset.5,OU=Test,DC=company,DC=net",X'4b4e17dfce09e049941f3bd0ecddd5bb',130849690271803391
"CN=IAMToolset.6,OU=Test,DC=company,DC=net",X'004454a17c997b43af8e4b2653b491c2',130849690477125089
"CN=IAMToolset.7,OU=Test,DC=company,DC=net",X'8f70d9c64514c1409364624e3b079d42',130849690693072263
"CN=IAMToolset.8,OU=Test,DC=company,DC=net",X'ff70df18badafb4491225b29470b9a3d',130849690864329935
"CN=IAMToolset.9,OU=Test,DC=company,DC=net",X'a081e666a2df3c4481a4f124b489fdd8',130849691049650737
"CN=IAMToolset.1,OU=Test,DC=company,DC=net",X'8c59987d0a5c2a4ea1e98d07a84e1144',130849691453887596
"CN=testmigration,OU=Test,DC=company,DC=net",X'a800d0e29335d248ad990ad31fd5dd52',130732638130905654
"CN=DEFRAVMDPKMCA1,OU=Test,DC=company,DC=net",X'0d67d30bcbbbed48af9ed7736f9b1992',9223372036854775807
I have the SSIS package with two connection managers. One of them is for flat file and I'm reading the column objectGUID from the csv as a string. The thing is I want to convert this column inside a dataflow task to a format which I can push to the [ObjectGuid] varbinary NULL column in the SQL table.
I have use the data conversion transform and tried converting to DT_GUID and DT_BYTES but they both fail at that data conversion step when I run the package.
So what I really want to see is that if I have an objectGUID in csv file as
X'168dea153aab0a45adf8079b94041e90' (string)
I want to see it in the SQL table as
0x168DEA153AAB0A45ADF8079B94041E90 (varBinary)
The error I get on SSIS runtime is:
Error: 0xC02020C5 at Data Flow Task, Data Conversion [2]: Data conversion failed while converting column "ObjectGuid.LeftCleaned" (16) to column "Copy of ObjectGuid.LeftCleaned" (6). The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
Error: 0xC0209029 at Data Flow Task, Data Conversion [2]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "Data Conversion.Outputs[Data Conversion Output].Columns[Copy of ObjectGuid.LeftCleaned]" failed because error code 0xC020907F occurred, and the error row disposition on "Data Conversion.Outputs[Data Conversion Output].Columns[Copy of ObjectGuid.LeftCleaned]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047022 at Data Flow Task, SSIS.Pipeline: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Data Conversion" (2) failed with error code 0xC0209029 while processing input "Data Conversion Input" (3). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
Please help
I believe the key is here:
"The value could not be converted because of a potential loss of data."
Ensure you have given a large enough Length to the Data Conversion task to accommodate for the largest possible value in the CSV. Also on the other side you need to make sure that the column can hold that same amount of data after conversion.

Unexplainable error while writing data into Excel file in SSIS

My DFT fetches data from an SP. It is followed by a conditional split to split dataset into 5 parts based on a column value. The sub-datasets are then written into an excel file. All 5 outputs write into the same excel file, But different tabs. I have explicitly mentioned the Column range and starting row for the data to be written. Data load into the Excel file fails with the following error.
[EXC DST EAC TREND HPS [1379]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
[EXC DST EAC TREND HPS [1379]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "Excel Destination Input" (1390)" failed because error code 0xC020907B occurred, and the error row disposition on "input "Excel Destination Input" (1390)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "EXC DST EAC TREND HPS" (1379) failed with error code 0xC0209029 while processing input "Excel Destination Input" (1390). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
The most frustrating part is that the Excel destinations fail at random. Lets name them as A,B,C,D and E. During 1st run A will fail and the rest will write data successfully. During the 2nd run, A will succeed but C and E will fail and so on.
Every time the Excel Dest fails, data is still written in that particular tab. But I can see some blank rows in the dataset. I added a data viewer before each Excel Dest and the data looks correct. There are no blank rows either. number of rows per data set are fixed (18).
I am using a template every time which has only Column headers.
4 columns have datatype of nvarchar with max data length of around 50. 12 columns are Numeric(38,2).
Any help would be very much appreciated.
Thanks,
Gaz
Reorganize the package so that the writes will occur sequentially. The error you are seeing is the result of concurrency issues. You can't write to the same file in 5 different places at once. You have to do let each write finish before the next one starts.

SSIS Blank Int Flat File Fail on Load

I have an SSIS package I am using to load a Fixed Width flat file. I have put in all the column lengths and have two packages against similar files working correctly. The third however keeps throwing the following error:
[Source 1 [16860]] Error: Data conversion failed. The data conversion for column "Line Number"
returned status value 2 and status text "The value could not be converted because of a
potential loss of data.".
[Source 1 [16860]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.
The "output column "Line Number" (16957)" failed because error code 0xC0209084
occurred, and the error row disposition on "output column "Line Number" (16957)"
specifies failure on error. An error occurred on the specified object of the specified
component. There may be error messages posted before this with more information about
the failure.
After doing some testing this happens for any column I have the is using the DT_I4 Data Type and has a blank in the column. I was going to try using a derived column, but this seems to fail for some of the columns even if I change them to a string data type to handle the blank as a NULL and then do a conversion to an INT later in the data flow.
In the source and the destination task I have the Retain NULL values checkbox ticked however this hasn't changed anything.
Any suggestions on handling this error where INT seems to fail at converting a blank to a NULL?
DT_I4 maps to a four byte signed integer in SSIS.
You were on the right track with your derived column. You just need to add the right expression.
You can try this expression:
ISNULL([Line Number]) ? "0":[Line Number]
This link may also be of use - see the postcode column in the example
http://www.bidn.com/blogs/DonnyJohns/ssas/1919/handling-null-or-implied-null-values-in-an-ssis-derived-column
I ended up using the approach from this blog post:
http://www.proactivespeaks.com/2012/04/02/ssis-transform-all-string-columns-in-a-data-flow-stream/
to handle all of the null and blank columns via a script task and data conversion.

Resources