I have project parameter whose value will be like 'Group_AR_21Sep2021' and I want to fetch this date and put it into one of the column of the table which I am loading from Excel. So I put this in derived column (DT_DBDATE)(SUBSTRING(#[$Project::ARFileName],10,18)) but it is throwing me this error
[Derived Column [2]] Error: Casting expression "(SUBSTRING(#[$Project::ARFileName],10,18))" from data type "DT_WSTR" to data type "DT_DBDATE" failed with error code 0xC00470C2.
The second parameter to the substring function should be the length of the date: (DT_DBDATE)(SUBSTRING(#[$Project::ARFileName],10,9))
However, this would still fail because SSIS is super fragile when typecasting to date. I would suggest using a script component instead:
1 - Select the project parameter in the read only variables collection
2 - Add a column to the output columns and make it DT_DBDATE, i.e. NewDate
3 - Update the script to do the cast to the new column:
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
Row.NewDate = Convert.ToDateTime(Variables.ARFileName.Substring(9,9));
}
Related
I've got a block of data with lat/longs and I'm trying to add a point to a Snowflake table from this data.
First I tried to accomplish it when I created the table with:
create or replace table geo (
best_lat double,
best_lon double,
geography geography as (ST_POINT(best_lon, best_lat)));
This errored out with SQL compilation error: error line 4 at position 2 Data type of virtual column does not match the data type of its expression for column 'GEOGRAPHY'. Expected data type 'GEOGRAPHY', found 'VARIANT'
Then I tried to add the column with:
alter table geo
add column geom geography as (select st_makepoint(best_lon, best_lat) from geo)
This errored out with SQL compilation error: Invalid virtual column expression [(SELECT ST_MAKEPOINT(GEO.BEST_LON, GEO.BEST_LAT) AS "ST_MAKEPOINT(BEST_LON, BEST_LAT)" FROM GEO AS GEO)]
Clearly I'm doing something wrong here. Can anyone provide some insight?
Snowflake doesn’t really support calculated columns, what it does do is allow you to set a default value for a column and this default can be a simple SQL statement. The syntax is documented here
Because this isn’t a pure calculated column, you can still insert values directly into the column, which will override the defined default value.
I am working with ssis. I have a column that contains date in format integer YYYYMMDD: For example to day I get 20190214.I am using derived column to convert it in format date YYYY-MM-DD in my case for example 2019-02-14
But sometimes I receive wrong values for example 20188101. How could I test that I have good values to be converted to date?
There are 3 methods to achieve that
(1) Using Another Derived Column
Beside of the first Derived COlumn, you can add another derived column with the following expression:
(DT_DATE)(LEFT([DateColumn],4) + "-" + SUBSTRING([DateColumn],5,2) + "-" + RIGHT([DateColumn],2))
If the date value is incorrect redirect the bad from Error Output configuration , and you can handle these bad values from the Error Output.
Helpful Links
USING THE ERROR OUTPUT OF THE DERIVED COLUMN
How to prevent CAST errors on SSIS?
(2) Using a Script Component
Add a script component to check if the value is correct, and add a Output Column of type i.e. OutDateColumn, use a similar code (VB.NET)
IF Not Row.DateColumn_IsNULL Then
dim dtDate as DateTime
If DateTime.TryParseExact(Row.DateColumn,"yyyyMMdd",System.Globalization.InvariantCulture,System.Globalization.DateTimeStyles.None , dtDate ) Then
Row.outDateColumn = dtDate.ToString("yyyy-MM-dd")
Else
Row.outDateColumn_IsNull = True
End If
Else
Row.outDateColumn_IsNull = True
End If
(3) Add a Data Conversion Transformation
After the derived column Transformation, add a Data COnversion Transformation and try to convert the Date Derived Column you add to DT_DATE, if conversion failed, than handle the bad values from Error Output as explained in the first method.
I created a running SSIS and I tried inserting data without error to SQL Server
but suddenly this error message pops up
Error: 0xC02020A1 at Data Flow Task, Flat File Source [59]: Data conversion failed. The data conversion for column "SCB_ActualDTime" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
It says that the error occurred in row 8 containing these data from the
SCB_ActualDTime
2017-04-16 15:28:07
It is really weird because I tried inserting the same data via SQL script and there is no error message.
Flat File is not a data source that force a specific data types for each column. so it may contains unicode characters or white spaces that prevent reading column as Datetime.
After working with many cases i decided to work with flat file in this way.
In the Flat File Connection Manager i will read all column as DT_STR column without specifying other data types
Inside The DataFlow Task i will add a script component to check if columns value can be converted to the estimated data type and then converted it
Assuming that the date column name is inColumn
In the DataflowTask, Add a script component , Mark inColumn as input column, add a new Output column outColumn with dataType DT_DBTIMESTAMP
Change script language to vb.net
Mark your date column as Input
Create an Output column for each date column
Inside the script, in Input0_ProcessInputRow sub use DateTime.ParseExact Function as follow:
In this example the date column is MailReceivingDate
Public Class ScriptMain
Inherits UserComponent
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
If Not Row.MailReceivingDate_IsNull AndAlso
Not String.IsNullOrEmpty(Row.MailReceivingDate.Trim) Then
Dim dtDate as Date
If DateTime.TryParseExact(Row.MailReceivingDate.Trim, "yyyy-MM-dd HH:mm:ss", New System.Globalization.CultureInfo("en-GB"),System.Globalization.DateTimeStyles.None,dtDate) Then
Row.OutColumn = dtDate
Else
'If column cannot be parsed
Row.outColumn_IsNull = True
End If
Else
Row.outColumn_IsNull = True
End If
End Sub
End Class
I'm creating a table B from an exisitng table A. In the table A I have a column ValDate which is varchar and contains Date. When I try to create the table B I have a function used in the query as given below and I get a conversion error. Table A contains null values as well.
MS Access:
((DateDiff("d",Date(),format(Replace(Replace([Table A].ValDate,".","/"),"00/00/0000","00:00:00"),"dd/mm/yyyy")))>0)).
Tables were in MS Access and are being migrated to SQL Server 2012.
SQL Server:
((DATEDIFF(day,FORMAT( GETDATE(), 'dd-MM-yyyy', 'en-US' ),FORMAT( ValDate, 'dd-MM-yyyy', 'en-US' ))>0))
or
((DateDiff(day,GETDATE(),Format(Replace(Replace([TableA].[ValidFrom],'.','/'),'00/00/0000','00:00:00'),'dd/mm/yyyy')))
I tried converting the date using several approachs like Convert , Format and Cast but I end up getting error as below.
Msg 8116, Level 16, State 1, Line 1
Argument data type date is invalid for argument 1 of isdate function.
I would really appreciate someone telling me what I'm missing here.
since you have date data in a string field is very likely you have some value that is not valid against your expected date format.
copy the data in a sql server table and then perform check and validation of the content of the string field.
have a look to the function try_convert that can be helpful when checking the content of the string field containing the date values.
when bad data is ruled out you can apply again your formula with (hopefully) a different result.
a better solution would be to create a separate field with appropriate datatype to store date values converted from the string field and apply your logic to that field.
I'm working in ETL process where I have a column in a table varchar with data as 28JUN1952.
This will always be in this format i.e. 2 digits for date, 3 letters for months and 4 numbers for years.
I was able to convert this into a date column by I need this to come in yyyyMMdd format i.e. 28JUN1952 become 19520628. I was able split the years months and date, but unable to add a padding in the month if it's a 1 digit month for e.g. Jan or Feb.
Is there any other way to convert the data format or any other alternative to add a padding?
I need the conversion in SSIS only.
The easiest route will be to add a Script Transformation.
On the Input and Output tab, expand the Output 0, select Output Columns, click Add Column. 2. Rename Column to something like ccyymmdd and change the data type. I'm not sure what your target type is (int32, string?) but I'll assume string as I'm lazy
On the Input Columns tab, select the string date source column (assuming it's called srcDate)
Inside the script task, assuming C#, use code like this
Inside script task
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
DateTime x = DateTime.MinDate;
if (DateTime.TryParse(Row.srcDate, out x))
{
Row.ccyymmdd = x.ToString("yyyyMMdd");
}
else
{
// Handle NULL values however you wish
// Assign "known bad value"
// Row.ccyymmdd = "99991231";
// Or set the null property to true (preferred)
Row.ccyymmdd_IsNull = true;
}
}
Try this:
declare #d varchar(20)='28JAN1952';
SELECT replace(CONVERT(date,RIGHT(#d,4)+SUBSTRING(#d,3,3)+LEFT(#d,2)),'-','')