We receive data in json format with timestamp represented in UTC. When convert json to timestamp_tz field it casts successfully (SQL is executed in Australia : +11h), however when the same result is converting back to date it refers back to UTC and returns result that is not matching timestamp_tz date. This turns out to be issue when using with view and try to add where clause xxx::date = 'yyyy-mm-dd'
Example:
select jj:TransactionDatetime::TIMESTAMP_TZ full_date,
jj:TransactionDatetime::TIMESTAMP_TZ::date round_date,
jj:TransactionDatetime::TIMESTAMP_TZ::date = '2020-11-14' is_it_same_date
from
(
select parse_json('
{
"TransactionDatetime": "2020-11-13 23:26:31+00"
}'
) jj ) dd
Result
FULL_DATE,ROUND_DATE,IS_IT_SAME_DATE
"2020-11-14 10:26:31.000","2020-11-13","false"
FULL_DATE is 14-Nov-2020 with time potion - that is converted from the original json UTC timestamp
ROUND_DATE is supposed to represent date only portion from FULL_DATE. It does that but using UTC representation and this way removes one day.
As result if I have view on top of that data users start using date in "where" clause and receive unexpected result
I expect that ROUND_DATE should be "2020-11-14"
I think that is unexpected behaviour that needs fixing at snowflake.Certainly there is workaround but should work natively.
Thank you
Here:
alter session set timezone = 'Australia/Sydney';
select CURRENT_TIMESTAMP(); -- returns 2020-11-24 23:44:24.013 +1100 as of now
select '2020-11-13 23:26:31+00'::TIMESTAMP_TZ as full_date; -- returns 2020-11-13 23:26:31.000 +0000
select '2020-11-13 23:26:31+00'::TIMESTAMP_TZ::date as date_only; -- return 2020-11-13
select '2020-11-13 23:26:31+00'::TIMESTAMP_LTZ as full_date; -- returns 2020-11-14 10:26:31.000 +1100
select '2020-11-13 23:26:31+00'::TIMESTAMP_LTZ::date as date_only; --returns 2020-11-14
Related
The CAST and TRY_CAST functions:
Converts a value of one data type into another data type. The semantics of CAST are the same as the semantics of the corresponding TO_ datatype conversion functions. If the cast is not possible, an error is raised.
A special version of CAST , :: that is available for a subset of data type conversions. It performs the same operation (i.e. converts a value of one data type into another data type), but returns a NULL value instead of raising an error when the conversion can not be performed.
For obviosuly incorrect input produces either an error or NULL(as expected):
SELECT CAST('aaaa-09-20' AS DATE) AS col;
Date 'aaaa-09-20' is not recognized
SELECT TRY_CAST('aaaa-09-20' AS DATE) AS col;
col
NULL
For incorrect format:
ALTER SESSION SET DATE_INPUT_FORMAT = 'YYYY-DD-MM';
SELECT CAST('2021-09-20' AS DATE) AS col;
Date '2021-09-20' is not recognized
SELECT TRY_CAST('2021-09-20' AS DATE) AS col;
col
NULL
The situation is different for DATE that is incorrect(outside of allowed range 1582-9999) for both CAST and TRY_CAST:
ALTER SESSION SET DATE_INPUT_FORMAT = 'YYYY-MM-DD';
SELECT TRY_CAST('22021-09-20' AS DATE) AS col, CAST('22021-09-20' AS DATE) AS col2;
This time there is no error but a normal output:
Classic UI:
SnowsightUI:
In this scenario there is no error message/NULL value. The values are not usable anyway:
CREATE OR REPLACE TABLE t AS
SELECT TRY_CAST('22021-09-20' AS DATE) AS col, CAST('22021-09-20' AS DATE) AS col2;
Error:
Date '+22021-09-20' is not recognized
SELECT CURRENT_VERSION();
-- 6.26.1
Is there any specific reason for different handling of out-of-scope values?
EDIT:
Function
'aaaa-09-20'
22021-09-20 (SELECT)
22021-09-20 (CTAS)
CAST
Error
No error, unusable date
Error
TRY_CAST
NULL
No error, unsuable date
Error
I don't see any differences between the CAST and TRY_CAST functions:
CREATE OR REPLACE TABLE t AS
SELECT TRY_CAST('22021-09-20' AS DATE) AS col;
-- fails with: Date '+22021-09-20' is not recognized
CREATE OR REPLACE TABLE t AS
SELECT CAST('22021-09-20' AS DATE) AS col;
-- fails with: Date '+22021-09-20' is not recognized
And this one returns the same values in both columns as you showed:
ALTER SESSION SET DATE_INPUT_FORMAT = 'YYYY-MM-DD';
SELECT TRY_CAST('22021-09-20' AS DATE) AS col, CAST('22021-09-20' AS DATE) AS col2;
+-------------+-------------+
| COL | COL2 |
+-------------+-------------+
| 22021-09-20 | 22021-09-20 |
+-------------+-------------+
The difference is related to which layer you cast your dates. The SELECT queries are processed in Cloud Services Layer, but the CTAS query fails in the Query Processing Layer. So this is an inconsistent behaviour between CSL and WH.
An interesting test to see the difference:
CREATE OR REPLACE TABLE t AS
SELECT '22021-09-20' AS col;
SELECT TRY_CAST(col AS DATE) FROM t;
-- returns 22021-09-20
SELECT CAST(col AS DATE) FROM t;
-- returns 22021-09-20
The above queries are executed in Cloud Services Layer as they only access metadata (not the table itself), so they are completed without any errors.
CREATE OR REPLACE TABLE t AS
SELECT seq4() AS id, '22021-09-20' col FROM table(generator(ROWCOUNT=>100000));
SELECT TRY_CAST(col AS DATE) FROM t WHERE id = 50000;
-- fails with Date '+22021-09-20' is not recognized
This one fails, because this one is processed in Query Processing Layer
I have some sample data in snowflake as follows;
created_at
----------
2022-06-10T18::35::57
2022-06-10T18::35::57
The datatype of this column is VARCHAR(16777216), I am trying to filter for the rows with date June 10,2022. Here is my query;
select *
from table
where to_date(created_at) = date('2022-06-10', 'yyyy-mm-dd');
But this gives me following error; Date '2022-06-10T18::35::57' is not recognized. If we replace to_date by try_to_date then we get 0 rows. Unfortunately I can't go to backend and change the properties of the table. Therefore, I need to resort to sql datetime functions.
Can I please get help here, on how to fix above errors? thanx
Using LEFT to get date part:
where to_date(LEFT(created_at, 10), 'YYYY-MM-DD') = date('2022-06-10', 'YYYY-MM-DD');
There are two issues here -
1 - It is not a date but a timestamp
2 - usage of '::' rather then standard ':' in time portion
Below will not work as its not a date -
with date_cte(created_at) as
(select * from values
('2022-06-10T18::35::57'),
('2022-06-10T18::35::57'))
select to_date('2022-06-10T18::35::57') from date_cte;
100040 (22007): Date '2022-06-10T18::35::57' is not recognized
Using timestamp will not work too due to '::' in time portion
with date_cte(created_at) as
(select * from values
('2022-06-10T18::35::57'),
('2022-06-10T18::35::57'))
select to_timestamp('2022-06-10T18::35::57','yyyy-mm-dd"T"hh24:mi:ss') from date_cte;
100096 (22007): Can't parse '2022-06-10T18::35::57' as timestamp with format 'yyyy-mm-dd"T"hh24:mi:ss'
Below takes care of both -
with date_cte(created_at) as
(select * from values
('2022-06-10T18::35::57'),
('2022-06-10T18::35::57'))
select to_timestamp('2022-06-10T18::35::57','YYYY-MM-DD"T"HH24::MI::SS') from date_cte;
TO_TIMESTAMP('2022-06-10T18::35::57','YYYY-MM-DD"T"HH24::MI::SS')
2022-06-10 18:35:57.000
2022-06-10 18:35:57.000
Now to compare - just convert in desired format using to_char -
with date_cte(created_at) as
(select * from values
('2022-06-10T18::35::57'),
('2022-06-10T18::35::57'))
select * from date_cte
where to_char(to_timestamp('2022-06-10T18::35::57','YYYY-MM-DD"T"HH24::MI::SS'),'yyyy-mm-dd')='2022-06-10';
CREATED_AT
2022-06-10T18::35::57
2022-06-10T18::35::57
One possible way is to do this:
SELECT *
FROM table
WHERE CAST(SUBSTRING(created_at, 0, 11) as date) = '2022-06-10'
Table is having the column as ret_period and contains varchar datatype. I'm trying to write the query for converting the datatype of column from varchar to date datatype .but it is throwing an error as Conversion failed when converting date and/or time from character string.
Can any one please let me know how to resolve this issue!
below is the sample data of column.
ret_period
012018
012019
012020
You could transform the string to a format that can be converted/casted to a DATE.
select *
, TRY_CAST(CONCAT(RIGHT(ret_period, 4),'-',LEFT(ret_period, 2),'-01') AS DATE) AS ret_date
from (values
('012018'),
('012019'),
('012020')
) q(ret_period)
Note however, that the yyyyMMDD format should be a safer than yyyy-MM-DD, since it doesn't depend on the language when casting it to a DATETIME.
Or use the DATEFROMPARTS function
select *
, DATEFROMPARTS(RIGHT(ret_period, 4), LEFT(ret_period, 2), 1) AS ret_date
from (values
('012018'),
('012019'),
('012020')
) q(ret_period)
ret_period
ret_date
012018
2018-01-01
012019
2019-01-01
012020
2020-01-01
I have this Timestr in varchar 28/12/2016 16:39:01
How to Derived the column into date and time in SSIS
Desired result
Date 2016-12-28 in Date format
Time 16:39:01 in time format
This is what i have in SQL so far
SELECT FORMAT(CAST((SUBSTRING(TimeStr,12,8)) AS DATETIME),'hh:mm:ss tt') AS Time
Code
Result
Using SSIS expressions you can try:
DATE
(DT_DBDATE)(SUBSTRING(Timestr,7,4) + "-" +
SUBSTRING(Timestr,4,2) + "-" + SUBSTRING(Timestr,1,2))
TIME
(DT_DBTIME)(SUBSTRING(Timestr,12,8))
Your Derived Column settings should look like this:
UPDATE: Avoid miliseconds in DT_DBTIME
(DT_DBTIME2,0)(SUBSTRING(Timestr,12,8))
Hope it helps.
To convert your varchar into a datetime, I would recomment Try_Convert() if 2012+, if not just Convert() would do the trick provided your data is reasonable.
try_convert(datetime,Timestr,103) -- returns 2016-12-28 16:39:01.000
To use Format()
Declare #Timestr varchar(25) = '28/12/2016 16:39:01'
Select DateTime12 = format(try_convert(datetime,#Timestr,103),'yyyy-MM-dd hh:mm:ss tt')
,DateOnly = format(try_convert(datetime,#Timestr,103),'yyyy-MM-dd')
,Time12 = format(try_convert(datetime,#Timestr,103),'hh:mm:ss tt')
,Time24 = format(try_convert(datetime,#Timestr,103),'HH:mm:ss')
Returns
DateTime12 DateOnly Time12 Time24
2016-12-28 04:39:01 PM 2016-12-28 04:39:01 PM 16:39:01
I'm new to the T-SQl world and this issue is puzzling me.
I'm running a query against a variable I have created which holds the date of the start of my financial year, which is 2014-04-01. The data I'm querying against holds the date as a date time field, so I have been using CAST AS DATE to remove the timestamp.
If I run this query in this fashion I receive 25 results:
select count(B.WorkOrderNumber) as AWOCount
from TSP1_Dev.AssetDataPortal.EllipseSiteList A left outer join
TSP1_Dev.AssetDataPortal.tblWorkOrdersActive B
on A.EquipmentLocation = B.EquipmentLocation
where
B.EquipmentLocation = 'BISHT'
and A.EquipmentClass = 'SW'
and right(rtrim(A.EquipmentDesc),4) = '(OU)'
and cast(B.CreationDate as Date) >= '2014-04-01';
However, on one occasion I forgot to cast the date, and when I did that I received 32 results:
select count(B.WorkOrderNumber) as AWOCount
from TSP1_Dev.AssetDataPortal.EllipseSiteList A left outer join
TSP1_Dev.AssetDataPortal.tblWorkOrdersActive B
on A.EquipmentLocation = B.EquipmentLocation
where
B.EquipmentLocation = 'BISHT'
and A.EquipmentClass = 'SW'
and right(rtrim(A.EquipmentDesc),4) = '(OU)'
and B.CreationDate >= '2014-04-01';
Can someone please tell me why? Apologies for my ignorance!
Neil
In the query you have string literals and from your comment I can only assume that it is Coldfusion that insert the string literal in your query. (I know nothing about Coldfusion but is there no way to use parameters instead?).
In SQL Server there is a setting for the order of date parts in strings. SET DATEFORMAT
What you see can be reproduced if you have the date format dmy.
The string literal 2014-04-01 is interpreted as yyyy-dd-mm when compared against a datetime. When the new data types date and datetime2 was added that behaviour was changed so string literal on the form yyyy-mm-dd will always be interpreted as just that regardless of set dateformat.
So when you cast to date the string literal is converted to the date 2014-04-01 but when you compare against a datetime the string literal is converted to 2014-01-04.
set dateformat dmy;
declare #T table(D datetime);
insert into #T(D) values
('20140101'),('20140201'),
('20140301'),('20140401'),
('20140501'),('20140601');
select count(*)
from #T
where D >= '2014-04-01';
select count(*)
from #T
where cast(D as date) >= '2014-04-01';
Result:
5
3
One way to avoid the problem for datetime is to use a string literal without dashes. 20140401 will be interpreted as yyyymmdd regardless of set dateformat.