CAST int to DATE failure when used with INNER JOIN - sql-server

I have the following query:
SELECT
T2.PseudoDateColumn
FROM
(
SELECT
*
FROM T1
WHERE
PseudoDateColumn <> -1
) T2
INNER JOIN T3
ON T2.T2_key = T3.T3_key
WHERE
CAST(CAST(T2.PseudoDateColumn AS VARCHAR) AS DATE)
BETWEEN
DATEADD(mm, -1, DATEADD(mm, DATEDIFF(mm, 0, GETDATE()), 0))
AND DATEADD(ms, -3, DATEADD(mm, 0, DATEADD(mm, DATEDIFF(mm, 0, GETDATE()), 0)))
PseudoDateColumn is an int in T1. Values that aren't real dates have a -1 placeholder. All other dates are real dates (just stored as an int). Because of this, I need to first filter out all of the -1 values before I cast PseudoDateColumn to a DATE, such that I can use BETWEEN. When I run this script without the join, it works fine. But, when I run it with the join, SQL throws an error:
"conversion failed when converting date and/or time from character
string"
I don't know why this is happening. It's like SQL is doing the CASTing even before the sub-query.
Update:
I mistakenly wrote SQL Server 2008. I'm actually running this on PDW AU5, which has SQL Server 2012. I don't think that makes a difference, though.
Also, here is some sample data in PseudoDateColumn. All values are in the form YYYYMMDD.
PseudoDateColumn
-----------------
20150112
20160305
20111009

I know this is an old post but I think this can help someone. I had a similar error when trying to convert/cast a VARCHAR input parameter to a UNIQUEIDENTIFIER in my inner join. All my data was valid and the conversion worked just fine if done outside the join, but it would fail during the join. In fact I was even getting results in the results pane of SQL Server Management Studio but the error was still being thrown. What worked for me was to use TRY_CAST or TRY_CONVERT instead of CAST or CONVERT. My friendly neighborhood DBA suggested this solution because it ensures early execution or something like that.

The most likely explanation for the behavior is a "bad" value stored in the PseudoDateColumn. But you should be able to debug that.
Storing "date" values in an integer column seems like an anti-pattern to me, when SQL Server has a perfectly suitable DATE datatype custom designed for storing date values.
But if you need to work with what you already have, and address the issue without altering the schema definition...
SQL Server has an ISDATE function that you could use test test the validity of a character string. Unfortunately, the check that ISDATE performs isn't as strict as an actual CAST operation, but it does catch the most egregiously "bad" values.
My recommendation:
Instead of converting the integers to dates (the integer values stored in your column...
Cast/convert your literal DATE values to integer, into the same datatype/format as the column (integer), and do the comparison to the raw integer column.
So your query would be something like this:
WHERE T2.PseudoDateColumn >= CONVERT(INT, CONVERT(VARCHAR(8), datexpr1, 112) )
AND T2.PseudoDateColumn < CONVERT(INT, CONVERT(VARCHAR(8), datexpr2, 112) )
In that example, dateexpr1 and dateexpr2 would be replaced with expressions that return a DATE datatype, like the expressions in the original query.
NOTE: This type of range comparison depends on the stored integer values to be canonical. (With year first (four digits), then month (two digits), then day (two digits). In that representation, we are guaranteed that the comparison
(int)a <= (int)b < (int)c
will yield the same results as
(date)a <= (date)b < (date)c
(Equivalent apart from the differences due to "bad" integer values that can't be converted to a valid date... for example, an integer value with invalid "month" and/or "day" parts, e.g. 20161300 (month is thirteen, and day is zero)
An integer comparison that checks for pseudodates the 2016 would "work" for that without throwing an error...
20160101 <= 20161300 < 2017000 -> TRUE
This approach entirely avoids the issue of attempting to convert a "bad" psuedodate integer value into a DATE. But it's always possible to convert a valid DATE into an integer in YYYYMMDD format.
As an added benefit of this pattern, with predicates on the bare PseudoDateColumn column, SQL Server may be able to make effective use of an index that has PseudoDateColumn as the leading column.
If you are dead set on converting PseudoDateColumn to a DATE, you could first perform a conditional test that validates the value, using the ISDATE function. Something like this:
CASE WHEN ISDATE(CONVERT(VARCHAR(20), T2.PsedoDateColumn)) = 1
THEN CONVERT(DATE,CONVERT(VARCHAR(20), T2.PseudoDateColumn),112)
ELSE NULL
END

Related

Comparing dates stored as varchar

I need to compare dates that are stored in my database as varchar against today's date.
Specifically, I need to exclude any records with a date that has passed.
I tried:
SELECT * FROM tblServiceUsersSchedule
WHERE ScheduleEndDate !='' AND ScheduleEndDate < '2015/05/31'
This selected values such as 17/06/2012 and 19/04/2015, which have both already passed, along with 01/06/2015 which hasn't.
I then tried to cast the data with:
SELECT *
FROM tblServiceUsersSchedule
WHERE CAST(ScheduleEndDate as DATETIME) < CAST('05/31/2015' as DATETIME) AND ScheduleEndDate !='' AND ScheduleEndDate is not null
But got the following error:
The coversion of a varchar data type to a datetime data type resulted
in an out-of-range value.
I checked the data behind and none are null, none are blank white space. All are dates in the format of dd/mm/yyyy.
I can't figure out how to compare the varchar date stored with todays date.
Storing date values as varchar is simply wrong.
If possible, you should alter the table to store them as date data type.
You can do it in a few simple steps:
Rename the current columns (I'm guessing ScheduleStartDate is also varchar) to columnName_old. This can be easily done by using sp_rename.
Use alter table to add the columns with the appropriate data type.
Copy the values from the old columns to the new columns using an update statement. Since all of the dates are stored in the same format, you can use convert like this: set ScheduleStartDate = convert(date, NULLIF(ltrim(rtrim(ScheduleStartDate_old)), ''), 103) If your sql server version is 2012 or higher, use try_convert. Note i've used the nullif, ltrim and rtrim to convert values that only contains white spaces to null.
Drop and recreate indexes that is referencing these columns. The simplest way to do this is by right-clicking the index on SSMS and choose script index as -> drop and create.
Use alter table to remove the old columns.
Note: if these columns are being referenced in any other objects on the database you will have to change these objects as well. This includes stored procedures, foreign keys etc`.
If you can't change the data types of the columns, and your sql server version is lower then 2012, you need to use convert like this:
SELECT * FROM tblServiceUsersSchedule
WHERE CONVERT(DATE, NULLIF(ScheduleEndDate, RTRIM(LTRIM('')), 103)
< CAST(GETDATE() As Date);
AND ScheduleEndDate IS NOT NULL
Note that if you have even a single row where the column's data is not in dd/MM/yyyy format this will raise an error.
For sql server versions 2012 or higher, use Try_convert. This function will simply return null if the conversion fails:
SELECT * FROM tblServiceUsersSchedule
WHERE TRY_CONVERT(DATE, NULLIF(ScheduleEndDate, RTRIM(LTRIM('')), 103)
< CAST(GETDATE() As Date);
AND ScheduleEndDate IS NOT NULL
Note: I've used CAST(GETDATE() as Date) to remove the time part of the current date. This means that you will only get records where the ScheduleEndDate is at least one day old. If you want to also get the records where the ScheduleEndDate is today, use <= instead of <.
One final thing: Using functions on columns in the where clause will prevent Sql Server to use any indexing on these columns.
This is yet another reason why you should change your columns to the appropriate data type.
Other than what people have already suggested that you should never store DATETIME as VARCHAR. Always store it in a DATETIME type column; I think you should change your condition in WHERE
ScheduleEndDate < '2015/05/31'
To this, in order to get all dates which hasn't passed yet
ScheduleEndDate >= '2015/05/31'
Your query should look like
SELECT * FROM tblServiceUsersSchedule
WHERE ScheduleEndDate IS NOT NULL
AND ScheduleEndDate >= '2015/05/31'
If you HAVE TO store your dates as varchar (and as per other other answers, this is a poor practice), then using an ISO style format like yyyy-mm-dd should allow textual comparisons without issue.
If your column is a date data type, and you're using SQL 2012 or later then use DATEFROMPARTS (or one of its variants) for date comparison, so
WHERE DateToCompare < DATEFROMPARTS (2019, 12, 31)
rather than
WHERE DateToCompare < '2019-12-31'
SQL handles the latter fine, but the former is more "correct".

SQL Server compare datetime fields

I have a datetime field in my table that I need to use in a where clause.
The fieldname is DatumAanname
In my dataviewgrid it shows for example "16/12/2014 15:57:04"
I looked at the page from microsoft with all the datetime options for a convert, but this format does not seems to be on that page.
The closest format I can see there is "16 12 2014 15:57:04" which should be 113
I tried a query like this
SELECT o.DatumAanname,
(convert(DATETIME, '16 dec 2014 15:57:04:000', 113)),
(convert(DATETIME, '16 dec 2014 15:57:04', 113)),
(convert(DATETIME, o.DatumAanname, 113))
FROM vwOpdracht o
this returns 4 fields which all look exact the same
but when I do
where (convert(datetime, o.DatumAanname, 113)) = (convert(datetime, '16 dec 2014 15:57:04:000', 113))
it returns 0 records
What am I doing wrong here ?
Dates have no formats, they are native types with their own binary storage. Formats and conversions apply only when converting from/to string and they frequently cause problems if incompatible collations and cultures are used.
The best way to avoid problems is simply to remove strings entirely: use date-typed columns in the table and query using parameterized queries. Actually, if the date is stored as text in the table there is no generic and fault-proof way to acoid problems.
Pass strongly typed values instead of strings (eg DateTime in C#, date, datetime,datetime2 in T-SQL). For example:
where o.DatumAanname= #dateParam
will return matching entries if the values match.
If passing date parameters isn't possible, you should only use the unambiguous ISO8691 format, eg:
where o.DatumAanname = '2014-12-16T15:57:04'
If you want to pass date-only values you can use the unseparated format which is also unambiguous
where o.DatumAanname = '20141216'
One thing to note is that date values must match exactly, down to the millisecond. Typically dates are used in range queries where this isn't an issue. In equality queries though, this can be a problem.
You can avoid this by using datetime2(0) as the field type, which ensures that milliseconds aren't stored at all.
Another option is to use a range instead of an equality query, eg:
where o.DatumAanname between '2014-12-16T15:57:04' and '2014-12-16T15:57:04.999'
or
where o.DatumAanname between '2014-12-16T15:57:00' and '2014-12-16T15:57:59'
if you want minute-level precision

Converting Date Strings (in different formats) Into Data Type In SQL Server

I have a column of data in the below string formats
When I copy and paste the above into an Excel sheet and change the column to a date format, all of the above will be converted into the same format correctly. Is there a similar feature in SQL server? I've tried using CAST and CONVERT, but have ran into conversion failed errors.
Follow-up
I found the values causing standard CAST and CONVERT functions to fail. There's some rows with decimal and int values as follows:
I ended up writing a CASE statement to account for the different format variations:
CASE
WHEN [CommissionStart Date] LIKE '%NULL%'
THEN '1/1/1900'
WHEN ISDATE([CommissionStart Date]) = 1
THEN CAST([CommissionStart Date] AS DATE)
ELSE CONVERT(DATETIME, CAST([CommissionStart Date] AS DECIMAL(11, 5)))
END
Cast or Convert should work provided the dates are all in a valid format. If a value can't be parsed as a date, there's no magic fix. You have to find them and deal with them manually. The IsDate() function can help you find them.
select * from MyTable where IsDate(TextDateColumn) = 0
IsDate() will return 0 for numeric values like 4567 or 5456.6826 so if you have a lot of those then you can also check IsNumeric()
select * from MyTable where IsDate(TextDateColumn) = 0 and IsNumeric(TextDateColumn) = 0
That should allow you to identify the unparseable dates that are causing the conversion errors.

How to convert SQL Server's timestamp column to datetime format

As SQL Server returns timestamp like 'Nov 14 2011 03:12:12:947PM', is there some easy way to convert string to date format like 'Y-m-d H:i:s'.
So far I use
date('Y-m-d H:i:s',strtotime('Nov 14 2011 03:12:12:947PM'))
SQL Server's TIMESTAMP datatype has nothing to do with a date and time!
It's just a hexadecimal representation of a consecutive 8 byte integer - it's only good for making sure a row hasn't change since it's been read.
You can read off the hexadecimal integer or if you want a BIGINT. As an example:
SELECT CAST (0x0000000017E30D64 AS BIGINT)
The result is
400756068
In newer versions of SQL Server, it's being called RowVersion - since that's really what it is. See the MSDN docs on ROWVERSION:
Is a data type that exposes automatically generated, unique binary numbers within a database. rowversion is generally used as a mechanism
for version-stamping table rows. The
rowversion data type is just an incrementing number and does not
preserve a date or a time. To record a date or time, use a datetime2
data type.
So you cannot convert a SQL Server TIMESTAMP to a date/time - it's just not a date/time.
But if you're saying timestamp but really you mean a DATETIME column - then you can use any of those valid date formats described in the CAST and CONVERT topic in the MSDN help. Those are defined and supported "out of the box" by SQL Server. Anything else is not supported, e.g. you have to do a lot of manual casting and concatenating (not recommended).
The format you're looking for looks a bit like the ODBC canonical (style = 121):
DECLARE #today DATETIME = SYSDATETIME()
SELECT CONVERT(VARCHAR(50), #today, 121)
gives:
2011-11-14 10:29:00.470
SQL Server 2012 will finally have a FORMAT function to do custom formatting......
The simplest way of doing this is:
SELECT id,name,FROM_UNIXTIME(registration_date) FROM `tbl_registration`;
This gives the date column atleast in a readable format.
Further if you want to change te format click here.
Using cast you can get date from a timestamp field:
SELECT CAST(timestamp_field AS DATE) FROM tbl_name
Works fine, except this message:
Implicit conversion from data type varchar to timestamp is not allowed. Use the CONVERT function to run this query
So yes, TIMESTAMP (RowVersion) is NOT a DATE :)
To be honest, I fidddled around quite some time myself to find a way to convert it to a date.
Best way is to convert it to INT and compare. That's what this type is meant to be.
If you want a date - just add a Datetime column and live happily ever after :)
cheers mac
My coworkers helped me with this:
select CONVERT(VARCHAR(10), <tms_column>, 112), count(*)
from table where <tms_column> > '2012-09-10'
group by CONVERT(VARCHAR(10), <tms_column>, 112);
or
select CONVERT(DATE, <tms_column>, 112), count(*)
from table where <tms_column> > '2012-09-10'
group by CONVERT(DATE, <tms_column>, 112);
"You keep using that word. I do not think it means what you think it means."
— Inigo Montoya
The timestamp has absolutely no relationship to time as marc_s originally said.
declare #Test table (
TestId int identity(1,1) primary key clustered
,Ts timestamp
,CurrentDt datetime default getdate()
,Something varchar(max)
)
insert into #Test (Something)
select name from sys.tables
waitfor delay '00:00:10'
insert into #Test (Something)
select name from sys.tables
select * from #Test
Notice in the output that Ts (hex) increments by one for each record, but the actual time has a gap of 10 seconds. If it were related to time then there would be a gap in the timestamp to correspond with the difference in the time.
for me works:
TO_DATE('19700101', 'yyyymmdd') + (TIME / 24 / 60 / 60)
(oracle DB)
Robert Mauro has the correct comment. For those who know the Sybase origins, datetime was really two separate integers, one for date, one for time, so timestamp aka rowversion could just be considered the raw value captured from the server. Much faster.
After impelemtation of conversion to integer
CONVERT(BIGINT, [timestamp]) as Timestamp
I've got the result like
446701117
446701118
446701119
446701120
446701121
446701122
446701123
446701124
446701125
446701126
Yes, this is not a date and time, It's serial numbers
Why not try FROM_UNIXTIME(unix_timestamp, format)?
I had the same problem with timestamp eg:'29-JUL-20 04.46.42.000000000 PM'. I wanted to turn it into 'yyyy-MM-dd' format. The solution that finally works for me is
SELECT TO_CHAR(mytimestamp, 'YYYY-MM-DD') FROM mytable;
I will assume that you've done a data dump as insert statements, and you (or whoever Googles this) are attempting to figure out the date and time, or translate it for use elsewhere (eg: to convert to MySQL inserts). This is actually easy in any programming language.
Let's work with this:
CAST(0x0000A61300B1F1EB AS DateTime)
This Hex representation is actually two separate data elements... Date and Time. The first four bytes are date, the second four bytes are time.
The date is 0x0000A613
The time is 0x00B1F1EB
Convert both of the segments to integers using the programming language of your choice (it's a direct hex to integer conversion, which is supported in every modern programming language, so, I will not waste space with code that may or may not be the programming language you're working in).
The date of 0x0000A613 becomes 42515
The time of 0x00B1F1EB becomes 11661803
Now, what to do with those integers:
Date
Date is since 01/01/1900, and is represented as days. So, add 42,515 days to 01/01/1900, and your result is 05/27/2016.
Time
Time is a little more complex. Take that INT and do the following to get your time in microseconds since midnight (pseudocode):
TimeINT=Hex2Int(HexTime)
MicrosecondsTime = TimeINT*10000/3
From there, use your language's favorite function calls to translate microseconds (38872676666.7 µs in the example above) into time.
The result would be 10:47:52.677
Some of them actually does covert to a date-time from SQL Server 2008 onwards.
Try the following SQL query and you will see for yourself:
SELECT CAST (0x00009CEF00A25634 AS datetime)
The above will result in 2009-12-30 09:51:03:000 but I have encountered ones that actually don't map to a date-time.
Not sure if I'm missing something here but can't you just convert the timestamp like this:
CONVERT(VARCHAR,CAST(ZEIT AS DATETIME), 110)

Average a time value in SQL Sever 2005

I've got a varchar field in SQL Sever 2005 that's storing a time value in the format "hh:mm"ss.mmmm".
What I really want to do is take the average using the built in aggregate function of those time values. However, this:
SELECT AVG(TimeField) FROM TableWithTimeValues
doesn't work, since (of course) SQL won't average varchars. However, this
SELECT AVG(CAST(TimeField as datetime)) FROM TableWithTimeValues
also doesn't work. As near as I can tell, SQL doesn't know how to convert a value with only time and no date into a datetime field. I've tried a wide variety of things to get SQL to turn that field into a datetime, but so far, no luck.
Can anyone suggest a better way?
SQL Server can convert a time-only portion of a datetime value from string to datetime, however in your example, you have a precision of 4 decimal places. SQL Server 2005 only recognizes 3 places. Therefore, you will need to truncate the right-most character:
create table #TableWithTimeValues
(
TimeField varchar(13) not null
)
insert into #TableWithTimeValues
select '04:00:00.0000'
union all
select '05:00:00.0000'
union all
select '06:00:00.0000'
SELECT CAST(TimeField as datetime) FROM #TableWithTimeValues
--Msg 241, Level 16, State 1, Line 1
--Conversion failed when converting datetime from character string.
SELECT CAST(LEFT(TimeField, 12) as datetime) FROM #TableWithTimeValues
--Success!
This will convert valid values into a DATETIME starting on 1900-01-01. SQL Server calculates dates based on 1 day = 1 (integer). Portions of days are then portions of the value 1 (i.e. noon is 0.5). Because a date was not specified in the conversion, SQL Server assigned the value of 0 days (1900-01-01), which accommodates our need to average the time portion.
To perform an AVG operation on a DATETIME, you must first convert the DATETIME to a decimal value, perform the aggregation, then cast back. For example
SELECT CAST(AVG(CAST(CAST(LEFT(TimeField, 12) as datetime) AS FLOAT)) AS DATETIME) FROM #TableWithTimeValues
--1900-01-01 05:00:00.000
If you need to store this with an extra decimal place, you can convert the DATETIME to a VARCHAR with time portion only and pad the string back to 13 characters:
SELECT CONVERT(VARCHAR, CAST(AVG(CAST(CAST(LEFT(TimeField, 12) as datetime) AS FLOAT)) AS DATETIME), 114) + '0' FROM #TableWithTimeValues
Try this
AVG(CAST(CAST('1900-01-01 ' + TimeField AS DateTime) AS Float))
You really should store those in a datetime column anyway. Just use a consistent date for that part (1/1/1900 is very common). Then you can just call AVG() and not worry about it.
I used Cadaeic's response to get an answer I was looking for, so I thought I should share the code....
I was looking for a query that would average ALL my times together and give me an overall Turn Around Time for all approvals. Below is a nested statement that gives you the AVG TAT for individual id's and and when nested an overall TAT
SELECT
-- calculates overall TAT for ALL Approvals for specified period of time
-- depending on parameters of query
CONVERT(VARCHAR, CAST(AVG(CAST(CAST(LEFT(Tat_mins, 12) as datetime) AS FLOAT)) AS DATETIME), 108) + '0'
from
(
-- tat is for individual approvals
SELECT
dbo.credit_application.decision_status,
dbo.credit_application.application_id,
cast(dbo.credit_application.data_entry_complete as date) as'Data Entry Date',
cast(dbo.credit_application.decision_date as DATE) as 'Decision Date',
avg(datediff(minute, dbo.credit_application.data_entry_complete, dbo.credit_application.decision_date)) as 'TAT Minutes',
convert (char(5), DateAdd(minute, Datediff(minute,dbo.credit_application.data_entry_complete, dbo.credit_application.decision_date),'00:00:00'),108) as 'TAT_Mins'
FROM dbo.credit_application
where Decision_status not in ('P','N')
group by dbo.credit_application.decision_status,
dbo.credit_application.data_entry_complete,
dbo.credit_application.decision_date
--dbo.credit_application.application_id
)bb
How do you think to average on datetime?
I guess that you need to GROUP BY some period (Hour?), and display Count(*)?
SQL Server stores datetime data as 2 4-byte integers, hence a datetime take 8 bytes. The first is days since the base date and the second is milliseconds since midnight.
You can convert a datetime value to an integer and perform mathematical operations, but the convert only returns the "days" portion of the datetime value e.g. select convert(int,getdate()). It is more difficult to return the "time" portion as an integer.
Is using SQL Server 2008 an option for you? That version has a new dedicated time data type.
Thanks, Andy.
I'd work out the difference between all of the dates and an arbitrary point (01/01/1900), average it and then add it back on to the arbitrary point.

Resources