I have a timestamp column having values like the following in my database:
2017-01-01 00:00:58.538-05
2017-01-01 00:16:58.54-05
The data type is varchar since datetime, datetime2 formats did not work. I need to convert this column to datetime format now. I just discovered about datetimeoffset. :(
However, the conversion still does not work as is, and I get an error. datetimeoffset wants to see at least -05:0 not only -05.
DECLARE #datevar datetimeoffset = '2017-01-01 00:00:58.538-05';
SELECT #datevar
Msg 241, Level 16, State 1, Line 3
Conversion failed when converting date and/or time from character string.
What can be to solve this? Worst case scenario I tend use python to preprocess each file to add :0 at the end of each value, and then save the file, and then bulk upload every file in the folder at the end. However, each file has ~30 million rows, and there are 365 of them per year. So I really do not want to do that.
Again, the data is already uploaded, an easy and FAST way to do this would be appreciated. There are about 8 billion rows in the table now, so I do not know whether this can be done using SQL Server Management Studio.
You can do the :00 appending in sql itself
SELECT Cast(dates + ':00' AS DATETIMEOFFSET),
dates
FROM (VALUES ('2017-01-01 00:00:58.538-05'),
('2017-01-01 00:16:58.54-05')) tc (dates)
considering none of your dates has minutes part of timezone. If some dates has minutes part then it needs to be handled differently
In your table it should be something like
SELECT Cast(datecolumn + ':00' AS DATETIMEOFFSET),
datecolumn
FROM Yourtable
to update the table
update t set yourcol = Cast(yourcol + ':00' AS DATETIMEOFFSET)
FROM Yourtable
Then alter the table datatype to datetimeoffset and make sure you upload data with time part in offset
Related
I need to get the year and month of a datetime column. For this I have come up with the following methods. Please note am using SQL Server 2019.
CONVERT(INT, FORMAT(A.dtmBilledDate, 'yyyyMM'))
CONVERT(INT, SUBSTRING(CONVERT(VARCHAR, A.dtmBilledDate, 112), 1, 6))
YEAR(A.dtmBilledDate) * 100 + MONTH(A.dtmBilledDate)
There could be some other methods also. I need the net output datatype to be integer. Now out of these formats which is the fastest way to retrieve a dataset that contains like 500,000-600,000 records.
Whenever I run the time keeps varying, some times I get it in 4-8 sec sometime it goes to 16 sec. By the way I ran in a production server.
We receive a csv file that has a column in this date format -- Wed Oct 14 08:00:00 CDT 2020, along with a column that has a count for each date/time
I am using an SSIS package to grab the file and import this data into a sql table, then I can format it the way I need to and then actually export the data in the format needed.
If there is a way to do this all within one SSIS package I am all ears but currently I am working on just getting the data into SQL and converted to the right format so that I can export it.
I need to get that file and convert that date format and split it up into two separate columns
One column will be just the date in this format 2020-10-14 00:00:00.000
One column will be just the time in this format 08:00:00.0000000
Updated to change the dates to match so it's not as confusing and also the error I am receiving when running the suggested code below.
Image of Error I'm recieving
Image of table with the data I am trying to convert
Image of table attributes
Screenshot of my screen when running a select * from the table I am pulling the data that I need converted
Screenshot of the error I receive when running the query by Aaron.
If this is the format it will always be in, and timezone is irrelevant, you can first try to convert it to a datetime, then you can extract the parts from that.
SET LANGUAGE us_english; -- important because not all languages understand "Oct"
;WITH src AS
(
SELECT dt = TRY_CONVERT(datetime, RIGHT(OpenedDateTime ,4)
+ SUBSTRING(OpenedDatetime, 4, 16))
--, other columns...
FROM [dbo].[VIRTUALROSTERIMPORT_Res_Import]
)
SELECT OpenedDateTime = CONVERT(datetime, CONVERT(date, dt)),
OnHour = CONVERT(time, dt)
--, other columns...
FROM src;
Results:
OpenedDateTime OnHour
-------------- ----------------
2020-10-14 08:00:00.0000000
If you need to shift from one timezone to another timezone, that's a different problem.
I was just showing the date formats, don't look so into the actual date examples I used. The time zone is irrelevant I just need the formats changed.
When I used The code Aaron suggested I got a conversion error: I'm assuming its because the columns are varchar in the table, but I cant get the dates to load as date formats bc SSIS keeps giving me truncated errors-- so I have to load it as varchar.
Below is the code I was running, I tweaked it to use the column and table names I am using.
SET LANGUAGE us_english; -- important because not all languages understand "Oct"
DECLARE #foo varchar(36) = 'Wed Oct 14 08:00:00 CDT 2020';
;WITH src(d) AS
(
SELECT TRY_CONVERT(datetime, RIGHT(#foo,4) + SUBSTRING(#foo, 4, 16))
)
SELECT OpenedDateTime = CONVERT(datetime, CONVERT(date, OpenedDateTime)),
onhour = CONVERT(time, OpenedDateTime)
FROM [dbo].[VIRTUALROSTERIMPORT_Res_Import];
Have a table I'm logging information from a .NET program into.
The VB.NET app explicity dictates the format of the DATETIME string like below
responsedt = Date.Now.ToString("yyyy-MM-dd HH:mm:ss.fff")
I then pass this into an INSERT statement that updates my table, however even though the entire setup of the SQL Server is en-GB (British English) the DateTime has gone in the following format:
2019-09-05 19:09:34.823
This was done yesterday so actually should be
2019-05-09 19:09:34.823
The day and month should be switched around, I have tried performing an update on the table post process to get it to update using the following code
FORMAT (xa.daterequested, 'yyyy-dd-MM HH:MM:ss.fff', 'en-gb')
How while this works in a SELECT statement it doesn't seem to work when I do the UPDATE statement.
It is not ideal to have to update all the records dates after the initial INSERT so a solution to either the .NET side of the issue or the SQL would be appreciated as its pickling my head.
You have 2 options to prevent the error from happening again:
Keep dates as date/time data types instead of converting them to strings.
Use formats that are not language or settings dependent. In SQL Server these would be YYYYMMDD hh:mm:ss.msss OR YYYY-MM-DDThh:mm:ss.mss (notice the T between date and time)
To correct the dates already inserted you could use the format codes in the CONVERT function.
UPDATE t SET
daterequested = STUFF( STUFF( StringDate, 5, 0, SUBSTRING(StringDate,7,2)), 9, 2, '')
FROM YourTable t
CROSS APPLY( SELECT CONVERT( varchar(25), '20190905 19:09:34.823', 121) AS StringDate) AS x;
As SQL Server returns timestamp like 'Nov 14 2011 03:12:12:947PM', is there some easy way to convert string to date format like 'Y-m-d H:i:s'.
So far I use
date('Y-m-d H:i:s',strtotime('Nov 14 2011 03:12:12:947PM'))
SQL Server's TIMESTAMP datatype has nothing to do with a date and time!
It's just a hexadecimal representation of a consecutive 8 byte integer - it's only good for making sure a row hasn't change since it's been read.
You can read off the hexadecimal integer or if you want a BIGINT. As an example:
SELECT CAST (0x0000000017E30D64 AS BIGINT)
The result is
400756068
In newer versions of SQL Server, it's being called RowVersion - since that's really what it is. See the MSDN docs on ROWVERSION:
Is a data type that exposes automatically generated, unique binary numbers within a database. rowversion is generally used as a mechanism
for version-stamping table rows. The
rowversion data type is just an incrementing number and does not
preserve a date or a time. To record a date or time, use a datetime2
data type.
So you cannot convert a SQL Server TIMESTAMP to a date/time - it's just not a date/time.
But if you're saying timestamp but really you mean a DATETIME column - then you can use any of those valid date formats described in the CAST and CONVERT topic in the MSDN help. Those are defined and supported "out of the box" by SQL Server. Anything else is not supported, e.g. you have to do a lot of manual casting and concatenating (not recommended).
The format you're looking for looks a bit like the ODBC canonical (style = 121):
DECLARE #today DATETIME = SYSDATETIME()
SELECT CONVERT(VARCHAR(50), #today, 121)
gives:
2011-11-14 10:29:00.470
SQL Server 2012 will finally have a FORMAT function to do custom formatting......
The simplest way of doing this is:
SELECT id,name,FROM_UNIXTIME(registration_date) FROM `tbl_registration`;
This gives the date column atleast in a readable format.
Further if you want to change te format click here.
Using cast you can get date from a timestamp field:
SELECT CAST(timestamp_field AS DATE) FROM tbl_name
Works fine, except this message:
Implicit conversion from data type varchar to timestamp is not allowed. Use the CONVERT function to run this query
So yes, TIMESTAMP (RowVersion) is NOT a DATE :)
To be honest, I fidddled around quite some time myself to find a way to convert it to a date.
Best way is to convert it to INT and compare. That's what this type is meant to be.
If you want a date - just add a Datetime column and live happily ever after :)
cheers mac
My coworkers helped me with this:
select CONVERT(VARCHAR(10), <tms_column>, 112), count(*)
from table where <tms_column> > '2012-09-10'
group by CONVERT(VARCHAR(10), <tms_column>, 112);
or
select CONVERT(DATE, <tms_column>, 112), count(*)
from table where <tms_column> > '2012-09-10'
group by CONVERT(DATE, <tms_column>, 112);
"You keep using that word. I do not think it means what you think it means."
— Inigo Montoya
The timestamp has absolutely no relationship to time as marc_s originally said.
declare #Test table (
TestId int identity(1,1) primary key clustered
,Ts timestamp
,CurrentDt datetime default getdate()
,Something varchar(max)
)
insert into #Test (Something)
select name from sys.tables
waitfor delay '00:00:10'
insert into #Test (Something)
select name from sys.tables
select * from #Test
Notice in the output that Ts (hex) increments by one for each record, but the actual time has a gap of 10 seconds. If it were related to time then there would be a gap in the timestamp to correspond with the difference in the time.
for me works:
TO_DATE('19700101', 'yyyymmdd') + (TIME / 24 / 60 / 60)
(oracle DB)
Robert Mauro has the correct comment. For those who know the Sybase origins, datetime was really two separate integers, one for date, one for time, so timestamp aka rowversion could just be considered the raw value captured from the server. Much faster.
After impelemtation of conversion to integer
CONVERT(BIGINT, [timestamp]) as Timestamp
I've got the result like
446701117
446701118
446701119
446701120
446701121
446701122
446701123
446701124
446701125
446701126
Yes, this is not a date and time, It's serial numbers
Why not try FROM_UNIXTIME(unix_timestamp, format)?
I had the same problem with timestamp eg:'29-JUL-20 04.46.42.000000000 PM'. I wanted to turn it into 'yyyy-MM-dd' format. The solution that finally works for me is
SELECT TO_CHAR(mytimestamp, 'YYYY-MM-DD') FROM mytable;
I will assume that you've done a data dump as insert statements, and you (or whoever Googles this) are attempting to figure out the date and time, or translate it for use elsewhere (eg: to convert to MySQL inserts). This is actually easy in any programming language.
Let's work with this:
CAST(0x0000A61300B1F1EB AS DateTime)
This Hex representation is actually two separate data elements... Date and Time. The first four bytes are date, the second four bytes are time.
The date is 0x0000A613
The time is 0x00B1F1EB
Convert both of the segments to integers using the programming language of your choice (it's a direct hex to integer conversion, which is supported in every modern programming language, so, I will not waste space with code that may or may not be the programming language you're working in).
The date of 0x0000A613 becomes 42515
The time of 0x00B1F1EB becomes 11661803
Now, what to do with those integers:
Date
Date is since 01/01/1900, and is represented as days. So, add 42,515 days to 01/01/1900, and your result is 05/27/2016.
Time
Time is a little more complex. Take that INT and do the following to get your time in microseconds since midnight (pseudocode):
TimeINT=Hex2Int(HexTime)
MicrosecondsTime = TimeINT*10000/3
From there, use your language's favorite function calls to translate microseconds (38872676666.7 µs in the example above) into time.
The result would be 10:47:52.677
Some of them actually does covert to a date-time from SQL Server 2008 onwards.
Try the following SQL query and you will see for yourself:
SELECT CAST (0x00009CEF00A25634 AS datetime)
The above will result in 2009-12-30 09:51:03:000 but I have encountered ones that actually don't map to a date-time.
Not sure if I'm missing something here but can't you just convert the timestamp like this:
CONVERT(VARCHAR,CAST(ZEIT AS DATETIME), 110)
I've got a varchar field in SQL Sever 2005 that's storing a time value in the format "hh:mm"ss.mmmm".
What I really want to do is take the average using the built in aggregate function of those time values. However, this:
SELECT AVG(TimeField) FROM TableWithTimeValues
doesn't work, since (of course) SQL won't average varchars. However, this
SELECT AVG(CAST(TimeField as datetime)) FROM TableWithTimeValues
also doesn't work. As near as I can tell, SQL doesn't know how to convert a value with only time and no date into a datetime field. I've tried a wide variety of things to get SQL to turn that field into a datetime, but so far, no luck.
Can anyone suggest a better way?
SQL Server can convert a time-only portion of a datetime value from string to datetime, however in your example, you have a precision of 4 decimal places. SQL Server 2005 only recognizes 3 places. Therefore, you will need to truncate the right-most character:
create table #TableWithTimeValues
(
TimeField varchar(13) not null
)
insert into #TableWithTimeValues
select '04:00:00.0000'
union all
select '05:00:00.0000'
union all
select '06:00:00.0000'
SELECT CAST(TimeField as datetime) FROM #TableWithTimeValues
--Msg 241, Level 16, State 1, Line 1
--Conversion failed when converting datetime from character string.
SELECT CAST(LEFT(TimeField, 12) as datetime) FROM #TableWithTimeValues
--Success!
This will convert valid values into a DATETIME starting on 1900-01-01. SQL Server calculates dates based on 1 day = 1 (integer). Portions of days are then portions of the value 1 (i.e. noon is 0.5). Because a date was not specified in the conversion, SQL Server assigned the value of 0 days (1900-01-01), which accommodates our need to average the time portion.
To perform an AVG operation on a DATETIME, you must first convert the DATETIME to a decimal value, perform the aggregation, then cast back. For example
SELECT CAST(AVG(CAST(CAST(LEFT(TimeField, 12) as datetime) AS FLOAT)) AS DATETIME) FROM #TableWithTimeValues
--1900-01-01 05:00:00.000
If you need to store this with an extra decimal place, you can convert the DATETIME to a VARCHAR with time portion only and pad the string back to 13 characters:
SELECT CONVERT(VARCHAR, CAST(AVG(CAST(CAST(LEFT(TimeField, 12) as datetime) AS FLOAT)) AS DATETIME), 114) + '0' FROM #TableWithTimeValues
Try this
AVG(CAST(CAST('1900-01-01 ' + TimeField AS DateTime) AS Float))
You really should store those in a datetime column anyway. Just use a consistent date for that part (1/1/1900 is very common). Then you can just call AVG() and not worry about it.
I used Cadaeic's response to get an answer I was looking for, so I thought I should share the code....
I was looking for a query that would average ALL my times together and give me an overall Turn Around Time for all approvals. Below is a nested statement that gives you the AVG TAT for individual id's and and when nested an overall TAT
SELECT
-- calculates overall TAT for ALL Approvals for specified period of time
-- depending on parameters of query
CONVERT(VARCHAR, CAST(AVG(CAST(CAST(LEFT(Tat_mins, 12) as datetime) AS FLOAT)) AS DATETIME), 108) + '0'
from
(
-- tat is for individual approvals
SELECT
dbo.credit_application.decision_status,
dbo.credit_application.application_id,
cast(dbo.credit_application.data_entry_complete as date) as'Data Entry Date',
cast(dbo.credit_application.decision_date as DATE) as 'Decision Date',
avg(datediff(minute, dbo.credit_application.data_entry_complete, dbo.credit_application.decision_date)) as 'TAT Minutes',
convert (char(5), DateAdd(minute, Datediff(minute,dbo.credit_application.data_entry_complete, dbo.credit_application.decision_date),'00:00:00'),108) as 'TAT_Mins'
FROM dbo.credit_application
where Decision_status not in ('P','N')
group by dbo.credit_application.decision_status,
dbo.credit_application.data_entry_complete,
dbo.credit_application.decision_date
--dbo.credit_application.application_id
)bb
How do you think to average on datetime?
I guess that you need to GROUP BY some period (Hour?), and display Count(*)?
SQL Server stores datetime data as 2 4-byte integers, hence a datetime take 8 bytes. The first is days since the base date and the second is milliseconds since midnight.
You can convert a datetime value to an integer and perform mathematical operations, but the convert only returns the "days" portion of the datetime value e.g. select convert(int,getdate()). It is more difficult to return the "time" portion as an integer.
Is using SQL Server 2008 an option for you? That version has a new dedicated time data type.
Thanks, Andy.
I'd work out the difference between all of the dates and an arbitrary point (01/01/1900), average it and then add it back on to the arbitrary point.