I am finding difficulty when it comes to retrieving date values in hive. My query is
create external table test1(DISPLAYSCALE int, CREATED_DATE date, LAST_EDITED_DATE date)
ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde' STORED AS INPUTFORMAT 'com.esri.json.hadoop.UnenclosedJsonInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
When I try to use the select * from test1 limit5 I get this error;
Failed with exception
java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.DateWritable cannot be cast to
org.apache.hadoop.io.Text
As per the json the datatype for CREATED_DATE and CREATED_DATE are esriFieldTypeDate and the values are in this format say 2013-11-20 09:39:25.000001.
So i used the date datatype while creating the table, copied it to HDFS using the unenclosed json and used the select * query to retrieve the columns, but I get the above error. To get the values we are creating the same table with string data type respectively instead of date and we are able to get the values .
Can you suggest a solution for this problem. This question may seem silly but I am pretty new to programming.
Date data type supports only format YYYY-MM-DD
Timestamps or VARCHAR data type you can be used rather than date data type.
Date data types do not exist in Hive. In fact the dates are treated as strings in Hive. Refer the below post
http://www.folkstalk.com/2011/11/date-functions-in-hive.html
EsriJsonSerDe (and GeoJsonSerDe) support for DATE and TIMETAMP type columns is added on Spatial-Framework-for-Hadoop master in git.
Alternately, you can try using org.openx.data.jsonserde.JsonSerDe or org.apache.hive.hcatalog.data.JsonSerDe (instead of EsriJsonSerDe - and with column type string rather than binary) together with UnenclosedEsriJsonInputFormat.
[disclosure: Spatial-Framework-for-Hadoop collaborator]
Related
I'm loading many files into a SQL SERVER database. I have one flat file that has a Date Column coming in as string[DT_STR].
I have TWO "date fields" in my database. One is varchar, one is datetime.
Converting the datetime column is no issue, I just use Data Conversion/Derived Column if necessary. However, this varchar column is giving me trouble. Our database values for this column should be in yyyymmdd format. However, on this single file the format of the dates change.
Normally I'd do a SUBSTRING(...) expression here, but the difficulty is that the format of these dates change. some examples of values could be
08/16/2017
8/16/2017
08/6/2017
08/06/2017
10/6/2017
10/06/2017
This makes the challenge harder. I tried LEN([DATE]) == NUM_HERE ? do_THING : OTHER_CALC, but this approach failed because the length of 10/6/2017 is the same as 8/06/2017 which will give me the wrong result. Does anyone have a good workaround for this?
Perhaps a simple convert to date and then into the final format. If 2012+, use try_convert() to trap any bogus dates.
Example
Declare #YourTable Table ([SomeCol] varchar(50))
Insert Into #YourTable Values
('08/16/2017')
,('8/16/2017')
,('08/6/2017')
,('08/06/2017')
,('10/6/2017')
,('10/06/2017')
Select *
,Formatted = convert(varchar(8),convert(Date,SomeCol),112)
from #YourTable
Returns
SomeCol Formatted
08/16/2017 20170816
8/16/2017 20170816
08/6/2017 20170806
08/06/2017 20170806
10/6/2017 20171006
10/06/2017 20171006
Convert the varchar data to datetime and convert that to a formatted string
SELECT CONVERT(varchar,(CONVERT(datetime, '8/6/2017')),112)
I have a datetimeoffset column DateEntry in my SQL Server table. When I want to convert it to a timestamp format with this query :
SELECT CAST(Table1.[DateEntry] AS timestamp)
FROM Table1
I get the following error :
Error : 529- Explicit conversion from data type datetimeoffset to
timestamp is not allowed.
TIMESTAMP in SQL Server has absolutely nothing to do with a date and time, therefore you cannot convert an existing date&time into a TIMESTAMP.
TIMESTAMP or more recently called ROWVERSION is really just a binary counter that SQL Server updates internally whenever row has been modified. You cannot set a TIMESTAMP column yourself, you can just read it out. It is used almost exclusively for optimistic concurrency checks - checking to see whether a row has been modified since it's been read, before updating it.
According to MSDN:
The timestamp data type is just an incrementing number and does not
preserve a date or a time. To record a date or time, use a datetime
data type.
If your are absolutely sure, you can use indirect conversion:
DECLARE #dto datetimeoffset = '2016-01-01 12:30:56.45678'
SELECT CONVERT(timestamp, CONVERT(varbinary(12), #dto))
See also #marc_s's answer.
Try the following script if this this is what you are trying your side
SELECT CAST(CAST(Table1.[DateEntry] AS datetime) as timestamp) FROM Table1
I had to restore a table that contained a datetime column.
I used bulk insert to insert the data from a CSV file. However the import couldn't insert the datetime values because SQL server saw it as a different format.
I ended up modifying the table, removing the datetime data type and replacing it with a varchar.
The issue is the data got converted from this format: 7/15/2015 3:41:57 PM to something like this: 47:47.0
Is there a way I can convert these values back or is the data lost?
As #ChrisSteele mentioned, your data is hosed. It likely got this way by Excel's cool feature to convert datetime strings to integers. Try re-saving the original file in notepad, or changing the format of the column from Date/Datetime to Text if you're using Excel.
So im having a rather strange problem, I have a Column (lets say Column A) in excel that has data that looks like this:
4/11/2015 10:14
I have a bunch of other columns, but anyways in my SQL Insert statement within excel, the data (when copying out) looks like this:
42105.4561921296
The ="INSERT INTO TABLE VALUES ('"&A1&"', Etc....)" is in the data format of "general" and the Date column is in the format of "Custom" where there is a M/DD/YYYY MM/HH type within.
The SQL Column is of the data type DATETIME, so it of course doesn't accept the weird number it gets.
Any ideas? changing the format of the "SQL INSERT" column doesn't change the results.
You are right - Excel formats only changes the way the numbers are displayed, not the underlying value of the cell. In this case, the value of the cell is an excel date-time value, which is the # of days since 1/1/1900, with decimals for the time.
I'd recommend using Excel's TEXT function to convert Excel's numeric date-time value to a text string that can be inserted into SQL:
Instead of:
INSERT INTO TABLE VALUES ('"&A1&"', Etc....)"
Try:
INSERT INTO TABLE VALUES ('"&TEXT(A1,"YYYY-MM-DD HH:MM")&"', Etc...)"
The best format to insert a date time into a datetime column is to present the date and time as YYYY-MM-DD Minute:Second or 2015-04-15 12:52
so to insert a datetime from Excel you can use this set of Excel functions:
(where A1 contains the datetime to be saved)
=YEAR(A1)&"-"&RIGHT("00"&MONTH(A1),2)&"-"&RIGHT("00"&DAY(A1),2)&" "&RIGHT("00"&HOUR(A1),2)&":"&RIGHT("00"&MINUTE(A1),2)&":"&RIGHT("00"&SECOND(A1),2)
I'm creating a table B from an exisitng table A. In the table A I have a column ValDate which is varchar and contains Date. When I try to create the table B I have a function used in the query as given below and I get a conversion error. Table A contains null values as well.
MS Access:
((DateDiff("d",Date(),format(Replace(Replace([Table A].ValDate,".","/"),"00/00/0000","00:00:00"),"dd/mm/yyyy")))>0)).
Tables were in MS Access and are being migrated to SQL Server 2012.
SQL Server:
((DATEDIFF(day,FORMAT( GETDATE(), 'dd-MM-yyyy', 'en-US' ),FORMAT( ValDate, 'dd-MM-yyyy', 'en-US' ))>0))
or
((DateDiff(day,GETDATE(),Format(Replace(Replace([TableA].[ValidFrom],'.','/'),'00/00/0000','00:00:00'),'dd/mm/yyyy')))
I tried converting the date using several approachs like Convert , Format and Cast but I end up getting error as below.
Msg 8116, Level 16, State 1, Line 1
Argument data type date is invalid for argument 1 of isdate function.
I would really appreciate someone telling me what I'm missing here.
since you have date data in a string field is very likely you have some value that is not valid against your expected date format.
copy the data in a sql server table and then perform check and validation of the content of the string field.
have a look to the function try_convert that can be helpful when checking the content of the string field containing the date values.
when bad data is ruled out you can apply again your formula with (hopefully) a different result.
a better solution would be to create a separate field with appropriate datatype to store date values converted from the string field and apply your logic to that field.