I am using LogStash 7.3.2 to fetch incremental data from SQL Server using this query:
select * from mytable where lastupdatetimestamp > :sql_last_value
I also have specified last_run_metadata_path in logstash config file.
It works fine but sometimes it is throwing an exception:-
Exception when executing JDBC query {:exception=>#
transition (daylight savings time 'gap'): 1942-09-01T00:00:00.000 (Asia/Kolkata)>}
Why am I getting this exception and due to this exception it does not save last timestamp value and again it fetches duplicate records from SQL Server.
Any help regarding this would be highly appreciated.
As stated here
In Logstash if a datetime string has no TZ part it is assumed to be a UTC time.
If you know that the datetime is a local time string then you need to tell the date filter which timezone the date was generated in. You may use one of the Joda timezones,
e.g. America/New_York, America/Detroit or EST5EDT - these are merely labels that point to Timezone Java code that know what changes in clocks occurred in all timezones for quite some time into the past.
See this page for info on how the updates are followed. http://www.oracle.com/technetwork/java/javase/tzdata-versions-138805.html 100
Once the datetime string is converted to a Timestamp object (by the date filter) it is considered UTC time.
Related
As part of my job duties, I'm responsible for extracting data from our vendor's Oracle 11g database, and loading it into our SQL Server 2016 database. I've been doing this successfully with SSIS and the Attunity Oracle connectors.
Today I was informed that there was a new column added to the existing Invoices table on the Oracle side. There was already a DATE column called Order Date, which contains valid date values with zero'd times, like 2017-12-25 00:00:00.
The new column is called Order Date Time and is also a DATE column. When I opened up the SSIS package and pulled up the Oracle source in my DFT, I previewed the data and found the values in Order Date Time to be 2432-82-75 50:08:01. I tried converting the column with CAST and all the TO_* functions, but the conversions either failed outright, or returned a string of zeros.
TO_CHAR("Order Date Time", 'YYYYMMDDHH24MISS')
yields 00000000000000
After a bit of Googling for "Oracle date value invalid", I'm now thinking that these DATE values are actually corrupted. Am I missing anything here? Is there some sort of special Oracle-specific technique for storing time values in a DATE column that I may not be aware of?
(And yes, it does bother me quite a bit that our vendor added another DATE column instead of just using the time portion of the existing Order Date column.)
Unfortunately, Oracle database engine allows inserting invalid date values, which leads to many problems especially when importing data to others database engines such as SQL Server.
To handle this issue, you have to implement the logic that fits your needs, as example:
You can exclude these records from you queries by filtering on acceptable date ranges: (WHERE date between ...)
You can Update records with invalid values by replacing with NULL
You can use a CASE statement in your query to replace values with NULL
I faced this issue one time while importing data to SQL Server from an Oracle data source, there was unacceptable date values, i decided to update all records where date are invalid and replace with NULL values before starting the import process.
There are many links related to this issue:
Detecting invalid values in the DB
How to identify invalid (corrupted) values stored in Oracle DATE columns
Corrupt date fields causing query failure in Oracle
Invalid Date in DATE Column SQLPlus VS SQLDeveloper
Ask Tom - date validation in oracle
Dealing with invalid dates
Error: Invalid date format
DB Connect; Oracle DB date field data is corrupt
I have
an SQL Server 2012, where datetime fields are stored in UTC time
an Excel File, that is integrated via ADO DB command objects to the SQL server
a function to check the current system settings, which allows to see the current system's difference to UTC
I need to display dates/times in Excel in local time (varies).
Until SQL Server 2016 (which we can't currently upgrade to), there is no Timezone support built-in to T-SQL.
There are various options that are playing around in my head, but I can't decide which one is worth pursuing:
- transmit the current time difference as a parameter to the Stored Procedure, and amend all date fields to apply that difference
- retrieve the UTC times, and return the field(s) as a formula (so instead of return field, field, created, modified I might return field, field, =fnUtcToLocal(#created#)... and so on
- retrieve UTC times, and use an adjacent formula (with QueryTable.FillAdjacentFormulas)
Regardless of the option, I've still got a problem left: When, for example, checking the system difference now in Excel when the user is in Eastern Standard Time, it is -5 hours. When my created date falls into the EDT time, the correction would still be -5 instead of -4 hours.
Does anyone have a smart idea on how I should approach this? Please let me know if any additonal detail is required.
I am working with Silverlight and I am getting a problem. In my Database I have stored some dates with a format like this yyyy/mm/dd 00:00:00, which means that I store only the year, month and day, getting the time to 00:00:00.
When the client performs an action and sends to the server I get the DateTime.Today which will keep the format of my database date, but when it is sendind I get yyyy/mm/dd 22:00:00, so when my server side function gets the date, it will return no values from the database.
How can I fix this to get the correct datetime?
Thank you
Use UTC times to make sure you don't run into timezone issues.
You can see if the DateTime is UTC or local based by checking the Kind property, and you can get the current UTC time by DateTime.UtcNow.
DateTime structure is very prone to DST, timezones and cultures when serialising it.
are you serialising it at client side before push it ? what is the difference between client and server timezones ?
I would suggest that you try and consume DateTime.UtcNow and then serialise the data. i prefer to serialise using Invariant culture
HTH
In addition to storing UTC time in the database, you can hard set the time to 12:00:00 for all date values by using the Date property of the DateTime class.
DateTime.UtcNow.Date
You can present the date to the user in their local timezone using the ToLocalTime method:
DateTime.ToLocalTime().Date
I'm sure every database engine has a similar function, but the SQL Server function to get UTC date is (surprisingly enough):
GETUTCDATE()
I hope this helps.
I transfer data from one SQL Server to another using SELECT * INTO .... statement (in Management Studio). Both databases have different timezone settings.
Is it possible to adjust all date/time fields to different timezone automatically? Maybe some query parameters or connection settings?
Since you are on SQL Server 2008, you could use the built-in functionality of the DATETIMEOFFSET datatype, and the SWITCHOFFSET function.
You could do something like:
INSERT INTO dbo.YourTargetTable( ...... )
SELECT
SWITCHOFFSET(TODATETIMEOFFSET(YourSourceTime, '+04:00'), '-09:00'),
.......
TODATETIMEOFFSET converts your "regular" date without any timezone information into a DATETIMEOFFSET type with a time zone (of your source location), and then you can apply a SWITCHOFFSET call to that value to "switch" your date/time to your target location's time zone.
This can be done in the scope of the INSERT....SELECT statement - no separate "row-by-row" updating necessary.
I think this will be helpful, please follow the link:
Effectively Converting dates between UTC and Local (ie. PST) time in SQL 2005
Shortly, you do need to build time zone and DST tables to get some help with time zones.
Thanks
Is there any way to prevent Silverlight/RIA Services from converting a datetime object on the wire?
Right now, I see datetimes set to 'Local' or 'Unspecified' being converted to the local time of the server when they are sent over the wire.
For example:
The client and server are both in UTC -04:00:00.
I use DateTime.Today (kind is either Local or Unspecified, it doesn't make a difference) on the Silverlight client. I see 23/08/2010 00:00:00.
I submit my changes and watch the data go over the wire. The field is expressed at 23/08/2010 00:00:00 (-04:00:00).
Great. Now I change my client to be in UTC +12:00:00
I use DateTime.Today on the client and now I see 24/08/2010 00:00:00.
HOWEVER - I submit my changes and watch the data again. Now I see 23/08/2010 08:00:00 (-04:00:00).
So it is apparent that the serializer is converting to the local time of the server, which I do not want. The value I want in the DB is 24/08/2010 00:00:00.
Using UTC is not a great option for this field as the database is part of our legacy application and the column contains invariant dates at this time. I don't want to start inserting UTC datetimes alongside the existing data.
Any ideas?
Thanks in advance
If I create the DateTime like this, it works:
new DateTime(DateTime.Now.Ticks, DateTimeKind.Unspecified).Date;