I have a cronjob that looks at load previous day data sourcing another table that gets refreshed on a daily basis. I am looking to update the job to source from origination table that holds entire year of data. However I am just looking to capture and load previous 1 day of data from the origination table. The origination table has a date updated field which is in timestamptz format (ex - 2022-08-01 20:20:20.736+00). Any recommendation on what function to place where the job picks up:
last_updated from >= 2022-08-01 10:00:00 and last_updated from <= 2022-08-02 10:00:00. Assuming I am running this on 2022-08-02 11:00:00.
Thanks,
Related
I am trying to create a calendar using Power Query functions and for that I used below syntax in blank query:
Source= Duration.TotalDays(DateTime.LocalNow() - #datetime(2014,01,01,00,00,00)) * 24
Date= List.DateTimes(#datetime(2014,01,01,00,00,00), Source ,#duration(0,1,0,0))
Then I convert to a table and apply query.
Connect dimension date table to date column in fact table.
The error occurs when I’m trying to mark table as date table:
‘The date column can only gave one timestamp per day. The date column
can’t have gaps in dates’
What I have done wrong?
As the error message says:
The date column can only have one timestamp per day.
While you are trying to add 24, one for each hour. See the requirements for setting a table as a date table:
if it is a Date/Time data type, it has the same timestamp across each value
i.e. you can have only one value for each date, and if it is not a date, but datetime value, all time values should be the same.
I'm creating a query that runs daily, and pushes data to the following tables:
Daily Dashboard
Weekly Dashboard
Monthly Dashboard
Yearly Dashboard
The daily file pushes data to the other 3 tables and is dropped daily. What I'm trying to figure out is how to put a trigger/command into the drop table command for the weekly and monthly tables.
The Weekly File Needs to Drop every Monday
And
The Monthly File needs to Drop on the 1st Day of each month
I'm assuming I'd need to have some sort of declaration like
DECLARE #CurrentMonth as date SET #CurrentMonth = month(GetDate())
DECLARE #CurrentWeek as date SET #CurrentWeek = week(GetDate())
And then compare the current week/month against the date the table was created
eg:
IF CurrentWeek > CreatedWeek - Drop Table
IF CurrentMonth > CreatedMonth - Drop Table
Does anyone know if thats achievable, or if there is a better way of going about this?
Any help/advice is appreciated
I'm trying to use table decorators ranges in one of the AppEngine RequestLog table in BigQuery. According to documentation log entries are objects of type LogEntry https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry.
There are two columns timestamp and receiveTimestamp. The first column description is "The time the event described by the log entry occurred" and for the second one "The time the log entry was received by Stackdriver Logging".
I tried to compare time range and number of records in table querying table using timestamp column and table decorator range.
Query where I'm using timestamp column.
SELECT count(*), MIN( timestamp ), max( timestamp )
FROM [project_id:dataset.appengine_googleapis_com_request_log_20170622]
WHERE timestamp between timestamp('2017-06-22 01:00:00') and
date_add(timestamp('2017-06-22 01:00:00'), 1, 'hour')
Query result.
1698320 | 2017-06-22 01:00:00 UTC | 2017-06-22 01:59:59 UTC
Query where I'm using table decorator range.
--select timestamp_to_msec(timestamp('2017-06-22 01:00:00')) as time1,
timestamp_to_msec(date_add(timestamp('2017-06-22 01:00:00'), 1, 'hour')) as time2
SELECT count(*), min(timestamp), max(timestamp)
FROM [project_id:dataset.appengine_googleapis_com_request_log_20170622#1498093200000-1498096800000]
Query result.
1534754 | 2017-06-22 00:40:45 UTC | 2017-06-22 01:35:59 UTC
I did not get the same date range and the same number of records. What each of these three timestamps mean? And how table decorators ranges works under the hood? (Does BigQuery make snapshots of tables, when it makes them)
Table Decorators docenter link description here explains that it uses snapshot.But it "References a snapshot of the table at " -- meaning the time the data was ingested into bigquery. However that's totally unrelated to the timestamp fields in your table, because those fields represent the time the related event happened, not the time the field is ingested into bigquery.
I am trying to migrate a column in Postgres from Timestamp without time zone to a timestamp with time zone. I want to do it for German time, so I used the following query :
alter table table_name alter column uploaddate type TIMESTAMP WITH TIME ZONE USING uploaddate at time zone 'UTC+01:30';
Unfortunately it's not working, it's adding 2015-06-30 07:30:48.785+05:30. I am currently in India, which is +5.30. How can I specify to the query to do it with German time zone. Thank you.
What is the timezone of the timestamps stored in the table? That is, if there is a value such as 2016-09-22 07:30 in the table, in what timezone is 07:30? This is the timezone that you want to use, not your current local timezone. So e.g. if all timestamps are expressed in german timezone you should do something like:
alter table table_name
alter column uploaddate type TIMESTAMP WITH TIME ZONE
USING uploaddate at time zone 'Europe/Berlin';
Don't forget that you can run the above inside a transaction so that you can inspect the results before you commit them (of course this will lock the entire table and block all other concurrent queries for that table).
I have 2 tables, 1 table has a date column, 1 table has a time column. I want to have date and time works seperatedly. This is what i use :
ALTER SESSION SET nls_date_format='dd/mm/yyyy';
I use this for the 1st table and then use this for the 2nd table :
ALTER SESSION SET nls_date_format='hh24:mi';
But it doesn't work right. When i do the select * from it all changes back to hh24:mi type. How can i have date and time seperatedly ?
As noted Oracle always has a date and time. If you want to see and use just the date or just the time you could use, for example to only work with the time, TO_CHAR(ColumnA, 'HH24:MI:SS')