SQL makes it easy to create certain For example, if you want to retrieve only the first of the month (not necessarily the '1st day' of the month) from the data accumulated in a daily batch for labels in a graph.
recorded_date
2022-09-05
2022-10-03
2022-11-01
2022-12-01
2023-01-03
-- collect from mysql
select min(recorded_date) recorded_date from daily_table group by DATE_FORMAT(`recorded_date`, "%Y-%m")
On the other hand, queryset requires that 'values' be specified before aggregation.
first_days = Industry.objects\
.values('recorded_date')\
.annotate(ym=Func(F('recorded_date'), Value('%Y%m'), function='DATE_FORMAT', output_field=CharField()))\
.annotate(count=Count('id'))
The queryset is written out as follows
SELECT
`vietnam_research_industry`.`recorded_date`,
DATE_FORMAT(`vietnam_research_industry`.`recorded_date`, '%Y%m') AS `ym`,
COUNT(`vietnam_research_industry`.`id`) AS `count`
FROM `vietnam_research_industry`
GROUP BY `vietnam_research_industry`.`recorded_date`, DATE_FORMAT(`vietnam_research_industry`.`recorded_date`, '%Y%m')
ORDER BY NULL
this is failed as follows
recorded_date
ym
count
2022-12-16
202212
757
2022-12-21
202212
757
2022-12-22
202212
757
2022-12-23
202212
757
2022-12-26
202212
757
2022-12-27
202212
757
2022-12-28
202212
757
2022-12-29
202212
757
2022-12-30
202212
757
Is there no choice but to use 'raw' in these cases?
Currently, all dates are retrieved without duplicates with 'distinct' and handled by setdefault in the dictionary object.
thanks :)
You can try below query for the same.
Industry.objects\
.annotate(month=ExtractMonth("recorded_date"), year=ExtractYear("recorded_date"), concat=Concat("month", Value("-"), "year", output_field=CharField()))\
.values('concat')\
.annotate(Min("recorded_date"))\
.values("recorded_date__min")
Let me know if it helps :)
Related
I'm storing interface data from my Router gathered through SNMP in a MariaDB. The data is structured in the following way (simplified):
id TIMESTAMP READING VALUE
============================================================
100 2020-04-15 11:29:51 if03_name eth0
101 2020-04-15 11:29:51 if03_totalBytesRx 654321
102 2020-04-15 11:29:51 if03_totalBytesTx 123456
103 2020-04-15 11:30:51 if03_totalBytesRx 765432
104 2020-04-15 11:30:51 if03_totalBytesTx 234567
Now to get the data received and transmitted for eth0 it is easily possible to select on the READING if03_totalBytesRx or if03_totalBytesTx.
For instance, I can execute the following query to get the average bits per second received at any given time:
SELECT
TIMESTAMP,
GREATEST(0,(VALUE - LAG(VALUE,1) OVER (ORDER BY TIMESTAMP))*8/(UNIX_TIMESTAMP(TIMESTAMP) - UNIX_TIMESTAMP(LAG(TIMESTAMP,1) OVER (ORDER BY TIMESTAMP)))) as value
FROM history
WHERE
READING = "if03_totalBytesRx"
Unfortunately, sometimes (I believe when an interface goes down and up again) the mapping between READING and interface changes, such that if03 is not eth0 any more (if03 could for instance be eth1 or also noSuchInstance), whereas eth0 is assigned another READING, for example if29:
id TIMESTAMP READING VALUE
============================================================
200 2020-04-15 12:16:51 if03_totalBytesRx 876543
201 2020-04-15 12:16:51 if03_totalBytesTx 345678
202 2020-04-15 12:17:51 if03_name noSuchInstance
203 2020-04-15 12:17:51 if03_totalBytesRx noSuchInstance
204 2020-04-15 12:17:51 if03_totalBytesTx noSuchInstance
205 2020-04-15 12:17:51 if29_name eth0
206 2020-04-15 12:17:51 if29_totalBytesRx 987654
207 2020-04-15 12:17:51 if29_totalBytesTx 456789
Note that if03_name is only stored in the DB when changed, not every minute.
Obviously that causes either no data when querying the READING if03_totalBytesRx (in the case of noSuchInstance) or false data, i.e. data from another interface.
What would be a viable way to select all the ifXX_totalBytesRx and ifXX_totalBytesTx for all the timeranges where the corresponding ifXX_name equals to eth0? (E.g. in the example above the union of 11:29:51 to 12:17:50 using if03 plus everything from 12:17:51 to NOW() using if29, assuming that id 207 is the last entry in the database)
Ok, I think I found a solution myself, however it is kind of slow, so I'm happy about any improvements:
First, I created a view ubnt that splits the original READING into a new field INTERFACE (the part before the underscore) and another field READING (the part after the underscore, reusing the name):
SELECT
id,
TIMESTAMP,
substring_index(READING,'_',1) AS INTERFACE,
substring_index(READING,'_',-1) AS READING,
VALUE
FROM history
WHERE READING like 'if%'
The second step is to create another view from this called ubnt_if that shows the timerange of a logical to physical interface mapping:
SELECT
TIMESTAMP AS START,
COALESCE(LEAD(TIMESTAMP,1) OVER ( PARTITION BY INTERFACE ORDER BY TIMESTAMP),NOW()) AS END,
INTERFACE AS LOGICAL,
VALUE AS PHYSICAL
FROM ubnt
WHERE READING = 'name'
The third step is to join the interface to the original data, based on the timestamp. For this, I used another view called ubnt_int:
SELECT
a.TIMESTAMP AS TIMESTAMP,
a.READING AS READING,
a.VALUE AS VALUE,
b.PHYSICAL AS INTERFACE
FROM (
ubnt a
LEFT JOIN ubnt_if b ON(
a.INTERFACE = b.LOGICAL AND
a.TIMESTAMP BETWEEN b.START AND b.END
)
)
This generates a table of data that I can filter based on the physical interface, not the "logical" interface that could change.
I was inspired by this answer to another question.
I am fairly green at SQL and have reached a road block. I have a Job that already runs a query and sends an email to our purchasing department agents via the agent.
Here is my sample data they receive as a text attachment:
po_num vend_num qty_needed external_email_addr
318 1 200 email#earthlink.net
318 1 910 email#earthlink.net
703 2 250 email#row.com
993 3 3600 email#cast.com
993 3 3600 email#cast.com
676 4 1 NULL
884 5 10000 email#Futures.com
118 5 2500 email#Futures.com
My goal is to automatically send each vendor one email of the qty_needed using the email address in external_email_addr field. Also, if the email address is NULL it would send me an email that this needs to be fixed.
I am not sure how complicated or simple this is but any help would be greatly appreciated.
Since the po_num is unique you will generate several mails per email address per day based on the example data you provided.
I dont have access to SQL at the moment so the syntax might need some sprucing up.
SELECT po_num,
vend_num,
qty_needed,
CASE WHEN external_email_addr ='' THEN COALESCE(external_email_addr,'defaultempty#fixthisproblem.com') ELSE external_email_ddr END AS email_address
FROM table_name
I have a CSV file that I load into my DB( Without using SSIS) with about 20 rows and 3 columns(as below):
**start_date_local** **end_date_local** **provider_unit**
18/04/2017 16:00 19/04/2017 16:00 501638-52973
19/04/2017 05:30 19/04/2017 23:00 501613-52345
07/04/2017 14:30 08/04/2017 15:30 201447-20266
each row/record should act as a parameter in my query (SQL server) that consist the following 'Where' clause
((Transmission.Transmission_StartDateAccurate >= '**start_date_local**')
AND (Transmission.Transmission_EndDateAccurate <= **'end_date_local'**))
AND Units.sn LIKE **'provider_unit'**
i would like to receive at my result set usage data (Which is in another table and that's fine) of each unit at its specific times - as follows:
start_date_local end_date_local provider_unit GB USAGE
18/04/2017 16:00 19/04/2017 16:00 501638-52973 35.3
19/04/2017 05:30 19/04/2017 23:00 501613-52345 42.4
07/04/2017 14:30 08/04/2017 15:30 201447-20266 4.5
please advice:)
Thanks,
Guy
If you are loading this data into a Staging table within your database, you can just join onto it to use the data in a set based manner, which will return your dataset for all instances.
Not having your full schema this is a bit of guess work, but you should be doing something like this:
select s.start_date_local
,s.end_date_local
,s.provider_unit
,t.GBUsage
from Transmission t
join StagingTable s
on(t.Transmission_StartDateAccurate >= s.start_date_local
and t.Transmission_EndDateAccurate <= s.end_date_local
)
join Units u
on(u.sn like s.provider_unit);
I have an SSIS package that is configured to log to SQL Server. With the package configured, a System Table is created under: MyDatabase > System Tables > dbo.sysssislog
This table has a schema that matches the identically named table held in msdb.
Within this table, each package execution has a unique executionid, which is defined as:
uniqueidentifier
The GUID of the execution instance of the executable that generated the logging entry
This is generated each time the SSIS package is run and held within the following system variable: System::ExecutionInstanceGUID
Sample Logs query:
SELECT [event] ,
[source] ,
[executionid] ,
[starttime] ,
[endtime]
FROM MyDatabase.[dbo].[sysssislog]
WHERE [event] IN ( 'PackageEnd', 'PackageStart' )
ORDER BY id desc, starttime
Produces:
event source starttime endtime executionid
PackageEnd Package 2017-04-10 11:12:01 2017-04-10 11:12:01 4EDBF979-5E99-44DB-AA08-839D5DCF3F2F
PackageStart Package 2017-04-10 11:12:01 2017-04-10 11:12:01 4EDBF979-5E99-44DB-AA08-839D5DCF3F2F
PackageEnd Package 2017-04-05 13:39:11 2017-04-05 13:39:11 9E212747-3CB7-44D8-8728-9E442082DB8B
PackageStart Package 2017-04-05 13:39:11 2017-04-05 13:39:11 9E212747-3CB7-44D8-8728-9E442082DB8B
Within my application I'm using various SQL Server Agent Job Stored Procedures to retrieve SSIS job information and history. For example:
EXEC msdb.dbo.sp_help_jobhistory
#job_name = N'MyJobName',
#step_id = null
GO
Produces (summary of columns, 1 execution = 3 rows):
job_id job_name run_date run_time run_duration
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170410 111145 16
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170410 111200 1
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170410 111145 15
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170405 133855 16
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170405 133910 1
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170405 133855 15
I'm building an ETL Admin page in my application that shows SSIS job history and a summary of the logs, but I cannot find a way to link logs, based on the unique executionid, to the Job History returned from the various Agent Job System Stored Procedures.
Is there a way to link the executionid from sysssislog to information held against the Agent Job execution history? The best I can come up with is using date/time matching to identify the logs that are closest in time to the agent job stats.
I've looked at using script tasks and firing custom events to log the System::ExecutionInstanceGUID to the agent job history, but I'm unable to use script tasks as it will not work when deployed to customers with later versions of SQL Server.
Please note, any solutions need to be compatible with 2008R2.
I ended up with the following solution, which isn't the prettiest and most performant, but it satisfied the requirement I had. I wanted the last n number of executions of my ssis package, which is specified by the 2 parameters. I hold this code in a stored procedure that is called by my application:
DECLARE #packageName NVARCHAR(250) = 'MySqlServerAgentJobName',
#rowsToTake INT = 15;
WITH cte
AS (SELECT TOP 15 msdb.dbo.agent_datetime(h.run_date, h.run_time) AS LastRunDate,
h.run_status RunStatus,
h.run_duration RunDurationSeconds,
(
SELECT TOP 1 executionid
FROM [dbo].[sysssislog]
WHERE starttime >= msdb.dbo.agent_datetime(h.run_date, h.run_time)
AND [event] = ('PackageStart')
) ExecutionId
FROM msdb.dbo.sysjobs j
INNER JOIN msdb.dbo.sysjobhistory h ON j.job_id = h.job_id
WHERE j.name = #packageName AND h.step_name = '(Job outcome)'
ORDER BY LastRunDate DESC
)
SELECT cte.LastRunDate,
cte.RunStatus,
cte.RunDurationSeconds,
cte.ExecutionId
FROM cte
It uses a built in function to parse a valid datetime from the agent job fields:
msdb.dbo.agent_datetime(h.run_date, h.run_time) AS LastRunDate
It retrieves the top 1 execution id from the sysssislogs table, filtering on [event] = ('PackageStart'), which exists for each time the job is executed and the starttime must be >= the LastRunDate that the agent job logged.
It produces results like this:
LastRunDate RunStatus RunDurationSecs ExecutionId
2017-07-12 12:32:18.000 1 210 99A8A715-890D-4115-975F-AC3DA660770D
2017-07-12 12:00:01.000 1 215 D152C1C6-530F-45D6-B962-41784EC7D9E5
2017-07-12 11:22:13.000 1 204 BBAC73C1-1600-477E-AFC9-1F66D7CFD07A
2017-07-12 10:25:44.000 1 213 A4C69B3C-462B-4000-B2BD-8C9EB47230DA
2017-07-12 10:16:19.000 1 230 3D4D8572-E4CC-4B70-875A-C5D3B513A480
2017-07-12 07:33:21.000 1 244 D8283955-B3EB-4E6D-BFDC-8D7C0D954574
2017-07-11 12:00:00.000 1 211 553FC580-54B2-490C-BB1D-6B1F9BB35C84
2017-07-10 16:21:16.000 1 212 B9A78077-A04C-49AC-A4C6-EF6AB636D862
2017-07-10 16:13:18.000 1 255 DFC853D8-67F5-43CE-B8A4-00B846425C6B
2017-07-10 14:04:00.000 1 217 8C6861B1-6CDE-44FD-8606-18F24F339C05
2017-07-10 13:58:27.000 1 224 5A05011C-9F14-4098-B579-0E75C1E76BDB
2017-07-10 13:53:14.000 1 231 9A2B52FB-2BD4-4FAA-A4A0-40AC48A41326
2017-07-10 13:42:06.000 1 210 9BDBB290-1C84-49ED-87E6-97522DB4AB81
2017-07-10 12:06:44.000 1 215 F4D0A847-F7E4-4F2B-B328-58C387D2C50E
2017-07-10 12:00:01.000 1 241 5EC52369-9832-4718-AF92-2C502D454A41
Currently I am working on Indexing services to search document from Windows server. I am using following Query to search "F-0350"
Query
Select rank,Filename,Size,HitCount from Scope() where CONTAINS(Contents, 'ISABOUT('F-0350' WEIGHT(1.0))')
Query Result:
Rank FileName Size HitCount
572 sf ob-0350.prod.xml 2521 4
572 sf g-0350.prod.xml 1199 4
572 sf qa-0350.prod.xml 1864 4
572 sf f-0350.prod.xml 1551 4
362 sf ob-0353.prod.xml 2469 4
I am expecting 4th record on top but its not coming.