how to run on an imported table and pass each row as a parameter to a query(SQL server) - sql-server

I have a CSV file that I load into my DB( Without using SSIS) with about 20 rows and 3 columns(as below):
**start_date_local** **end_date_local** **provider_unit**
18/04/2017 16:00 19/04/2017 16:00 501638-52973
19/04/2017 05:30 19/04/2017 23:00 501613-52345
07/04/2017 14:30 08/04/2017 15:30 201447-20266
each row/record should act as a parameter in my query (SQL server) that consist the following 'Where' clause
((Transmission.Transmission_StartDateAccurate >= '**start_date_local**')
AND (Transmission.Transmission_EndDateAccurate <= **'end_date_local'**))
AND Units.sn LIKE **'provider_unit'**
i would like to receive at my result set usage data (Which is in another table and that's fine) of each unit at its specific times - as follows:
start_date_local end_date_local provider_unit GB USAGE
18/04/2017 16:00 19/04/2017 16:00 501638-52973 35.3
19/04/2017 05:30 19/04/2017 23:00 501613-52345 42.4
07/04/2017 14:30 08/04/2017 15:30 201447-20266 4.5
please advice:)
Thanks,
Guy

If you are loading this data into a Staging table within your database, you can just join onto it to use the data in a set based manner, which will return your dataset for all instances.
Not having your full schema this is a bit of guess work, but you should be doing something like this:
select s.start_date_local
,s.end_date_local
,s.provider_unit
,t.GBUsage
from Transmission t
join StagingTable s
on(t.Transmission_StartDateAccurate >= s.start_date_local
and t.Transmission_EndDateAccurate <= s.end_date_local
)
join Units u
on(u.sn like s.provider_unit);

Related

I want a record of the first day of each month

SQL makes it easy to create certain For example, if you want to retrieve only the first of the month (not necessarily the '1st day' of the month) from the data accumulated in a daily batch for labels in a graph.
recorded_date
2022-09-05
2022-10-03
2022-11-01
2022-12-01
2023-01-03
-- collect from mysql
select min(recorded_date) recorded_date from daily_table group by DATE_FORMAT(`recorded_date`, "%Y-%m")
On the other hand, queryset requires that 'values' be specified before aggregation.
first_days = Industry.objects\
.values('recorded_date')\
.annotate(ym=Func(F('recorded_date'), Value('%Y%m'), function='DATE_FORMAT', output_field=CharField()))\
.annotate(count=Count('id'))
The queryset is written out as follows
SELECT
`vietnam_research_industry`.`recorded_date`,
DATE_FORMAT(`vietnam_research_industry`.`recorded_date`, '%Y%m') AS `ym`,
COUNT(`vietnam_research_industry`.`id`) AS `count`
FROM `vietnam_research_industry`
GROUP BY `vietnam_research_industry`.`recorded_date`, DATE_FORMAT(`vietnam_research_industry`.`recorded_date`, '%Y%m')
ORDER BY NULL
this is failed as follows
recorded_date
ym
count
2022-12-16
202212
757
2022-12-21
202212
757
2022-12-22
202212
757
2022-12-23
202212
757
2022-12-26
202212
757
2022-12-27
202212
757
2022-12-28
202212
757
2022-12-29
202212
757
2022-12-30
202212
757
Is there no choice but to use 'raw' in these cases?
Currently, all dates are retrieved without duplicates with 'distinct' and handled by setdefault in the dictionary object.
thanks :)
You can try below query for the same.
Industry.objects\
.annotate(month=ExtractMonth("recorded_date"), year=ExtractYear("recorded_date"), concat=Concat("month", Value("-"), "year", output_field=CharField()))\
.values('concat')\
.annotate(Min("recorded_date"))\
.values("recorded_date__min")
Let me know if it helps :)

SQL recursive get BOM from PSP

I am having a MS SQL Server (2016) and a database which contains i.a. table like this : (it´s a view created in an Autodesk PSP Database - please don´t ask why ... :-) )
CHILD_AIMKEY
QUANTITY
PARENT_AIMKEY
StatusOfParent
StatusOfChild
5706657
1
5664344
100
103
5706745
1
5664344
100
103
5707104
1
5664344
100
103
5707109
1
5664344
100
100
5801062
1
5664344
100
103
The "children" can contain other "children" and in that case they would be their "parents".
So it´s a standard structured BOM table from a CAD PDM System.
If I do the following "Select Statement" I get all the children of the top level parent:
SELECT [CHILD_AIMKEY] , [POSITION], [QUANTITY] ,[PARENT_AIMKEY],[StatusOfParent],[StatusChild] FROM database_table where Parent_aimkey = '5664344'
(as shown in the table above)
My first question is : How to recursivly process all children of each parent from that table ? (Could be an other table or direct output)
The format should be: Parent_Aimkey, Child_Aimkey, Quantity
The second question is a bit more complicated:
I try it with some "pseudo code":
If Tree_Level_of_DIRECT_Parent < 3 then show CHILD_AIMKEY,QUANTITY in queryresult_above
If Tree_Level_of_DIRECT_Parent > 2 and StatusOf_DIRECT_Parent = 103 and StatusOf_DIRECT_Child = 103 then show CHILD_AIMKEY,QUANTITY in queryresult_above
Is that in some way possible ? (If there is a need to extend the database view of an other field or another table, that´s no problem)
I know this looks a bit confusing, but what I need is the Autodesk Inventor structured BOM in an SQL Statement or stored procedure.
Any would be really much appreciated
Thanks
Alex.

Linking sysssislog executionid to SSIS Agent Job history

I have an SSIS package that is configured to log to SQL Server. With the package configured, a System Table is created under: MyDatabase > System Tables > dbo.sysssislog
This table has a schema that matches the identically named table held in msdb.
Within this table, each package execution has a unique executionid, which is defined as:
uniqueidentifier
The GUID of the execution instance of the executable that generated the logging entry
This is generated each time the SSIS package is run and held within the following system variable: System::ExecutionInstanceGUID
Sample Logs query:
SELECT [event] ,
[source] ,
[executionid] ,
[starttime] ,
[endtime]
FROM MyDatabase.[dbo].[sysssislog]
WHERE [event] IN ( 'PackageEnd', 'PackageStart' )
ORDER BY id desc, starttime
Produces:
event source starttime endtime executionid
PackageEnd Package 2017-04-10 11:12:01 2017-04-10 11:12:01 4EDBF979-5E99-44DB-AA08-839D5DCF3F2F
PackageStart Package 2017-04-10 11:12:01 2017-04-10 11:12:01 4EDBF979-5E99-44DB-AA08-839D5DCF3F2F
PackageEnd Package 2017-04-05 13:39:11 2017-04-05 13:39:11 9E212747-3CB7-44D8-8728-9E442082DB8B
PackageStart Package 2017-04-05 13:39:11 2017-04-05 13:39:11 9E212747-3CB7-44D8-8728-9E442082DB8B
Within my application I'm using various SQL Server Agent Job Stored Procedures to retrieve SSIS job information and history. For example:
EXEC msdb.dbo.sp_help_jobhistory
#job_name = N'MyJobName',
#step_id = null
GO
Produces (summary of columns, 1 execution = 3 rows):
job_id job_name run_date run_time run_duration
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170410 111145 16
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170410 111200 1
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170410 111145 15
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170405 133855 16
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170405 133910 1
52916CFE-A652-4AAA-A052-738E4B349966 MyJobName 20170405 133855 15
I'm building an ETL Admin page in my application that shows SSIS job history and a summary of the logs, but I cannot find a way to link logs, based on the unique executionid, to the Job History returned from the various Agent Job System Stored Procedures.
Is there a way to link the executionid from sysssislog to information held against the Agent Job execution history? The best I can come up with is using date/time matching to identify the logs that are closest in time to the agent job stats.
I've looked at using script tasks and firing custom events to log the System::ExecutionInstanceGUID to the agent job history, but I'm unable to use script tasks as it will not work when deployed to customers with later versions of SQL Server.
Please note, any solutions need to be compatible with 2008R2.
I ended up with the following solution, which isn't the prettiest and most performant, but it satisfied the requirement I had. I wanted the last n number of executions of my ssis package, which is specified by the 2 parameters. I hold this code in a stored procedure that is called by my application:
DECLARE #packageName NVARCHAR(250) = 'MySqlServerAgentJobName',
#rowsToTake INT = 15;
WITH cte
AS (SELECT TOP 15 msdb.dbo.agent_datetime(h.run_date, h.run_time) AS LastRunDate,
h.run_status RunStatus,
h.run_duration RunDurationSeconds,
(
SELECT TOP 1 executionid
FROM [dbo].[sysssislog]
WHERE starttime >= msdb.dbo.agent_datetime(h.run_date, h.run_time)
AND [event] = ('PackageStart')
) ExecutionId
FROM msdb.dbo.sysjobs j
INNER JOIN msdb.dbo.sysjobhistory h ON j.job_id = h.job_id
WHERE j.name = #packageName AND h.step_name = '(Job outcome)'
ORDER BY LastRunDate DESC
)
SELECT cte.LastRunDate,
cte.RunStatus,
cte.RunDurationSeconds,
cte.ExecutionId
FROM cte
It uses a built in function to parse a valid datetime from the agent job fields:
msdb.dbo.agent_datetime(h.run_date, h.run_time) AS LastRunDate
It retrieves the top 1 execution id from the sysssislogs table, filtering on [event] = ('PackageStart'), which exists for each time the job is executed and the starttime must be >= the LastRunDate that the agent job logged.
It produces results like this:
LastRunDate RunStatus RunDurationSecs ExecutionId
2017-07-12 12:32:18.000 1 210 99A8A715-890D-4115-975F-AC3DA660770D
2017-07-12 12:00:01.000 1 215 D152C1C6-530F-45D6-B962-41784EC7D9E5
2017-07-12 11:22:13.000 1 204 BBAC73C1-1600-477E-AFC9-1F66D7CFD07A
2017-07-12 10:25:44.000 1 213 A4C69B3C-462B-4000-B2BD-8C9EB47230DA
2017-07-12 10:16:19.000 1 230 3D4D8572-E4CC-4B70-875A-C5D3B513A480
2017-07-12 07:33:21.000 1 244 D8283955-B3EB-4E6D-BFDC-8D7C0D954574
2017-07-11 12:00:00.000 1 211 553FC580-54B2-490C-BB1D-6B1F9BB35C84
2017-07-10 16:21:16.000 1 212 B9A78077-A04C-49AC-A4C6-EF6AB636D862
2017-07-10 16:13:18.000 1 255 DFC853D8-67F5-43CE-B8A4-00B846425C6B
2017-07-10 14:04:00.000 1 217 8C6861B1-6CDE-44FD-8606-18F24F339C05
2017-07-10 13:58:27.000 1 224 5A05011C-9F14-4098-B579-0E75C1E76BDB
2017-07-10 13:53:14.000 1 231 9A2B52FB-2BD4-4FAA-A4A0-40AC48A41326
2017-07-10 13:42:06.000 1 210 9BDBB290-1C84-49ED-87E6-97522DB4AB81
2017-07-10 12:06:44.000 1 215 F4D0A847-F7E4-4F2B-B328-58C387D2C50E
2017-07-10 12:00:01.000 1 241 5EC52369-9832-4718-AF92-2C502D454A41

SQL query duplicates, self-join, and calculated field

I'm putting together a report to track latency in a SQL Availability Group. I need to find the difference between two times. I have server1, database1 and time1. I need to compare it to server2, database1, and time2. The data is stored in a table, and new data will be appended to the table. The query I'm trying to create will get these new entries (the field Hardened_time_MS_Diff will be NULL) and I need to get the difference between the times returned for the Primary and Secondary servers. Here's an example of the data:
Server_name Database_name Last_Hardened_Time
Server1 ABC 2015-10-08 10:10:05.180
Server2 ABC 2015-10-08 10:10:05.643
My query to get the Hardened_time_MS_Diff values looks like this:
Select a1.server_name, a1.database_name,
Case when a1.Last_Hardened_Time >= a2.Last_Hardened_Time
then
DATEDIFF (MS, a2.Last_Hardened_Time, a1.Last_Hardened_Time)
Else
DATEDIFF (MS, a1.Last_Hardened_Time, a2.Last_Hardened_Time)
End as Hardened
FROM [database].[dbo].[ag_latency] a1
JOIN ag_latency a2
ON a1.database_name = a2.database_name
AND a1.server_name <> a2.server_name
where a1.Hardened_time_MS_Diff is NULL
Right now it's returning some crazy numbers, and duplicates. The first run it'll work fine. The second run my data will look correct only for the servers whose Last_Hardened_time hasn't changed. If it changed I'll get an absolutely huge number for the second batch like this:
Server_name Database_name Last_Hardened_Time Hardened_time_MS_Diff
Server1 ABC 2015-10-09 12:00:05.013 26
Server2 ABC 2015-10-09 12:00:05.040 26
Server1 ABC 2015-10-09 12:15:07.843 -902803
Server2 ABC 2015-10-09 12:15:07.877 -902863
How can I get this to do what I want it to do?
Also, the part to get the data is in Powershell, so if there's an easier way to manipulate the data table in PoSH and then push that data down let me know.
Thanks in advance!
Simply answer your question, your join clause won't work when you have multiple pair of timestamps - it will join not only the row1-row2 and row3-row4, but also join row1-row4 and row2-row3, which the later two doesn't make sense to your report.
One possible solution would be eliminate any time difference greater than a threshold (say 1000 MilliSec, according to your sample data). So your where clause would look like
where a1.Hardened_time_MS_Diff is NULL AND (DATEDIFF (MS, a2.Last_Hardened_Time, a1.Last_Hardened_Time) < 1000 AND DATEDIFF (MS, a2.Last_Hardened_Time, a1.Last_Hardened_Time) > -1000)
Another solution would be more robust, but requires you to change your table schema. You need to add a column to the table, indicate which 2 rows are paired. For example, assign 1 to first two rows and 2 to last two rows, and then join table a1 and a2 by this column.
Key Server_name Database_name Last_Hardened_Time Hardened_time_MS_Diff
1 Server1 ABC 2015-10-09 12:00:05.013 26
1 Server2 ABC 2015-10-09 12:00:05.040 26
2 Server1 ABC 2015-10-09 12:15:07.843 -902803
2 Server2 ABC 2015-10-09 12:15:07.877 -902863
For the powershell part, can you please elaborate what you need more specifically?
Henry

Nightly Excel Spreadheet import into mssql database

I am currently working with a nightly import that I need to create, but am not sure what the best route would be to update/insert into the current table. This is all done in MS SQL Server 2012 and pulling the Excel file from another server. I am trying to figure out how I can loop through the columns and pull out the data I need. If I could rearrange the data, I would but am currently stuck with what I have.
In my current table tblHW I have Columns such as PmpCount, , NumberStages, Pmpmodel_pmp1, serialnum_Pmp1, pmpModel_pmp2, Pmpmodel_pmp2, serialnum_pmp2, partnum_motor1, serialnumberMotor1, etc…. I apologize in advance for not being able to post a real table or a picture.
Example:
|Name | PmpCount| numstages| pmpmodel_pmp1| stages_pmp1| Sn_pmp1|
|AN 91-23G | 4| 500| FX2347| 250| 354197|
|BR DN 895R| 5| 521| D2442| 45| 875164|
|ALN 1-60J | 5| 521| H21342| 95| 594126|
|pmpmodel_pmp2| stages_pmp2| sn_pmp2| Partnum_mtr1| sn_mtr1|
|FX2347 | 250| 354198| NULL| NULL|
|FX17500 | 143| 102547| M7544| 4512241|
|FX17500 | 143| 458790| M7544| 4512364|
The information I want to move into tblHW comes from the tbl Pull_Down. Here is the setup:
|Name | Run_ID | Part1| SN1 | Attribute1_7|
|AN 21-919G| Oct 08, 2013 / 100845| BOD| NA| 3RD U|
|FR 55-013A| Oct 17, 2013 / 100853| Pmp| 2EA3A022| 78|
|FR 55-013A| Oct 01, 2014 / 101383| Cbl| N/A| REDALEAD|
|FR 43-223J| Apr 03, 2013 / 100594| BOD| NA| 3RD U|
|VH 204 | May 17, 2014 / 101145| BOD| 3RD U|
|Part2| SN2 | Attribute2_7| Part3 | SN3 | Attribute3_7|
|Pmp | 2EA3F379| 78| Pmp| 2EA3N380| 117|
|Pmp | 2EA3C020| 117| Pmp| 2EA3Y021| 117|
|MLE | J14312161| 120| BOD| N/A| 3RD U|
|Other| NA| Pmp| 2EA2X774| 78|
|BOD | NULL| Pmp| 2EA4F075| 38|
A bit more information. I am receiving this information in the form of five excel spreadhsheets each with over 400 columns. The columns giving me the biggest headache are the 20 part columns that I need to place into the SQL table.
I need to somehow move each row into the tblHW but need to do something like this:
The first row AN 21-919G needs to have SN1 to be inserted into sn_mtr1 since it is a BOD, SN2 into SN_pmp1 since it is a PMP, and SN3 into sn_pmp2 since it is the second PMP here. I also need to get the pmp count, in this case 2 and then add the attribute1_7 and attribute2_7 to put into numstages when the prts are PMP.
Situations like this is the whole purpose for SSIS to exist: Integration Services!
First one would question as to why the data you need is in Excel, and if there is a more direct route, one could exploit, as a linked server (if the source is another RDBMS).
Based in what information you provide, we make the following assumptions:
A) We have no control on the source output and we must import the data from Excel.
B) The files always have consistent columns (probably created by an automated process).
In SSIS you can easily create a source connection for the Excel file. If the Excel file name is dynamic, you can create a script to modify the connection string for that connection before importing data. Then set the destination connection to the SQL Server. The last step is creating a Data Flow Task where you can map the source to the destination columns.
Example:

Resources