My client org has a requirement where Account has two child objects. One is Banking Commission and the other is Invoices. Banking Commission has Processing Period(Date) and MDR column. Invoice has Invoice date and Gateway column. A table displaying Account with all the banking commissions(Only MDR column) for a given processing period and all the invoices(Only Gateway)for aa given invoice date period needs to be displayed.
Shown in image
I have created something like this shown in image where A is MDR and B is Gateway and Total is last total column . I have not added individual totals for now . I want to show all MDR and Gateway separately as per requirement.
This is my join query :
q1 = load "Companies_with_Banking_Commissions";
q2 = load "Invoices";
q = cogroup q1 by ('Company.Name', 'Company.Industry', 'Company.TP_Office__c', 'Company.Sales_Partner__c', 'Company.Sales_Person__c','Processing_Period__c_Month') full, q2 by ('Company.Name', 'Company.Industry', 'Company.TP_Office__c', 'Company.Sales_Partner__c', 'Company.Sales_Person__c', 'Invoice_Date__c_Month');
q = foreach q generate coalesce(q1.'Company.Name', q2.'Company.Name') as 'Company.Name', coalesce(q1.'Company.Industry', q2.'Company.Industry') as 'Company.Industry', coalesce(q1.'Company.TP_Office__c', q2.'Company.TP_Office__c') as 'Company.TP_Office__c', coalesce(q1.'Company.Sales_Partner__c', q2.'Company.Sales_Partner__c') as 'Company.Sales_Partner__c', coalesce(q1.'Company.Sales_Person__c', q2.'Company.Sales_Person__c') as 'Company.Sales_Person__c', coalesce(q1.'Processing_Period__c_Month', q2.'Invoice_Date__c_Month') as 'Processing_Period__c_Month', coalesce(sum(q1.'Total_MDR__c') , 0) as 'A', coalesce(sum(q2.'Invoice_Amount_Including_VAT__c'),0) as 'B';
q = foreach q generate 'Company.Name', 'Company.Industry', 'Company.TP_Office__c', 'Company.Sales_Partner__c', 'Company.Sales_Person__c', 'Processing_Period__c_Month', 'A', 'B', A + B as 'Total';
q = order q by ('Company.Name' asc, 'Company.Industry' asc, 'Company.TP_Office__c' asc, 'Company.Sales_Partner__c' asc, 'Company.Sales_Person__c' asc, 'Processing_Period__c_Month' asc);
q = limit q 2000;
Related
i am attempting a simple exercise
i have Two kinesis data stream
order-stream
shipment-stream
SQL 1 Orders
%flink.ssql
CREATE TABLE orders (
orderid VARCHAR(6),
orders VARCHAR,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts - INTERVAL '5' SECOND
)
WITH (
'connector' = 'kinesis',
'stream' = 'order-stream',
'aws.region' = 'us-east-1',
'scan.stream.initpos' = 'TRIM_HORIZON',
'format' = 'json',
'json.timestamp-format.standard' = 'ISO-8601'
);
SQL 2 shipment
CREATE TABLE shipment (
orderid VARCHAR(6),
shipments VARCHAR(6),
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts - INTERVAL '5' SECOND
)
WITH (
'connector' = 'kinesis',
'stream' = 'shipment-stream',
'aws.region' = 'us-east-1',
'scan.stream.initpos' = 'TRIM_HORIZON',
'format' = 'json',
'json.timestamp-format.standard' = 'ISO-8601'
);
Generating Fake Data into Kinesis Via Python
try:
import datetime
import json
import random
import boto3
import os
import uuid
import time
from dotenv import load_dotenv
load_dotenv(".env")
except Exception as e:
pass
STREAM_NAME_Order = "order-stream"
STREAM_NAME_Shipments = "shipment-stream"
def send_data(kinesis_client):
order_items_number = random.randrange(1, 10000)
order_items = {
"orderid": order_items_number,
"orders": "1",
'ts': datetime.datetime.now().isoformat()
}
shipping_data = {
"orderid": order_items_number,
"shipments": random.randrange(1, 10000),
'ts': datetime.datetime.now().isoformat()
}
partition_key = uuid.uuid4().__str__()
res = kinesis_client.put_record(
StreamName=STREAM_NAME_Order,
Data=json.dumps(order_items),
PartitionKey=partition_key)
print(res)
time.sleep(2)
res = kinesis_client.put_record(
StreamName=STREAM_NAME_Shipments,
Data=json.dumps(shipping_data),
PartitionKey=partition_key)
print(res)
if __name__ == '__main__':
kinesis_client = boto3.client('kinesis',
aws_access_key_id=os.getenv("DEV_ACCESS_KEY"),
aws_secret_access_key=os.getenv("DEV_SECRET_KEY"),
region_name="us-east-1",
)
for i in range(1, 10):
send_data(kinesis_client)
%flink.ssql(type=update)
SELECT DISTINCT oo.orderid , TUMBLE_START(oo.ts, INTERVAL '10' MINUTE) as event_time
FROM orders as oo
GROUP BY orderid , TUMBLE(oo.ts, INTERVAL '10' MINUTE);
issue with Joining
%flink.ssql(type=update)
SELECT DISTINCT oo.orderid , TUMBLE_START(oo.ts, INTERVAL '10' MINUTE) as event_time , ss.shipments
FROM orders as oo
JOIN shipment AS ss ON oo.orderid = ss.orderid
GROUP BY oo.orderid , TUMBLE(oo.ts, INTERVAL '10' MINUTE) , ss.shipments
Error Messages
TableException: Rowtime attributes must not be in the input rows of a regular join. As a workaround you can cast the time attributes of input tables to TIMESTAMP before.
java.io.IOException: Fail to run stream sql job
at org.apache.zeppelin.flink.sql.AbstractStreamSqlJob.run(AbstractStreamSqlJob.java:172)
at org.apache.zeppelin.flink.sql.AbstractStreamSqlJob.run(AbstractStreamSqlJob.java:105)
at org.apache.zeppelin.flink.FlinkStreamSqlInterpreter.callInnerSelect(FlinkStreamSqlInterpreter.java:89)
at org.apache.zeppelin.flink.FlinkSqlInterrpeter.callSelect(FlinkSqlInterrpeter.java:503)
at org.apache.zeppelin.flink.FlinkSqlInterrpeter.callCommand(FlinkSqlInterrpeter.java:266)
at org.apache.zeppelin.flink.FlinkSqlInterrpeter.runSqlList(FlinkSqlInterrpeter.java:160)
at org.apache.zeppelin.flink.FlinkSqlInterrpeter.internalInterpret(FlinkSqlInterrpeter.java:112)
at org.apache.zeppelin.interpreter.AbstractInterpreter.interpret(AbstractInterpreter.java:47)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:110)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:852)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:744)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.ParallelScheduler.lambda$runJobInScheduler$0(ParallelScheduler.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.flink.table.api.TableException: Cannot generate a valid execution plan for the given query:
Also tried
%flink.ssql(type=update)
SELECT DISTINCT oo.orderid ,
TUMBLE_START( oo.ts, INTERVAL '1' MINUTE),
ss.shipments
FROM orders as oo
JOIN shipment AS ss ON oo.orderid = ss.orderid
GROUP BY oo.orderid ,
TUMBLE(CAST(oo.ts AS TIME) ,INTERVAL '1' MINUTE) ,
ss.shipments
Error Message :
SQL validation failed. From line 2, column 17 to line 2, column 57: Call to auxiliary group function 'TUMBLE_START' must have matching call to group function '$TUMBLE' in GROUP BY clause
i am not sure what exactly needs to be done here any help would be great. looking fwed to hear back from expert
TableException: Rowtime attributes must not be in the input rows of a regular join.
As a workaround you can cast the time attributes of input tables to TIMESTAMP before.
It is not possible for a regular join to have time attributes in its results, because the time attribute could not be well-defined. This is because the rows in a dynamic table must be at least roughly ordered by the time attribute, and there's no way to guarantee this for the result of a regular join (as opposed to an interval join, temporal join, or lookup join).
In versions of Flink before 1.14, the implementation dealt with this by not allowing regular joins to have time attributes in the input tables. While this avoided the problem, it was overly restrictive.
In your case I suggest you rewrite the join as an interval join, so that the output of the join will have a time attribute, making it possible to apply windowing to it.
In the second query, I'm not 100% sure what the problem is, but I suspect the problem is that in one case you are using oo.ts, vs CAST(oo.ts AS TIME) in the other. I think they need to be the same. I don't think Flink's SQL planner is smart enough to figure out what's going on here.
I have 2 tables on orders with the order validity date and in transit stock (stock reaching where the order will be serviced).
(using sample data to simplify for understanding)
I am looking for a final calculation like this in my final table -
have done the calculation till column 4 in power BI
if this was in excel i could have simply done
used_stock(2) = serviced(1) + used_stock(1)
avail_stock(2) = total_qty(2) - used_stock(2)
serviced(2) = min(order(2),avail_stock(2))
My base tables look like this -
order table -
intransit table -
I have done the total_qty measure calculation by finding the cumulative sum of shipment quantity for the dates before selected value of order validity date.
I am trying to do the rest of the measures but ending up in circular references. Is there a way I can do it?
edit -
Clarifying it a bit more for the logic needed -
let's say the 2nd order is 15 and the 2nd shipment reaches on 24th, then the base data and output table should look like this -
With present proposed solution the table will erroneously look like -
Try with these below measures-
total_qty =
VAR current_row_date = MIN('order'[order valid till date])
RETURN
CALCULATE(
SUM(intrasit[quantity in shipment]),
FILTER(
ALL(intrasit),
intrasit[expected date of reaching] < current_row_date
)
)
used_stock =
VAR current_row_date = MIN('order'[order valid till date])
RETURN
CALCULATE(
SUM('order'[order quantity]),
FILTER(
ALL('order'),
'order'[order valid till date] < current_row_date
)
) + 0
avail_stock = [total_qty] - [used_stock]
serviced =
IF(
MIN('order'[order quantity]) <= [avail_stock],
MIN('order'[order quantity]),
[avail_stock]
)
Here is your final output-
I want to create a report in power Bi that shows the turnover for each year but in my database a turnover can not be equal to the total amount of the bills because there can be assets to be deducted. so in my database there is operation_code table which contains a SENS column this column indicates if this operation is either input or output and the turnover will be calculated as follows: if (CODE_OP.SENS = output), + sum (ENT.HTMT / ENT .DEVP) else if (CODE_OP.SENS = input), - sum (ENT.HTMT / ENT.DEVP)
i did a test with SQL Statements :
SELECT YEAR(ENT.PIDT) ,T020.SENS,
CASE
WHEN T020.SENS= 2 THEN
sum (ENT.HTMT / ENT.DEVP)
WHEN T020.SENS = 1 THEN -1 * sum (ENT.HTMT / ENT.DEVP)
END as CA FROM ENT
left join T020 on
T020.OP = ENT.OP
WHERE ENT.PICOD=4 AND ENT.TICOD='C' AND ENT.DOS=998
GROUP BY YEAR(ENT.PIDT)--,T020.SENS
but when i try to do it in power BI Using DAX by creating a measure
Measure =
Switch ( True();
ALL(T020[SENS])=2,0;SUM(ENT[HTMT])/SUM(ENT[DEVP]) ;
ALL(T020[SENS])=1,0; SUM(ENT[HTMT])/(SUM(ENT[DEVP]))
)
an error appear when i try to use the measure that it is impossible to display the visual element . enter image description here
The following DAX should do the trick:
Measure =
CALCULATE(
SUMX(ENT, DIVIDE(ENT[HTMT], ENT[DEVP])),
T020[SENS] = 2
)
- CALCULATE(
SUMX(ENT, DIVIDE(ENT[HTMT], ENT[DEVP])),
T020[SENS] = 1
)
I am writing a query which will be used to populate a report used for putting away stock in a warehouse.
The report has 3 parameters, a source stock location, source bin number and a destination stock location.
The stock will be currently held in source stock location and bin number.
The destination stock location is where the stock needs to be moved to.
Each Variant has a default bin number in each stock location.
Multiple variants can have the same default bin number.
Bin numbers may not be alphabetical around the warehouse, and so each bin number is assigned a walk route, as the most efficient walking route around the warehouse.
The report will look at the default bin number associated with the item in the destination stock location, and if empty, offer that as the put away suggestion.
If the default bin number is not empty, it will look for the next available empty bin in the destination stock location (where walk route is higher than default bin), and then offer that as the put away suggestion.
The query works fine, and does exactly that, however it is reporting the same bin as previous "NextBinNo" suggestions in rows above.
How can I get the OUTER APPLY NextBinNo to filter out any previously suggested bins in higher rows of data? Also if two items have the same default bin number, it should use the NextBinNo for the second row with this default bin no.
My current query:
Select
row_number() Over(Order by DestSL.sl_id) as RowNo,
Stock_location.sl_name,
bin_number.bn_bin_number,
variant_detail.vad_variant_code,
variant_detail.vad_description,
variant_transaction_header.vth_current_quantity,
variant_transaction_header.vth_batch_number,
purchase_order_header.poh_order_number,
supplier_detail.sd_ow_account,
DestSL.sl_id as 'DestinationSLID',
DestSL.sl_name as 'DestinationStockLocation',
DestDefaultBin.bn_bin_number as 'DestinationDefaultBin',
DestDefaultBin.bn_walk_route as 'DestinationDefaultWalkRoute',
isnull(DestDefaultBinQty.BinQty,0) as QtyInDefaultBin,
NextBinNo.NextBinNo as 'NextBinNo',
NextBinNo.NextWalkRoute as 'NextBinWalkRoute',
isnull(NextBinNo.BinQty,0) as 'NextBinQty',
case when DestDefaultBin.bn_bin_number is null
then 'Not Stocked in This Location'
Else
case when isnull(DestDefaultBinQty.BinQty,0) > 0
then
case when NextBinNo.NextBinNo is NULL
then 'No Free Bin'
Else NextBinNo.NextBinNo
End
Else DestDefaultBin.bn_bin_number
End
End as 'Put Away Destination'
From variant_transaction_header
join bin_number on bin_number.bn_id = variant_transaction_header.vth_bn_id
join stock_location on stock_location.sl_id = variant_transaction_header.vth_sl_id
join variant_detail on variant_detail.vad_id = variant_transaction_header.vth_vad_id
join transaction_type on transaction_Type.tt_id = variant_transaction_header.vth_tt_id
left join purchase_order_line on purchase_order_line.pol_id = variant_transaction_header.vth_pol_id
left join purchase_order_header on purchase_order_header.poh_id = purchase_order_line.pol_poh_id
left join supplier_detail on supplier_detail.sd_id = purchase_order_header.poh_sd_id
join stock_location DestSL on DestSL.sl_id = #DestinationStockLoc
left join variant_stock_location DestVSL on DestVSL.vsl_vad_id = variant_detail.vad_id and DestVSL.vsl_sl_id = DestSL.sl_id
left join bin_number DestDefaultBin on DestDefaultBin.bn_id = DestVSL.vsl_bn_id
left join
(select sum(variant_transaction_header.vth_current_quantity) as BinQty,
variant_transaction_header.vth_bn_id
from variant_transaction_header
join transaction_type on transaction_Type.tt_id = variant_transaction_header.vth_tt_id
Where variant_transaction_header.vth_current_quantity > 0
and transaction_type.tt_transaction_type = 'IN' and transaction_Type.tt_update_current_qty = 1
Group by variant_transaction_header.vth_bn_id) as DestDefaultBinQty on DestDefaultBinQty.vth_bn_id = DestDefaultBin.bn_id
Outer Apply
(select top 1
row_number() Over(Order by NextBin.bn_bin_number) as RowNo,
NextBin.bn_bin_number as NextBinNo,
NextBin.bn_walk_route as NextWalkRoute,
BinQty.BinQty
from
Stock_location DestSL
Join bin_number NextBin on NextBin.bn_sl_id = DestSL.sl_id
left join
(select sum(variant_transaction_header.vth_current_quantity) as BinQty,
variant_transaction_header.vth_bn_id
from variant_transaction_header
join transaction_type on transaction_Type.tt_id = variant_transaction_header.vth_tt_id
Where variant_transaction_header.vth_current_quantity > 0
and transaction_type.tt_transaction_type = 'IN' and transaction_Type.tt_update_current_qty = 1
Group by variant_transaction_header.vth_bn_id) as BinQty on BinQty.vth_bn_id = NextBin.bn_id
Where NextBin.bn_sl_id = #DestinationStockLoc
and NextBin.bn_walk_route > DestDefaultBin.bn_walk_route
And isnull(BinQty.BinQty,0) = 0
order by NextBin.bn_walk_route, nextbin.bn_bin_number) as NextBinNo
where variant_transaction_header.vth_current_quantity > 0
and transaction_type.tt_transaction_type = 'IN' and transaction_Type.tt_update_current_qty = 1
and stock_location.sl_id = #SourceStockLoc and bin_number.bn_id = #SourceBinNo
You can see my current results below:
Row2 is using the NextBinNo as the default bin has stock.
Row3 is also suggesting using AA08A2 as the next bin.
Row 6 is currently suggesting AA01A2 but that has already been suggested in Row1.
Just to answer this, in the end I could not achieve what I wanted directly in SQL.
The data is ultimately being returned to a report writer which can run Visual Basic code.
I had to execute the NextBinNo sub query separately in VB.
In VB I can define a string which contains a list of all the "used" bin numbers and so on each row this is referenced in the WHERE of the query to check it has not been used in a row above.
I could then return this value as a new column dynamically inserted in the dataset at run time.
I am trying to model the following relationship with the intent of designing classes for EF code first.
Program table:
ProgramID - PK
ProgramName
ClusterCode
Sample data
ProgramID ProgramName ClusterCode
--------------------------------------
1 Spring A
2 Fall A
3 Winter B
4 Summer B
Cluster table:
ID
ClusterCode
ClusterDetails
Sample data:
ID ClusterCode ClusterDetails
---------------------------------
1 A 10
2 A 20
3 A 30
4 B 20
5 B 40
I need to join the Program table to the Cluster table so I can get the list of cluster details for each program.
The SQL would be
Select
from Programs P
Join Cluster C On P.ClusterCode = C.ClusterCode
Where P.ProgramID = 'xxx'
Note that for the Program table, ClusteCode is not unique.
For Cluster table, neither ClusterCode nor ClusterDetail is unique.
How would I model this so I can take advantage of navigation properties and code-first?
assuming you have mapped above two tables and make an association between them and you are using C#, you can use a simple join :
List<Sting> clustedDets=new ArrayList<String>();
var q =
from p in ClusterTable
join c in Program on p equals c.ClusterTable
select new { p.ClusterDetails };
foreach (var v in q)
{
clustedDets.Add(v.ClusterDetails);
}