I get the below data from the 'World of Tanks' API and would like to show it in Power BI. The challenge is the arrays I get for damage and piercing_power. I would like to show 3 rows from this data with the damage and piercing power next to each other, how would I do that?
"4": {
"nation_i18n": "U.S.S.R.",
"name": "_76mm_L-11",
"price_gold": 0,
"level": 4,
"damage": [
110,
110,
156
],
"nation": "ussr",
"rate": 12.27,
"price_credit": 25990,
"tanks": [
1,
2049
],
"module_id": 4,
"name_i18n": "76 mm L-11",
"piercing_power": [
68,
75,
38
],
"turrets": [
259,
3,
3843
]
}
After converting to a Table and Expanding, I would Add a Column and use the Table.FromColumns function to construct a nested table on each row from the damage and piercing_power list-type columns. Here's my attempt at that:
= Table.FromColumns ( { [damage] , [piercing_power] } , { "damage" , "piercing_power" } )
That table can then be expanded to get your required output.
Here's my full script:
let
Source = Json.Document(File.Contents("C:\Users\mikeh\Downloads\WOT.json")),
#"Converted to Table" = Record.ToTable(Source),
#"Expanded Value" = Table.ExpandRecordColumn(#"Converted to Table", "Value", {"nation_i18n", "name", "price_gold", "level", "damage", "nation", "rate", "price_credit", "tanks", "module_id", "name_i18n", "piercing_power", "turrets"}, {"nation_i18n", "name", "price_gold", "level", "damage", "nation", "rate", "price_credit", "tanks", "module_id", "name_i18n", "piercing_power", "turrets"}),
#"Added MyTable" = Table.AddColumn(#"Expanded Value", "MyTable", each Table.FromColumns ( { [damage] , [piercing_power] } , { "damage" , "piercing_power" } )),
#"Expanded MyTable" = Table.ExpandTableColumn(#"Added MyTable", "MyTable", {"damage", "piercing_power"}, {"damage.1", "piercing_power.1"}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded MyTable",{{"name", "name.1"}})
in
#"Renamed Columns"
Related
As a part of an hourly process, we source data from a table in RDS and ingest into the raw data layer Snowflake.
After ingesting, we have to update certain fields in the final mart table based on new data ingested.
This mart table is a very wide table with more than 100 fields and the size of the table is around 10 GB.
So these updates are taking 6-7 minutes.
Edit: Update Query and Explain Plan:
update <target_table> tgt
set email_id =src.email_id,
first_name = (
case
when len(src.updated_first_name) < 2 then first_name
when src.updated_first_name is null or src.updated_first_name = '' then first_name
else src.updated_first_name
end ),
last_name = (
case
when len(src.updated_last_name) < 2 then last_name
when src.updated_last_name is null or src.updated_last_name =
'' then last_name
else src.updated_last_name
end),
link = src._link,
title = src.updated_title,
is_verified = 1,
last_verified = src.verified_time,
last_active_type_update = src.verified_time,
active_type = 'ENGAGED',
title_id = src.title_id,
function = src.function,
role = src.role
from <sourcetable> src
where lower(tgt.email_id) = lower(src.email_id)
and src.demographic_disposition in ('Updated Last Name','Updated
Title','Updated First Name','Verified','Updated
Country','Partial
Verify')
and src.verified_time > '<last_runtime>';
Explain Plan:
{
"GlobalStats": {
"partitionsTotal": 728,
"partitionsAssigned": 486,
"bytesAssigned": 9182406144
},
"Operations": [
[
{
"id": 0,
"operation": "Result",
"expressions": [
"number of rows updated",
"number of multi-joined rows updated"
]
},
{
"id": 1,
"parent": 0,
"operation": "Update",
"objects": [
"<target_table>"
]
},
{
"id": 2,
"parent": 1,
"operation": "InnerJoin",
"expressions": [
"joinKey: (LOWER(src.EMAIL_ID) = LOWER(tgt.EMAIL_ID))"
]
},
{
"id": 3,
"parent": 2,
"operation": "Filter",
"expressions": [
"(src.DEMOGRAPHIC_DISPOSITION IN 'Updated Last Name' IN 'Updated Job Title' IN 'Updated First Name' IN 'Verified' IN 'Updated Country' IN 'Partial Verify') AND (src.VERIFIED_TIME > '<last_runtime>') AND (LOWER(src.EMAIL_ID) IS NOT NULL)"
]
},
{
"id": 4,
"parent": 3,
"operation": "TableScan",
"objects": [
"<src_table>"
],
"expressions": [
"EMAIL_ID",
"LINK",
"UPDATED_FIRST_NAME",
"UPDATED_LAST_NAME",
"UPDATED_TITLE",
"DEMOGRAPHIC_DISPOSITION",
"VERIFIED_TIME",
"FUNCTION",
"ROLE",
"TITLE_ID"
],
"alias": "BH",
"partitionsAssigned": 1,
"partitionsTotal": 243,
"bytesAssigned": 1040384
},
{
"id": 5,
"parent": 2,
"operation": "Filter",
"expressions": [
"LOWER(tgt.EMAIL_ID) IS NOT NULL"
]
},
{
"id": 6,
"parent": 5,
"operation": "JoinFilter",
"expressions": [
"joinKey: (LOWER(src.EMAIL_ID) = LOWER(tgt.EMAIL_ID))"
]
},
{
"id": 7,
"parent": 6,
"operation": "TableScan",
"objects": [
"<target_table>"
],
"expressions": [
"EMAIL_ID",
"FIRST_NAME",
"LAST_NAME"
],
"partitionsAssigned": 485,
"partitionsTotal": 485,
"bytesAssigned": 9181365760
}
]
]
}
Now the new requirement is that this update jobs run every 5 minutes so that we can have near real time updated data in Snowflake.
However, we have tried all query optimisation and are unable to bring the execution time of the updates to less than 5 minutes.
We are using a Small warehouse in Snowflake and due to budget constraints we cant increase it further to fasten the update query performance.
Is their any other budget friendly way to do near real time updates(after ingesting the data) in Snowflake?
[
{
"id": 200,
"date_created": "2021-01-14T17:15:55",
"sale": "2500.00",
},
{
"id": 201,
"date_created": "2021-01-15T15:10:30",
"sale": "2000.00",
},
{
"id": 202,
"date_created": "2021-02-4T11:14:10",
"sale": "4000.00",
}
]
I am unable to sum it by the monthly basis in react js or next js.
Can someone guide me on how can I do it in React?
I want the output as:
monthly basis.
For Jan total sum = 4500
For Feb total sum = 4000
etc...
You have to browse your array and to add each sale.
Example :
const a = [
{
"id": 200,
"date_created": "2021-01-14T17:15:55",
"sale": "2500.00",
},
{
"id": 201,
"date_created": "2021-01-15T15:10:30",
"sale": "2000.00",
},
{
"id": 202,
"date_created": "2021-02-4T11:14:10",
"sale": "4000.00",
}
]
const sum = a.reduce((previous, current) => previous + new Number(current.sale), 0)
console.log(sum)
I want to group the data based on the type and type_id
Here is the array
var addArray = [
{
"id": 24,
"language_id": 3,
"type": "service",
"type_id": 2,
"key": "service seeker",
"value": " need service"
},
{
"id": 23,
"language_id": 3,
"type": "service",
"type_id": 2,
"key": "phone",
"value": "phone number"
},
{
"id": 24,
"language_id": 3,
"type": "service",
"type_id": 7,
"key": "tester",
"value": "service tester"
}
{
"id": 19,
"language_id": 3,
"type": "offer",
"type_id": 4,
"key": "source",
"value": "resource"
}
]
I have tried let result = _.groupBy(addArray,'type') it is grouping the data based on type but I need to group by type as well as type_id
Expected output
If you need the a flat grouping based on two or more properties, use the _.groupBy() callback to combine the properties to a string:
const addArray = [{"id":24,"language_id":3,"type":"service","type_id":2,"key":"service seeker","value":" need service"},{"id":23,"language_id":3,"type":"service","type_id":2,"key":"phone","value":"phone number"},{"id":24,"language_id":3,"type":"service","type_id":7,"key":"tester","value":"service tester"},{"id":19,"language_id":3,"type":"offer","type_id":4,"key":"source","value":"resource"}]
const result = _.groupBy(addArray, o => `${o.type}-${o.type_id}`)
console.log(result)
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.11/lodash.min.js"></script>
If you need a multi level grouping, start by grouping by the type, then map the groups with _.values(), and group them again by type_id:
const { flow, partialRight: pr, groupBy, mapValues } = _
const fn = flow(
pr(groupBy, 'type'),
pr(mapValues, g => groupBy(g, 'type_id'))
)
const addArray = [{"id":24,"language_id":3,"type":"service","type_id":2,"key":"service seeker","value":" need service"},{"id":23,"language_id":3,"type":"service","type_id":2,"key":"phone","value":"phone number"},{"id":24,"language_id":3,"type":"service","type_id":7,"key":"tester","value":"service tester"},{"id":19,"language_id":3,"type":"offer","type_id":4,"key":"source","value":"resource"}]
const result = fn(addArray)
console.log(result)
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.11/lodash.min.js"></script>
I've a table in postgres named: op_user_event_data, which has a column named data, where I store a jsonb, and what I have at the moment is a json like this:
{
"aisles": [],
"taskGroups": [
{
"index": 0,
"tasks": [
{
"index": 1,
"mandatory": false,
"name": "Dados de Linear",
"structuresType": null,
"lines": [
{
"sku": {
"skuId": 1,
"skuName": "Limiano Bola",
"marketId": [
1,
3,
10,
17
],
"productId": 15,
"brandId": [
38,
44
]
},
"taskLineFields": [
{
"tcv": {
"value": "2126474"
},
"columnType": "skuLocalCode",
"columnId": 99
},
{
"tcv": {
"value": null
},
"columnType": "face",
"columnId": 29
},
]
},
{
"sku": {
"skuId": 3,
"skuName": "Limiano Bolinha",
"marketId": [
1,
3,
10,
17
],
"productId": 15,
"brandId": [
38,
44
]
},
"taskLineFields": [
{
"tcv": {
"value": "2545842"
},
"columnType": "skuLocalCode",
"columnId": 99
},
{
"tcv": {
"value": null
},
"columnType": "face",
"columnId": 29
},
]
},
{
"sku": {
"skuId": 5,
"skuName": "Limiano Bola 1/2",
"marketId": [
1,
3,
10,
17
],
"productId": 15,
"brandId": [
38,
44
]
},
"taskLineFields": [
{
"tcv": {
"value": "5127450"
},
"columnType": "skuLocalCode",
"columnId": 99
},
{
"tcv": {
"value": "5.89"
},
"columnType": "rsp",
"columnId": 33
}
]
}
Basically I've an object which has
Aisles [],
taskGroups,
id and name.
Inside the taskGroups as shown in the json, one of the atributes is tasks which is an array, that also have an array called lines which have an array of sku and tasklines.
Basically:
taskGroups -> tasks -> lines -> sku or taskLineFields.
I've tried different queries to get the sku but when I try to get anything further than 'lines' it just came as blank or in some other tries 'cannot call elements from scalar'
Can anyone help me with this issue? Note this is just a sample json.
Anyone knows how make this to work:
I Want all lines where lines->taskLineFields->columnType = 'offer'
All I can do is this, but throwing error on scalar:
SELECT lines->'sku' Produto, lines->'taskLineFields'->'tcv'->>'value' ValorOferta
FROM sd_bel.op_user_event_data,
jsonb_array_elements(data->'taskGroups') taskgroups,
jsonb_array_elements(taskgroups->'tasks') tasks,
jsonb_array_elements(tasks->'columns') columns,
jsonb_array_elements(tasks->'lines') lines
WHERE created_by = 'belteste'
AND lines->'taskLineFields'->>'columnType' = 'offer'
say your data is in some json_column in your table
with t as (
select json_column as xyz from table
),
tg as ( select json_array_elements(xyz->'taskGroups') taskgroups from t),
tsk as (select json_array_elements(taskgroups->'tasks') tasks from tg)
select json_array_elements(tasks->'lines') -> 'sku' as sku from tsk;
I have tried to unnest the JSON array with the function json_array_elements() and tried to count the elements of the array using json_array_length(field_name) not being successful. I am using PostgreSQL 9.4.5.
I was looking to query the result for the element "name" this is the data held on the json type array field crew:
[
{
"workHours": "9",
"workers": "50",
"checker_rate": 100,
"rate": 150,
"name": "Ramona",
"last": null,
"boxRate": 2,
"checkTraining": false,
"editing": true,
"ix": 0,
"breakPay": 3.0833333333333335,
"trainingPay": 0
},
{
"workHours": "4",
"workers": "50",
"checker_rate": 120,
"rate": 160,
"name": "Ramon",
"last": "Rosas",
"boxRate": 2,
"checkTraining": false,
"editing": false,
"id": 1,
"breakPay": 1.5416666666666667,
"trainingPay": 0
}
]
Your problem stems from the incorrect use of the type json[]. A json array is a single json object and its type is json, not json[]. Example:
create table test (id int, crew json);
insert into test values
(1, '
[
{
"workHours": "9",
"workers": "50",
"checker_rate": 100,
"rate": 150,
"name": "Ramona",
"last": null,
"boxRate": 2,
"checkTraining": false,
"editing": true,
"ix": 0,
"breakPay": 3.0833333333333335,
"trainingPay": 0
},
{
"workHours": "4",
"workers": "50",
"checker_rate": 120,
"rate": 160,
"name": "Ramon",
"last": "Rosas",
"boxRate": 2,
"checkTraining": false,
"editing": false,
"id": 1,
"breakPay": 1.5416666666666667,
"trainingPay": 0
}
]');
The function json_array_elements() works as expected:
select id, elem->'name' as name
from test
cross join json_array_elements(crew) elem;
id | name
----+----------
1 | "Ramona"
1 | "Ramon"
(2 rows)
One of the queries (or both) should work well with json[]:
select id, elem->'name' as name
from test
cross join json_array_elements(crew[1]) elem;
select id, elem->'name' as name
from test
cross join unnest(crew)
cross join json_array_elements(unnest) elem;