I want to combine these two JSON into one using matillion ETL for snowflake
JSON statement 1:
[{
"StepId": 1,
"ParameterFileGroup": 1,
"ParameterGroup": 1,
"Parameter": 4,
"Filter": "",
"SortKey": "",
"Skip": -1,
"ParameterFileGroup": 1,
"ParameterGroup": 1,
JSON statement 2:
{
"ConditionId": 4,
"Threshold": "37",
"ActionPlan": 3,
"TriggerAction": 5
}
the result I want:
[{
"StepId": 1,
"ParameterFileGroup": 1,
"ParameterGroup": 1,
"Parameter": 4,
"Filter": "",
"SortKey": "",
"Skip": -1,
"ParameterFileGroup": 1,
"ParameterGroup": 1,
{
"ConditionId": 4,
"Threshold": "37",
"ActionPlan": 3,
"TriggerAction": 5
}}
]
Assuming these are in 2 columns: JSON1, JSON2. Use the calculator component. add a new calculation called: JSON3. The calculation is: "JSON1" || '\r' || "JSON2" || '\r' || '}]'
Related
I am using mongo DB In which I'm updating a row for multiple types with different payloads and conditions but every time I update the row it overrides the previous one
for the first time the request. data is
request.data: {
"farm_area_count": 1,
"farm_area": [
{
"area_id": 1,
"area_name": "Area 1",
"area_acerage": 4,
"area_structure_type": "polyhouse",
"zone_latest_id": 0
}
]
}
output is
{
"farm_area_count": 1,
"farm_area": [
{
"area_id": 1,
"area_name": "Area 1",
"area_acerage": 4,
"area_structure_type": "polyhouse",
"zone_latest_id": 0
}
]
}
for the second time the request. data is
request.data:
{
"farm_area_count": 1,
"farm_area": [
{
"area_id": 1,
"zone_latest_id": 1,
"zone_name":"test zone",
"zone_acerage":2
}
]
}
the output should be
{
"farm_area_count": 1,
"farm_area": [
{
"area_id": 1,
"area_name": "Area 1",
"area_acerage": 4,
"area_structure_type": "polyhouse",
"zone_latest_id": 1,
"zone_name":"test zone",
"zone_acerage":2
}
]
}
but the output that I'm getting is
{
"farm_area_count": 1,
"farm_area": [
{
"area_id": 1,
"zone_latest_id": 1,
"zone_name":"test zone",
"zone_acerage":2
}
]
}
here is the updated code
collection.update_one({"_id": ObjectId(str(kwargs['pk']))}, {"$set": request.data})
How can I update a value in a document based on applying functions to another field (which is in a different embedded document)?
With the sample data below, I want to
get the col field for the farm having id 12
multiply that by 0.025
add the current value of the statistic.crypt field
ensure the value is a double by converting it with $toDouble
store the result back into statistic.crypt
data:
{
"_id": {
"$oid": "6128c238c144326c57444227"
},
"statistic": {
"balance": 112570,
"diamond": 14,
"exp": 862.5,
"lvl": 76,
"mn_exp": 2.5,
"lvl_mn_exp": 15,
"coll_ms": 8047,
"all_exp": 67057.8,
"rating": 0,
"crypt": 0
},
"inventory": {
"farm": [{
"id": 12,
"col": 100,
"currency": "diamond",
"cost": 2,
"date": "2021-09-02 18:58:39"
}, {
"id": 14,
"col": 1,
"currency": "diamond",
"cost": 2,
"date": "2021-09-02 16:57:08"
}],
"items": []
},
...
}
My initial attempt is:
self.collection
.update_many({"inventory.farm.id": 12}, [{
"$set": {
"test": {
'$toDouble': {
"$sum": [
{'$multiply':["$inventory.farm.$[].col", 0.025]},
'$test'
]
}
} }
},])
This does not work as it applies to test rather than statistic.crypt, and I cannot figure out how to modify it to apply to statistic.crypt.
A field can be updated based on another in the following stages:
add a field containing the farm
set statistic.crypt to the result of the mathematical expression (applied to the newly embedded farm)
remove extra fields
In code:
self.collection.update_many({"inventory.farm.id": 12 }, [
{
$addFields: {
hh: {
$filter: {
input: "$inventory.farm",
as: "z",
cond: { $eq: ["$$z.id", 12] },
},
},
},
},
{
$set: {
"statistic.crypt": {
$toDouble: {
$sum: [
{
$multiply: [{ $first: "$hh.col" }, 0.025],
},
"statistic.crypt",
],
},
},
},
},
{
$project: {
id_pr: 1,
id_server: 1,
role: 1,
warns: 1,
id_clan: 1,
statistic: 1,
design: 1,
date: 1,
inventory: 1,
voice: 1,
},
},)
I'd like a MongoDB query that returns records where the sum of certain attributes satisfies a constraint, for instance given the following documents:
[
{ _id: 1, q1a: 20, q1b: 50, q1c: 30},
{ _id: 2, q1a: 50, q1b: 30, q1c: 20},
{ _id: 3, q1a: 0, q1b: 0, q1c: 0},
]
Id like to run a query that return all and only docs where (q1a + q1b + q1c) == 100, which in this example is records 1 and 2 above.
Is there a way to express this in MongoDB without using $where and writing the sum as a Javascript function?
You can use $expr along with $sum:
db.collection.find({ $expr: { $eq: [ 100, { $sum: [ "$q1a", "$q1b", "$q1c" ] } ] } })
Mongo Playground
I need to convert a data like this:
{peopleList: [{id:1, name: 'joe'}, {id: 2, name: 'john'}], page: 1, rowPerPage: 8}
to this model:
{entities: {'0': {id: 0, name: 'joe'}, '1': {id: 1, name: 'john'}, page: 1, rowPerPage: 8}, result: [0, 1]}
but when I add this schema:
const people = new schema.Entity('peopleList');
const normalizedData = normalize(_data, { peopleList: [people] });
I get this output:
{
"entities": {
"peopleList": {
"1": {
"id": 1,
"name": "joe"
},
"2": {
"id": 2,
"name": "john"
}
}
},
"result": {
"peopleList": [
1,
2
],
"page": 1,
"rowPerPage": 8
}
}
I don't know exactly how to make a proper schema that create result filed as my desire. maybe the correct way is to have it in result and this output is correct. any idea?
{
actName: null,
applicable: {
applicable: [ 5, 4, 1 ]
},
status: 1,
id: 2
}
{
actName: null,
applicable: {
applicable: [ 3, 2 ]
},
status: 1,
id: 1
}
Is that possible to find value in array, like if i search integer value 2 in applicable array return one row with id 1.
with t(j) as (values
('{
"actName": null,
"applicable": {
"applicable": [ 5, 4, 1 ]
},
"status": 1,
"id": 2
}'::jsonb),
('{
"actName": null,
"applicable": {
"applicable": [ 3, 2 ]
},
"status": 1,
"id": 1
}')
)
select j ->> 'id' as id
from t
where exists (
select 1
from jsonb_array_elements_text(j -> 'applicable' -> 'applicable') s(i)
where i = '2'
)
;
id
----
1
With JSONB's #> you can query for any element by following the structure of your document, for example:
WITH data(d) AS (VALUES
('{
"actName": null,
"applicable": {
"applicable": [ 5, 4, 1 ]
},
"status": 1,
"id": 2
}'::JSONB),
('{
"actName": null,
"applicable": {
"applicable": [ 3, 2 ]
},
"status": 1,
"id": 1
}')
)
SELECT d ->> 'id' AS id
FROM data
WHERE d #> '{"applicable":{"applicable":[1]}}';
Assumption here is that you have jsonb with array of json docs in your database, query may be the following:
WITH test_data AS (
SELECT '[{
"actName": "null",
"applicable": {
"applicable": [5,4,1]
},
"status":1,
"id":2
},
{
"actName": "null",
"applicable": {
"applicable": [3,2]
},
"status": 1,
"id": 1
}]'::JSONB AS jsonb_value
)
SELECT
jsonb_doc->>'id' AS id,
jsonb_doc->'applicable' #>'{applicable}' AS appl_array_result
FROM
test_data td,
jsonb_array_elements(td.jsonb_value) AS jsonb_doc
WHERE (jsonb_doc->'applicable' #>'{applicable}') #> '2'::JSONB;
Output:
id | appl_array_result
----+-------------------
1 | [3, 2]
(1 row)