Related
I am transforming JSON objects and index them in Solr 9. When resolving lists / arrays of objects, I am using nested child documents, so array elements are stored as own documents.
Now, I ran into the issue that I would like to use copy fields on nested child documents and store this value in the parent.
JSON:
{
"legalName": "Some Name",
"addresss": {
"street": "Bala Street",
"houseNr": 13,
"city": "Random City",
"postalCode": 1234,
"country": "NL"
},
"otherLegalNames": [
{
"text": "TEXT IN EN",
"lang": "EN"
},
{
"text": "TEXT IN DE",
"lang": "DE"
},
{
"text": "TEXT IN NL",
"lang": "NL"
}
]
}
When indexing this object, I flatten basic structs like address but keep arrays, e.g., otherLegalNames, and stored them through child docs.
Basically, the documents look like this (q=*:*&fl=*,[child]):
{
"id": "5493006O42CR4SHELP26",
"legalName": "Some Name",
"addresss.street": "Bala Street",
"addresss.houseNr": 13,
"addresss.city": "Random City",
"addresss.postalCode": 1234,
"addresss.country": "NL",
"otherLegalNames": [
{
"id": "5493006O42CR4SHELP26/otherLegalNames#0",
"text": "TEXT IN EN",
"lang": "EN"
},
{
"id": "5493006O42CR4SHELP26/otherLegalNames#1",
"text": "TEXT IN DE",
"lang": "DE"
},
{
"id": "5493006O42CR4SHELP26/otherLegalNames#2",
"text": "TEXT IN NL",
"lang": "NL"
}
]
}
Now I would like to search for these docs by their legalName and must therefore search in the parent legalName field but also include all text fields stored under otherLegalNames. During research, I found that copy fields are the way to go, but I am not sure how I would handle child documents with such copy fields.
My goal would be to get a searchableLegalNames field with the value: ["Some Name", "TEXT IN EN", "TEXT IN DE", "TEXT IN NL"] or similar to perform a Ngram based search on legalName including every language.
Is this possible to achieve with copy fields or are child documents not supported for this purpose? If so, how should I restructure my schema? It's really hard to flatten every legal name, as this array might be empty or contain an arbitrary number of otherLegalNames.
Thanks.
Regards Artur
i'm new to Wagtail and struggling to create content programmatically from a Django management command.
I have defined a StructBlock and Page with a StreamField like so:
class TaskBlock(blocks.StructBlock):
tasks = blocks.ListBlock(
blocks.StructBlock(
[
("title", blocks.CharBlock()),
("length", blocks.IntegerBlock())
]
)
)
class Meta:
template = "tasks/task_block.html"
icon = "edit"
label = "Task"
class TaskPage(Page):
tasks_first = StreamField([("tasks_first", TaskBlock())], null=True)
content_panels = Page.content_panels + [StreamFieldPanel("tasks_first")]
My command looks like:
class Command(BaseCommand):
def handle(self, *args, **options):
parent_page = Page.objects.get(title="Task Master").specific
task_page = TaskPage(
title="Task Page 5",
slug="task-page-5",
tasks_first=json.dumps(
[
{
"type": "tasks_first",
"value": [
{
"type": "tasks",
"value": [
{"type": "title", "value": "this is a task"},
{"type": "length", "value": 5}
],
}
],
}
]
),
)
parent_page.add_child(instance=task_page)
revision = task_page.save_revision()
revision.publish()
task_page.save()
The page is successfully created with a Task block, but the task is blank and does not contain the test data defined in the json.
I'm probably missing some basic knowledge on how to define the json for the TaskBlock() in the StreamField tasks_first.
Could you give me some guidance?
Probably clearer to start from the inside and work outwards here...
The JSON representation of a StructBlock value is a simple dict, where the keys are the sub-block names and the values are their values - you don't need "type" and "value" here. So, a value for the innermost StructBlock would look like:
{"title": "this is a task", "length": 5}
The value for ListBlock is a list of these values. If we only want one task in the list, this will be:
[
{"title": "this is a task", "length": 5}
]
TaskBlock is a StructBlock with a single sub-block named "tasks", so again this should be a simple dict:
{
"tasks": [
{"title": "this is a task", "length": 5}
]
}
And finally we place this in the stream - this is where the "type" and "value" come in.
[
{
"type": "tasks_first",
"value": {
"tasks": [
{"title": "this is a task", "length": 5}
]
}
}
]
This, then, is what you pass to json.dumps(...).
Finally, are you sure you need a StreamField at all here? If you just want a page with a list of tasks on, you could do this with an InlinePanel where each child object is one task.
I'm working on a app where I retrieve json data via http, and I have made the listview.builder and all of that. Now I want to make a filter button to show the lists that only have values below a certain integer.
for example:
my list of maps goes something like this
[
{
"name": "jess",
"age": "28",
"job": "doctor"
},
{
"name": "jack",
"age": "30",
"job": "jobless"
},
{
"name": "john",
"age": "24",
"job": "doctor"
},
{
"name": "sara",
"age": "23",
"job": "teacher"
}...etc
]
Now I want to press that filter button and in my listview show only those that are below or above the age of 25.
You can map over it and add your "filter" inside the condition. Since you save age as String you have to parse it to an int:
var filteredList = myMapList.map((e){
int? parsedAge = int.tryParse(e["age"]!);
if(parsedAge != null && parsedAge >= 25){
return e;
}
}).toList();
I got the answer and it was there all the time I just didn't thought about that much.
In case anyone faces the same issue, here's how it worked for me.
list filteredList = peopleDetailsList.where((element) => int.parse(element['age'] < 25).toList();
I'm new to programming (especially JSON format), so please forgive me not for using proper terminology :)
Using Python 3.7 Requests module, I receive a JSON response. To keep things simple, I made an example:
{
"Bob":
{
"Age": "15",
"LastExamGrade": "45",
},
"Jack":
{
"Age": "16",
"LastExamGrade": "58",
}
}
What I would like to do is parse the JSON responses to extract two items from each response/structure and save it to a list like this (I think this is called a tuple of tuples?):
[("Bob","45"),("Jack","58")]
Then, after receiving doing this, I will receive another similar response, such as the following (where the only thing that changed is the exam grade):
{
"Bob":
{
"Age": "15",
"LastExamGrade": "54",
},
"Jack":
{
"Age": "16",
"LastExamGrade": "70",
}
}
I want to also save the name and grade into a tuple of tuples (list).
Lastly, I would like to subtract the first exam score of each person from their last exam score, and save this into a final list, which includes the name, final exam grade, and grade improvement, like this:
[("Bob","54","9"),("Jack","67","12")]
What is the simplest way to do this using Python 3? As for my own research, I've searched all throughout StackOverflow, but couldn't find out how to parse a JSON like mine (For example, in mine, the name is outside of the curly braces), and had difficulty doing math operations for JSON items.
I'd recommend using a dedicated package for calculations like pandas:
first_exam_grades = pd.DataFrame.from_dict(first_exam_results, orient='index').astype(int)
second_exam_grades = pd.DataFrame.from_dict(second_exam_results, orient='index').astype(int)
improvements = second_exam_grades.LastExamGrade.to_frame()
improvements['Improvement'] = second_exam_grades.LastExamGrade - first_exam_grades.LastExamGrade
This will give you something that looks like this:
Now you can output it anyway you'd like
list(zip(*([improvements.index.tolist()] + [improvements[c].values.tolist() for c in improvements])))
This will give you [('Bob', 54, 9), ('Jack', 70, 12)] as you want.
One possible solution, using coroutines. Coroutine receive_message holds up to last two values LastExamGrade from the message for each student and produces list of student name, last grade and improvement over last grade:
json_messages = [
# 1st message:
{
"Bob":
{
"Age": "15",
"LastExamGrade": "45",
},
"Jack":
{
"Age": "16",
"LastExamGrade": "58",
}
},
# 2nd message
{
"Bob":
{
"Age": "15",
"LastExamGrade": "54",
},
"Jack":
{
"Age": "16",
"LastExamGrade": "70",
}
},
# 3nd message (optional)
{
"Bob":
{
"Age": "15",
"LastExamGrade": "14",
},
"Jack":
{
"Age": "16",
"LastExamGrade": "20",
}
}
]
def receive_message():
d, message = {}, (yield)
while True:
for k, v in message.items():
d.setdefault(k, []).append(v['LastExamGrade'])
d[k] = d[k][-2:] # store max last two messages
message = yield [(k, *tuple(v if len(v)==1 else [v[1], str(int(v[1])-int(v[0]))])) for k, v in d.items()]
receiver = receive_message()
next(receiver) # prime coroutine
for msg in json_messages:
print(receiver.send(msg))
Prints:
[('Bob', '45'), ('Jack', '58')]
[('Bob', '54', '9'), ('Jack', '70', '12')]
[('Bob', '14', '-40'), ('Jack', '20', '-50')]
First time posting! I am converting JSON data (dictionary) from a server into a csv file. The keys and values taken are fine apart from the nest "Astronauts", which is an array. Basically every individual JSON string is a datum that may contains from 0 to an unlimited number of astronauts which features I would like to extract as independent values. For instance something like this:
Astronaut1_Spaceships_First: Katabom
Astronaut1_Spaceships_Second: The Kraken
Astronaut1_name: Jebeddia
(...)
Astronaut2_gender: Hopefully female
and so on. The problem here is that the nest is set as an array and not a dictionary so I do not know what to do. I have tried the dpath library as well as flattering the nest but nothing did change. Any ideas?
import json
import os
import csv
import datetime
import dpath.util #Dpath library needs to be installed first
datum = {"Mission": "Make Earth Greater Again", "Objective": "Prove Earth is flat", "Astronauts": [{"Spaceships": {"First": "Katabom", "Second": "The Kraken"}, "Name": "Jebeddiah", "Gender": "Hopefully male", "Age": 35, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}, {"Spaceships": {"First": "The Kraken", "Second": "Minnus I"}, "Name": "Bob", "Gender": "Hopefully female", "Age": 23, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}]}
#Parsing process
parsed = json.loads(datum) #datum is the JSON string retrieved from the server
def flattenjson(parsed, delim):
val = {}
for i in parsed.keys():
if isinstance(parsed[i], dict):
get = flattenjson(parsed[i], delim)
for j in get.keys():
val[i + delim + j] = get[j]
else:
val[i] = parsed[i]
return val
flattened = flattenjson(parsed,"__")
#process of creating csv file
keys=['Astronaut1_Spaceship_First','Astronaut2_Spaceship_Second', 'Astronaut1_Name] #reduced to 3 keys for this example
writer = csv.DictWriter(OD, keys ,restval='Null', delimiter=",", quotechar="\"", quoting=csv.QUOTE_ALL, dialect= "excel")
writer.writerow(flattened)
.
#JSON DATA FROM SERVER
{
"Mission": "Make Earth Greater Again",
"Objective": "Prove Earth is flat",
"Astronauts": [ {
"Spaceships": {
"First": "Katabom",
"Second": "The Kraken"
},
"Name": "Jebeddiah",
"Gender": "Hopefully male",
"Age": 35,
"Prefered colleages": [],
"Following missions": [
{
"Payment_status": "TO BE CONFIRMED"
}
]
},
{
"Spaceships": {
"First": "The Kraken",
"Second": "Minnus I"
},
"Name": "Bob",
"Gender": "Hopefully female",
"Age": 23,
"Prefered colleages": [],
"Following missions": [
{
"Payment_status": "TO BE CONFIRMED"
}
]
},
]
}
]
Firstly, the datum you have defined here is not the datum that would be extracted from the server. The datum from the server would be a string. The datum you have in this program is already processed. Now, assuming datum to be:
datum = '{"Mission": "Make Earth Greater Again", "Objective": "Prove Earth is flat", "Astronauts": [{"Spaceships": {"First": "Katabom", "Second": "The Kraken"}, "Name": "Jebeddiah", "Gender": "Hopefully male", "Age": 35, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}, {"Spaceships": {"First": "The Kraken", "Second": "Minnus I"}, "Name": "Bob", "Gender": "Hopefully female", "Age": 23, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}]}'
You don't need the the dpath library. The problem here is that your json flattener doesn't handle embedded lists. Try using the one I've put below.
Assuming that you want a one line csv file,
import json
def flattenjson(data, delim, topname=''):
"""JSON flattener that can handle embedded lists and dictionaries"""
flattened = {}
def internalflat(int_data, name=topname):
if type(int_data) is dict:
for key in int_data:
internalflat(int_data[key], name + key + delim)
elif type(int_data) is list:
i = 1
for elem in int_data:
internalflat(elem, name + str(i) + delim)
i += 1
else:
flattened[name[:-len(delim)]] = int_data
internalflat(data)
return flattened
#If you don't want mission or objective in csv file
flattened_astronauts = flattenjson(json.loads(datum)["Astronauts"], "__", "Astronaut")
keys = flattened_astronauts.keys().sort()
writer = csv.DictWriter(OD, keys ,restval='Null', delimiter=",", quotechar="\"", quoting=csv.QUOTE_ALL, dialect= "excel")
writer.writerow(flattened_astronauts)