Obtaining keys and values from JSON nested array in nest - arrays

Question

First time posting! I am converting JSON data (dictionary) from a server into a csv file. The keys and values taken are fine apart from the nest "Astronauts", which is an array. Basically every individual JSON string is a datum that may contains from 0 to an unlimited number of astronauts which features I would like to extract as independent values. For instance something like this:
Astronaut1_Spaceships_First: Katabom
Astronaut1_Spaceships_Second: The Kraken
Astronaut1_name: Jebeddia
(...)
Astronaut2_gender: Hopefully female
and so on. The problem here is that the nest is set as an array and not a dictionary so I do not know what to do. I have tried the dpath library as well as flattering the nest but nothing did change. Any ideas?
import json
import os
import csv
import datetime
import dpath.util #Dpath library needs to be installed first
datum = {"Mission": "Make Earth Greater Again", "Objective": "Prove Earth is flat", "Astronauts": [{"Spaceships": {"First": "Katabom", "Second": "The Kraken"}, "Name": "Jebeddiah", "Gender": "Hopefully male", "Age": 35, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}, {"Spaceships": {"First": "The Kraken", "Second": "Minnus I"}, "Name": "Bob", "Gender": "Hopefully female", "Age": 23, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}]}
#Parsing process
parsed = json.loads(datum) #datum is the JSON string retrieved from the server
def flattenjson(parsed, delim):
val = {}
for i in parsed.keys():
if isinstance(parsed[i], dict):
get = flattenjson(parsed[i], delim)
for j in get.keys():
val[i + delim + j] = get[j]
else:
val[i] = parsed[i]
return val
flattened = flattenjson(parsed,"__")
#process of creating csv file
keys=['Astronaut1_Spaceship_First','Astronaut2_Spaceship_Second', 'Astronaut1_Name] #reduced to 3 keys for this example
writer = csv.DictWriter(OD, keys ,restval='Null', delimiter=",", quotechar="\"", quoting=csv.QUOTE_ALL, dialect= "excel")
writer.writerow(flattened)
.
#JSON DATA FROM SERVER
{
"Mission": "Make Earth Greater Again",
"Objective": "Prove Earth is flat",
"Astronauts": [ {
"Spaceships": {
"First": "Katabom",
"Second": "The Kraken"
},
"Name": "Jebeddiah",
"Gender": "Hopefully male",
"Age": 35,
"Prefered colleages": [],
"Following missions": [
{
"Payment_status": "TO BE CONFIRMED"
}
]
},
{
"Spaceships": {
"First": "The Kraken",
"Second": "Minnus I"
},
"Name": "Bob",
"Gender": "Hopefully female",
"Age": 23,
"Prefered colleages": [],
"Following missions": [
{
"Payment_status": "TO BE CONFIRMED"
}
]
},
]
}
]

Answer 1

Firstly, the datum you have defined here is not the datum that would be extracted from the server. The datum from the server would be a string. The datum you have in this program is already processed. Now, assuming datum to be:
datum = '{"Mission": "Make Earth Greater Again", "Objective": "Prove Earth is flat", "Astronauts": [{"Spaceships": {"First": "Katabom", "Second": "The Kraken"}, "Name": "Jebeddiah", "Gender": "Hopefully male", "Age": 35, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}, {"Spaceships": {"First": "The Kraken", "Second": "Minnus I"}, "Name": "Bob", "Gender": "Hopefully female", "Age": 23, "Prefered colleages": [], "Following missions": [{"Payment_status": "TO BE CONFIRMED"}]}]}'
You don't need the the dpath library. The problem here is that your json flattener doesn't handle embedded lists. Try using the one I've put below.
Assuming that you want a one line csv file,
import json
def flattenjson(data, delim, topname=''):
"""JSON flattener that can handle embedded lists and dictionaries"""
flattened = {}
def internalflat(int_data, name=topname):
if type(int_data) is dict:
for key in int_data:
internalflat(int_data[key], name + key + delim)
elif type(int_data) is list:
i = 1
for elem in int_data:
internalflat(elem, name + str(i) + delim)
i += 1
else:
flattened[name[:-len(delim)]] = int_data
internalflat(data)
return flattened
#If you don't want mission or objective in csv file
flattened_astronauts = flattenjson(json.loads(datum)["Astronauts"], "__", "Astronaut")
keys = flattened_astronauts.keys().sort()
writer = csv.DictWriter(OD, keys ,restval='Null', delimiter=",", quotechar="\"", quoting=csv.QUOTE_ALL, dialect= "excel")
writer.writerow(flattened_astronauts)