I have recently switched from using ULKJson to SuperObject and I have been looking around at the examples that come with the package and have made some headway with most it it, but it appears as I have come across a snag. To be more specific, I cannot seem to find an example to show how to access item in an array like the one in the example below.
{
"name": "John Smith",
"tel": 555-5555,
"age": 18,
"height": 1.8,
"place": [{"address": "PO Box 1234", "city": "Florida", "code": 2000},
{"address": "1 Sparrow street", "city": "Florida", "code": 2000}]
}
To access the regular items I use the following code which seems to work just fine.
procedure TForm1.Button1Click(Sender: TObject);
var
SO : ISuperObject;
age, height, tel : Integer;
name : String;
begin
SO := TSuperObject.ParseFile('JSON.txt',true);
name := SO.S['name'];
age := SO.I['age'];
tel := SO.I['tel'];
height := SO.I['height'];
Memo1.Lines.Clear;
Memo1.Lines.Add('Name: ' + name);
Memo1.Lines.Add(#10#13);
Memo1.Lines.Add('Age: ' + age);
Memo1.Lines.Add(#10#13);
Memo1.Lines.Add('Telephone: ' + tel);
Memo1.Lines.Add(#10#13);
Memo1.Lines.Add('Height: ' + height);
Memo1.Lines.Add(#10#13);
end;
However, I am not sure how to access the items in the Place array and I am sure I am just overlooking something simple, but I could not find any examples in the demos which showed how to access this data and was hoping one of the gurus here might be able to offer some assistance or atleast point me to a guide where I can learn from myself.
The way I would do it would be simply:
var
location:ISuperObject;
begin
for location in SO['place'] do
Memo1.Lines.Add(location.S['address']); //etc.
end;
end;
And as TLama has suggested, the short guide really is a great source to learn from.
Related
I have a Ruby on Rails application to enter results and create a league table for a football competition.
I'm trying to input some results by creating records in the database through heroku and I get error messages.
The application isn't perfectly designed: to enter the results, I have to create the fixtures and enter the score for each team. Then, independently I have to record each goal scorer, creating a record for each goal which is either associated with an existing player or requires me to firstly create a new player and then create the goal.
When I ran the code below heroku, I got this error:
syntax error, unexpected ':', expecting keyword_end
Maybe I'm missing something simple about lopping through an array within a hash?
Thank you for any advice!
coalition = Team.find_by(name: "Coalition")
moscow_rebels = Team.find_by(name: "Moscow Rebels")
red_star = Team.find_by(name: "Red Star")
unsanctionables = Team.find_by(name: "The Unsanctionables")
cavalry = Team.find_by(name: "Cavalry")
galactics = Team.find_by(name: "The Galactics")
happy_sundays = Team.find_by(name: "Happy Sundays")
hardmen = Team.find_by(name: "Hardmen")
international = Team.find_by(name: "International")
evropa = Venue.find_by(name: "Evropa")
s28 = Season.find_by(number: 28)
start_time = DateTime.new(2020,9,6,11,0,0,'+03:00')
scheduled_matches_1 =
[
{team_1: cavalry, team_1_goals: 1, team_1_scorers: ["Minaev"], team_2_goals: 6, team_2_scorers: ["Kovalev", "Kovalev", "Kovalev", "Thomas", "Thomas", "Grivachev"], team_2: coalition, time: start_time, venue: evropa, season: s28},
{team_1: hardmen, team_1_goals: 4, team_1_scorers: ["Jones", "Jones", "Jones", "Fusi"], team_2_goals: 2, team_2_scorers: ["Kazamula", "Ario"], team_2: galactics, time: start_time + 1.hour, venue: evropa, season: s28},
{team_1: international, team_1_goals: 9, team_1_scorers: ["Kimonnen", "Kimonnen", "Kimonnen", "Burya", "Burya", "Zakharyaev", "Zakharyaev", "Lavruk", "Rihter"], team_2_goals: 0, team_2_scorers: [], team_2: happy_sundays, time: start_time+2.hours, venue: evropa, season: s28}
]
scheduled_matches.each do |match|
new_fixture = Fixture.create(time: match[:time], venue: match[:venue], season: match[:season])
tf1 = TeamFixture.create(team: match[:team_1], fixture: new_fixture)
tf2 = TeamFixture.create(team: match[:team_2], fixture: new_fixture)
ts1 = TeamScore.create(team_fixture: tf1, total_goals: match{:team_1_goals})
ts2 = TeamScore.create(team_fixture: tf2, total_goals: match{:team_2_goals})
match[:team_1_scorers].each do |scorer|
if Player.exists?(team: tf1.team, last_name: scorer)
Goal.create(team_score: ts1, player: Player.find_by(last_name: scorer))
else
new_player = Player.create(team: tf1.team, last_name: scorer)
Goal.create(team_score: ts1, player: new_player)
end
end
match[:team_2_scorers].each do |scorer_2|
if Player.exists?(team: tf2.team, last_name: scorer_2)
Goal.create(team_score: ts2, player: Player.find_by(last_name: scorer_2))
else
new_player = Player.create(team: tf2.team, last_name: scorer_2)
Goal.create(team_score: ts2, player: new_player)
end
end
end
It looks like you are using braces when you meant to use brackets to access the hash. Below is one of the issues, but the same issue is in ts2.
ts1 = TeamScore.create(team_fixture: tf1, total_goals: match{:team_1_goals})
should be match[:team_1_goals]
ts1 = TeamScore.create(team_fixture: tf1, total_goals: match[:team_1_goals])
It may be because you have scheduled_matches_1 at the top and scheduled_matches.each do... further down.
But the real issue here is that your variable names match the data content, rather than being used to hold the content. If a new team joins your league, you have to change the code. Next week, you are going to have to change the hard-coded date value. Your scheduled_matches_1 data structure includes the active record objects returned by the first set of Team.findByName() calls. It would be easier to fetch these objects from the database inside your loops, and just hold the team name as a string in the hash.
There is some duplication too. Consider that each fixture has a home team and an away team. Each team has a name, and an array (possibly empty) of the players who scored. We don't need the number of goals; we can just count the number of players in the 'scorers' array. The other attributes, like the location and season belong to the fixture, not the team. So your hash might be better as
{
"fixtures": [
{
"home": {
"name": "Cavalry",
"scorers": [
"Minaev"
]
},
"away": {
"name": "Coalition",
"scorers": [
"Kovalev",
"Kovalev",
"Kovalev",
"Thomas",
"Thomas",
"Grivachev"
]
},
"venue": "Evropa",
"season": "s28"
}
]
}
because then you can create a reusable method to process each team. And maybe create a new method that returns the player (which it either finds or creates) which can be called by the loop that adds the goals.
Also, as it stands, I'm not sure the code can handle 'own goals', either. Perhaps something for a future iteration :)
I am trying to flatten a JSON file to be able to load it into PostgreSQL all in AWS Glue. I am using PySpark. Using a crawler I crawl the S3 JSON and produce a table. I then use an ETL Glue script to:
read the crawled table
use the 'Relationalize' function to flatten the file
convert the dynamic frame to a dataframe
try to 'explode' the request.data field
Script so far:
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = glue_source_database, table_name = glue_source_table, transformation_ctx = "datasource0")
df0 = Relationalize.apply(frame = datasource0, staging_path = glue_temp_storage, name = dfc_root_table_name, transformation_ctx = "dfc")
df1 = df0.select(dfc_root_table_name)
df2 = df1.toDF()
df2 = df1.select(explode(col('`request.data`')).alias("request_data"))
<then i write df1 to a PostgreSQL database which works fine>
Issues I face:
The 'Relationalize' function works well except the request.data field which becomes a bigint and therefore 'explode' doesn't work.
Explode cannot be done without using 'Relationalize' on the JSON first due to the structure of the data. Specifically the error is: "org.apache.spark.sql.AnalysisException: cannot resolve 'explode(request.data)' due to data type mismatch: input to function explode should be array or map type, not bigint"
If I try to make the dynamic frame a dataframe first then I get this issue: "py4j.protocol.Py4JJavaError: An error occurred while calling o72.jdbc.
: java.lang.IllegalArgumentException: Can't get JDBC type for struct..."
I tried to also upload a classifier so that the data would flatten in the crawl itself but AWS confirmed this wouldn't work.
The JSON format of the original file is as follows, that I an trying to normalise:
- field1
- field2
- {}
- field3
- {}
- field4
- field5
- []
- {}
- field6
- {}
- field7
- field8
- {}
- field9
- {}
- field10
# Flatten nested df
def flatten_df(nested_df):
for col in nested_df.columns:
array_cols = [c[0] for c in nested_df.dtypes if c[1][:5] == 'array']
for col in array_cols:
nested_df =nested_df.withColumn(col, F.explode_outer(nested_df[col]))
nested_cols = [c[0] for c in nested_df.dtypes if c[1][:6] == 'struct']
if len(nested_cols) == 0:
return nested_df
flat_cols = [c[0] for c in nested_df.dtypes if c[1][:6] != 'struct']
flat_df = nested_df.select(flat_cols +
[F.col(nc+'.'+c).alias(nc+'_'+c)
for nc in nested_cols
for c in nested_df.select(nc+'.*').columns])
return flatten_df(flat_df)
df=flatten_df(df)
It will replace all dots with underscore. Note that it uses explode_outer and not explode to include Null value in case array itself is null. This function is available in spark v2.4+ only.
Also remember, exploding array will add more duplicates and overall row size will increase. Flattening struct will increase column size. In short, your original df will explode horizontally and vertically. It may slow down processing data later.
Therefore my recommendation would be to identify feature related data and store only those data in postgresql and original json files in s3.
Once you have rationalized the json column, you don't need to explode it. Relationalize transforms the nested JSON into key-value pairs at the outermost level of the JSON document. The transformed data maintains a list of the original keys from the nested JSON separated by periods.
Example :
Nested json :
{
"player": {
"username": "user1",
"characteristics": {
"race": "Human",
"class": "Warlock",
"subclass": "Dawnblade",
"power": 300,
"playercountry": "USA"
},
"arsenal": {
"kinetic": {
"name": "Sweet Business",
"type": "Auto Rifle",
"power": 300,
"element": "Kinetic"
},
"energy": {
"name": "MIDA Mini-Tool",
"type": "Submachine Gun",
"power": 300,
"element": "Solar"
},
"power": {
"name": "Play of the Game",
"type": "Grenade Launcher",
"power": 300,
"element": "Arc"
}
},
"armor": {
"head": "Eye of Another World",
"arms": "Philomath Gloves",
"chest": "Philomath Robes",
"leg": "Philomath Boots",
"classitem": "Philomath Bond"
},
"location": {
"map": "Titan",
"waypoint": "The Rig"
}
}
}
Flattened out json after rationalize :
{
"player.username": "user1",
"player.characteristics.race": "Human",
"player.characteristics.class": "Warlock",
"player.characteristics.subclass": "Dawnblade",
"player.characteristics.power": 300,
"player.characteristics.playercountry": "USA",
"player.arsenal.kinetic.name": "Sweet Business",
"player.arsenal.kinetic.type": "Auto Rifle",
"player.arsenal.kinetic.power": 300,
"player.arsenal.kinetic.element": "Kinetic",
"player.arsenal.energy.name": "MIDA Mini-Tool",
"player.arsenal.energy.type": "Submachine Gun",
"player.arsenal.energy.power": 300,
"player.arsenal.energy.element": "Solar",
"player.arsenal.power.name": "Play of the Game",
"player.arsenal.power.type": "Grenade Launcher",
"player.arsenal.power.power": 300,
"player.arsenal.power.element": "Arc",
"player.armor.head": "Eye of Another World",
"player.armor.arms": "Philomath Gloves",
"player.armor.chest": "Philomath Robes",
"player.armor.leg": "Philomath Boots",
"player.armor.classitem": "Philomath Bond",
"player.location.map": "Titan",
"player.location.waypoint": "The Rig"
}
Thus in your case, request.data is already a new column flattened out from request column and its type is interpreted as bigint by spark.
Reference : Simplify/querying nested json with the aws glue relationalize transform
So, I'm trying to set up check_json.pl in NagiosXI to monitor some statistics. https://github.com/c-kr/check_json
I'm using the code with the modification I submitted in pull request #32, so line numbers reflect that code.
The json query returns something like this:
[
{
"total_bytes": 123456,
"customer_name": "customer1",
"customer_id": "1",
"indices": [
{
"total_bytes": 12345,
"index": "filename1"
},
{
"total_bytes": 45678,
"index": "filename2"
},
],
"total": "765.43gb"
},
{
"total_bytes": 123456,
"customer_name": "customer2",
"customer_id": "2",
"indices": [
{
"total_bytes": 12345,
"index": "filename1"
},
{
"total_bytes": 45678,
"index": "filename2"
},
],
"total": "765.43gb"
}
]
I'm trying to monitor the sized of specific files. so a check should look something like:
/path/to/check_json.pl -u https://path/to/my/json -a "SOMETHING" -p "SOMETHING"
...where I'm trying to figure out the SOMETHINGs so that I can monitor the total_bytes of filename1 in customer2 where I know the customer_id and index but not their position in the respective arrays.
I can monitor customer1's total bytes by using the string "[0]->{'total_bytes'}" but I need to be able to specify which customer and dig deeper into file name (known) and file size (stat to monitor) AND the working query only gives me the status (OK,WARNING, or CRITICAL). Adding -p all I get are errors....
The error with -p no matter how I've been able to phrase it is always:
Not a HASH reference at ./check_json.pl line 235.
Even when I can get a valid OK from the example "[0]->{'total_bytes'}", using that in -p still gives the same error.
Links pointing to documentation on the format to use would be very helpful. Examples in the README for the script or in the -h output are failing me here. Any ideas?
I really have no idea what your question is. I'm sure I'm not alone, hence the downvotes.
Once you have the decoded json, if you have a customer_id to search for, you can do:
my ($customer_info) = grep {$_->{customer_id} eq $customer_id} #$json_response;
Regarding the error on line 235, this looks odd:
foreach my $key ($np->opts->perfvars eq '*' ? map { "{$_}"} sort keys %$json_response : split(',', $np->opts->perfvars)) {
# ....................................... ^^^^^^^^^^^^^
$perf_value = $json_response->{$key};
if perfvars eq "*", you appear to be looking for $json_reponse->{"{total}"} for example. You might want to validate the user's input:
die "no such key in json data: '$key'\n" unless exists $json_response->{$key};
This entire business of stringifying the hash ref lookups just smells bad.
A better question would look like:
I have this JSON data. How do I get the sum of total_bytes for the customer with id 1?
See https://stackoverflow.com/help/mcve
I have a table with column "Long Description" typically the data looks like the following.
Foundation area wall, 12" H. x 20" W. x 8" projection. Galvanized. Refer to model No. SV208 (SKU 100002) for foundation area wall cover. No. FV208-12: Height: 12", Width: 20", Projection: 8", Type: Foundation Area Wall, Material: Galvanized, Pkg Qty: 1
What I am trying to do is parse out the end attributes. For example after "area wall cover." and beginning with "No." I'd like to extract the following. (Below)
Some things to note. The string '. No.' always begins the attributes in this column. All attributes are separated by columns. The attribute names differ and the amount of attributes per product also differ. Is there a way this can be done with T-SQL?
No. FV208-12:
Height: 12"
Width: 20"
Projection: 8"
Type: Foundation Area Wall
Material: Galvanized
Pkg Qty: 1
You can use a variation of the following to achieve what I believe you're attempting to achieve:
DECLARE #StartAttributesKey VARCHAR(50) = 'area wall cover. ' ,
#LongDescription VARCHAR(MAX) = 'Foundation area wall, 12" H. x 20" W. x 8" projection. Galvanized. Refer to model No. SV208 (SKU 100002) for foundation area wall cover. No. FV208-12: Height: 12", Width: 20", Projection: 8", Type: Foundation Area Wall, Material: Galvanized, Pkg Qty: 1';
SELECT REPLACE(SUBSTRING(#LongDescription, CHARINDEX(#StartAttributesKey, #LongDescription, 0) + LEN(#StartAttributesKey),
LEN(#LongDescription) - CHARINDEX(#StartAttributesKey, #LongDescription, 0)), ',', CHAR(10));
Using this in a query would be similar to:
DECLARE #StartAttributesKey VARCHAR(50) = 'area wall cover. '
SELECT REPLACE(SUBSTRING(LongDescription, CHARINDEX(#StartAttributesKey, LongDescription, 0) + LEN(#StartAttributesKey),
LEN(LongDescription) - CHARINDEX(#StartAttributesKey, LongDescription, 0)), ',', CHAR(10))
FROM [someTable] WHERE ID = 1
If you copy (or print) the result, you will see each attribute on a separate line.
I just started to play with Angular.js and have a question about ngOptions: Is it possible to label the optgroup?
Lets assume 2 objects - cars and garages.
cars = [
{"id": 1, "name": "Diablo", "color": "red", "garageId": 1},
{"id": 2, "name": "Countach", "color": "white", "garageId": 1},
{"id": 3, "name": "Clio", "color": "silver", "garageId": 2},
...
]
garages = [
{"id": 1, "name": "Super Garage Deluxe"},
{"id": 2, "name": "Toms Eastside"},
...
]
With this code I got nearly the result I want:
ng-options = "car.id as car.name + ' (' + car.color + ')' group by car.garageId for car in cars"
Result in the select:
-----------------
1
Diablo (red)
Countach (white)
Firebird (red)
2
Clio (silver)
Golf (black)
3
Hummer (silver)
-----------------
But I want to label the optgroups like "Garage 1", "Garage 2", ... or even better display the name of the garage and not just the garageId.
The angularjs.org documentation for select says nothing about labels for the optgroup, but I would like to extend the group by part of ngOptions like group by car.garageId as 'Garage ' + car.garageId or group by car.garageId as getGarageName(car.garageId) - which sadly is not working.
My only solution so far is to add a new property "garageDisplayName" to the car objects and store there the id + garage name and use that as group by parameter. But I don't want to update all cars whenever a garage name is changed.
Is there a way to label the optgroups with ngOptions, or should I use ngRepeat in that case?
You can just call getGarageName() in the group by without using an as...
ng-options="car.id as car.name + ' (' + car.color + ')' group by getGarageName(car.garageId) for car in cars"
Instead of storing the garage id in each car, you might want to consider storing a reference to the garage object in the garages array. That way you can change the garage name and there is no need to change each car. And the group by simply becomes...
group by car.garage.name