Array of Hashes to JSON File Ruby - arrays

I'm trying to create JSON files dynamically, where the content of each file belongs only to a certain entity.
I have an array of Ruby objects, where each object represents an entity.
entities = [#<Entity:0x0055f364d9cd78 #name="entity-B", #internal_asn_number=64514, #classification_id="B">,#<Entity:0x0055f364d89070 #name="entity-A", #internal_asn_number=64513, #classification_id="A">]
And I have an array of hashes, which contains several rules for each entity and I would like to write these hashes to JSON files, one file per entity.
I have the following code:
array_hashes = [{"rulename"=>"entity-B_source_fqdn_1", "if"=>{"source.fqdn"=>"mail.tec.dsr.entityB.com"}, "then"=>{"event_description.text"=>"B"}}, {"rulename"=>"entity-B_destination_fqdn_1", "if"=>{"destination.fqdn"=>"mail.tec.dsr.entity_B.com"}, "then"=>{"event_description.text"=>"B"}}, {"rulename"=>"entity-A_source_fqdn_1", "if"=>{"source.fqdn"=>"194-65-57-128.entityA.com"}, "then"=>{"event_description.text"=>"A"}}, {"rulename"=>"entity-A_destination_fqdn_1", "if"=>{"destination.fqdn"=>"194-65-57-128.entityA.com"}, "then"=>{"event_description.text"=>"A"}}]
path = "/home/mf370/Desktop/RubyProject/"
file_name = "_url_fqdn.conf"
entities.each do |entity|
File.open(path + entity.name + file_name, "w") do |f|
array_hashes.each do |rule|
if rule['rulename'].match(entity.name)
f.write(JSON.pretty_generate([rule]))
end
end
end
This code works and creates the files dynamically, however the content of the files is not what I was expecting.
This is the output from the code above:
[
{
"rulename": "entity-A_source_fqdn_1",
"if": {
"source.fqdn": "194-65-57-128.entityA.com"
},
"then": {
"event_description.text": "A"
}
}
][
{
"rulename": "entity-A_destination_fqdn_1",
"if": {
"destination.fqdn": "194-65-57-128.entityA.com"
},
"then": {
"event_description.text": "A"
}
}
]
And this is the output that I was looking for:
[
{
"rulename": "entity-A_source_fqdn_1",
"if": {
"source.fqdn": "194-65-57-128.entityA.com"
},
"then": {
"event_description.text": "A"
}
},
{
"rulename": "entity-A_destination_fqdn_1",
"if": {
"destination.fqdn": "194-65-57-128.entityA.com"
},
"then": {
"event_description.text": "A"
}
}
]
Probably this is very simple to solve, but I ran out of ideas on how to solve this, I'm new to Ruby Programming. Thank you for your help!

The problem is that you pack each rule inside an array :
JSON.pretty_generate([rule])
You should create an array of matching rules, and dump it as JSON :
entities.each do |entity|
File.open(File.join(path, entity.name + file_name), "w") do |f|
matching_rules = array_hashes.select{ |rule| rule['rulename'].match(entity.name) }
f.write(JSON.pretty_generate(matching_rules))
end
end

The Problem is that you wrap each rule in an array.
THis should solve the issue:
json_entities = []
entities.each do |entity|
File.open(path + entity.name + file_name, "w") do |f|
array_hashes.each do |rule|
if rule['rulename'].match(entity.name)
json_entities << rule
end
end
end
end
f.write(JSON.pretty_generate(json_entities))

Related

display content-type(Strategy) fields value in the form of array in groovy

I have a snippets of groovy code (inside strategy.groovy file) as shown below in which I want to display the output in the form of desired array.
The content-type Strategy (in crafter) has the following fields:
Title English (title_en_t)
Title French (title_fr_t)
Description English (description_en_t)
Description French (description_fr_t)
Players (players_o)
groovy code (inside strategy.groovy file):
private createStrategyElement(rootStrategy, ctx) {
Element strategy = new DefaultElement("strategy")
strategy.addElement("players_o.first_name_s").text = rootStrategy.players_o.item.component.first_name_s != null ? rootStrategy.players_o.item.component.first_name_s : "" //Line A
strategy.addElement("players_o.last_name_s").text = rootStrategy.players_o.item.component.last_name_s != null ? rootStrategy.players_o.item.component.last_name_s : "" //Line B
return strategy
}
Line A and Line B display the o/p in the following fashion:
current o/p:
players_o.first_name_s: "[John, The]"
players_o.last_name_s: "[Cena, Undertaker]"
PROBLEM STATEMENT:
I am wondering what changes I need to do in the groovy code above at Line A and Line B so that it displays the o/p in this fashion:
DESIRED O/P:
players: [{
"item": [
{
"first_name_s": "John",
"last_name_s": "Cena"
},
{
"first_name_s": "The",
"last_name_s": "Undertaker"
}
]
}]
don't know what is the crafter-cms. and you have not provided the input for your code.
i found DefaultElement and addElement in dom4j library - so i assume it's the one used in crafter-cms
https://dom4j.github.io/javadoc/2.0.1/org/dom4j/tree/DefaultElement.html
you are expecting json in output but dom4j used to work with xml...
however. just a try
private createStrategyElement(rootStrategy, ctx) {
Element strategy = new DefaultElement("strategy")
def players = strategy.addElement('players')
for(i in rootStrategy.players_o.item.component){
def item = players.addElement('item')
item.addElement('first_name_s').text = i.first_name_s
item.addElement('last_name_s').text = i.last_name_s
}
return strategy
}
the following code i used to debug it in standard groovy console:
#Grab(group='org.dom4j', module='dom4j', version='2.1.3')
import org.dom4j.tree.DefaultElement
import org.dom4j.Element
import org.dom4j.io.XMLWriter
import org.dom4j.io.OutputFormat
def rootStrategy=[
players_o:[
item:[
component:[
[
first_name_s:'John',
last_name_s: 'Cena',
],
[
first_name_s:'The',
last_name_s: 'Undertaker'
],
]
]
]
]
private createStrategyElement(rootStrategy, ctx) {
Element strategy = new DefaultElement("strategy")
def players = strategy.addElement('players')
for(i in rootStrategy.players_o.item.component){
def item = players.addElement('item')
item.addElement('first_name_s').text = i.first_name_s
item.addElement('last_name_s').text = i.last_name_s
}
return strategy
}
println new StringWriter().with{w->
new XMLWriter(w, OutputFormat.createPrettyPrint()).write( createStrategyElement(rootStrategy,null) )
w.toString()
}

Merging 2 json files into new json with no duplicates

My dedicated servers are generating 2 laptime arrays and I would like to use a script to merge them into a single, new json file, with duplicate "steamids" removed (and kept grouped together as they still are) and both arrays under a single loggedTimes {} (so I can feed it to a html script that produces laptimes and a leaderboard). In other words, I want the structure to remain.
The first laptime file and the second laptime file go through the following command
jq 'reduce . as $item ({}; . * $item)' laptimes_data_ams.json laptimes_data_kow.json > laptimes.json
to then generate the (badly) merged laptime file.
I can get a file reduced but can't get any further than that. I checked threads by other around here and whenever I try their suggestions the script just refuses to work. Anybody available to lend me a hand in generating a working script to keep this final structure post-merge?
{
"loggedTimes" : {
steamids" : {
"idnumber1" : "name1",
"idnumber2" : "name2"
},
"vehicles" : {
"vehiclenumber1" : {
"laptimes" : {
"idnumber1" : {
"lapTime" : time1,
"logtime" : log1,
"name" : "name 1",
"rank" : rank1,
"refId" : id1,
"vehicleid" : vehiclenumber1,
"wet" : 0
},
"idnumber2" : {
"lapTime" : time2,
"logtime" : log2,
"name" : "name 2",
"rank" : rank2,
"refId" : id2,
"vehicleid" : vehiclenumber1,
"wet" : 0
}
}
}
"vehiclesnumber2" : {
//you get the idea by now
}
}
}
You haven't specified how the merge is to be performed, but one option would be to let the key-value pairs in the second file dominate. In that case, you could write:
jq -n '
input as $one
| input as $two
| ($one + $two)
| .loggedTimes.steamids = ($one.loggedTimes.steamids + $two.loggedTimes.steamids)
' 1.json 2.json
With your input, this produces output from which the following is an extract:
{
"loggedTimes": {
"steamids": {
"76561197960277005": "[DECOCO]koker_SZ",
"76561197960436395": "JOJO",
...
},
"vehicles": {
"-1142039519": {
"lapTimes": {}
},
"-1201605905": {
"lapTimes": {
"76561197984026143": {
"lapTime": 609101,
"logtime": 1606516985,
"name": "Night Dick",
"rank": 1,
"refId": 36032,
"vehicleId": -1201605905,
"wet": 0
}
}
}
...
}
}
}

How to process JSON nested array in Logstash

I have a nested field with arrays in array in JSON like the following:
{
"foo": {
"bar": [
[
"a",
"b"
],
[
"c",
"d"
]
]
}
}
The following is my config file:
input {
file {
codec => "json"
path => "pathtofile"
type => "footype"
start_position => "beginning"
}
}
filter {
json {
source => "message"
remove_field => [ "host", "message", "path" ]
}
}
output {
elasticsearch {
action => "index"
index => "bar"
hosts => [ "http://localhost:9200" ]
}
}
I got the following error:
09:40:47.725 [[main]>worker0] WARN logstash.outputs.elasticsearch -
Failed action. {:status=>400, :action=>["index", {:_id=>nil,
:_index=>"bar", :_type=>"footype", :_routing=>nil},
2017-02-13T01:40:30.387Z myconnection %{message}],
:response=>{"index"=>{"_index"=>"bar", "_type"=>"footype",
"_id"=>"AVo1IN0vK2jgwdCXqZ-q", "status"=>400,
"error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper
[foo.bar] of different type, current_type [long], merged_type
[text]"}}}}
I have a feeling that it's the array problem. I have done some research and know that array is not well supported. But I need to ingest the array in elasticsearch. Is there a way to actually do that?
Any helps will be appreciated.
I solved this by using a ruby filter:
ruby {
code => '
j = 0
for i in event.get("[foo][bar]") do
#i is an array element in the big array
l = 0
for k in i do
event.set("item#" + j.to_s + "#" + l.to_s, k)
l = l + 1
end
j = j + 1
end
'
}
This will eventually produce fields
item#0#0 = "a"
item#0#1 = "b"
item#1#0 = "c"
item#1#1 = "d"

ruby sorting hashes inside array

I have an array of hashes that I would like to sort based on the :reference values. But some of the elements within the array do not have a :reference key and therefore cannot be sorted. Is there a way to ignore this field and just sort the hash elements that contain this key?
I've tried the following approach but I'm getting an argument error
ArgumentError: comparison of NilClass with String failed
sort_by at org/jruby/RubyEnumerable.java:503
arr1 = [{:reference=> "F123",
:name=> "test4"
},
{
:reference=> "ZA4",
:name=> "test3"
},
{
:reference=> "A43",
:name=> "test2"
},
{
:name=> "test1"
},
{
:name=> "homework1"
}]
arr1 = arr1.sort_by { |hash| hash[:reference] }
puts arr1
The correct output should look like this :
=> arr1= [
{:reference=>"A43", :name=>"test2"},
{:reference=>"F123", :name=>"test4"},
{:reference=>"ZA4", :name=>"test3"},
{:name=> "test1"},
{:name=> "homework1"}
]
You can only sort on values that can be compared, so if you've got values of different types it's best to convert them to the same type first. A simple work-around is to convert to string:
arr1.sort_by { |hash| hash[:reference].to_s }
You can also assign a default:
arr1.sort_by { |hash| hash[:reference] || '' }
Edit: If you want the nil values sorted last:
arr1.sort_by { |hash| [ hash[:reference] ? 0 : 1, hash[:reference].to_s ] }
If you don't mind temporary variables, you could split the array into two arrays, one containing hashes with :reference and the other those without:
with_ref, without_ref = arr1.partition { |h| h[:reference] }
Then sort the first one:
with_ref.sort_by! { |h| h[:reference] }
And finally concatenate both arrays:
arr1 = with_ref + without_ref

Convert a ruby nested hash in array of hashes

I'm trying to flat a nested hash. My hash has this structure:
{
"errorDescription":"",
"message":"",
"resultCode":"OK",
"resultObj":{
"categoryList":[
{
"asNew":"N",
"categoryId":40000000,
"categoryList":[
{
"asNew":"N",
"categoryId":40000007,
"categoryList":[
],
"categoryName":"CATALOGO",
"categoryType":"TYPE_NODE",
"contentId":40000007,
"contentTitle":"CATALOGO",
"contentType":"CATEGORY_NODE",
"orderId":0,
"urlPictures":""
},
{
"asNew":"N",
"categoryId":40000018,
"categoryList":[
{
"asNew":"N",
"categoryId":40000019,
"categoryList":[
],
"categoryName":"CANALI CALCIO",
"categoryType":"TYPE_VOD",
"contentId":40000019,
"contentTitle":"CALCIO & SPORT",
"contentType":"CATEGORY_LEAF",
"orderId":0,
"urlPictures":""
},
{
"asNew":"N",
"categoryId":40000020,
"categoryList":[
],
"categoryName":"CANALI CINEMA",
"categoryType":"TYPE_VOD",
"contentId":40000020,
"contentTitle":"CINEMA & SERIE",
"contentType":"CATEGORY_LEAF",
"orderId":1,
"urlPictures":""
}
],
"categoryName":"CANALI TV",
"categoryType":"TYPE_NODE",
"contentId":40000018,
"contentTitle":"CANALI TV",
"contentType":"CATEGORY_NODE",
"orderId":1,
"urlPictures":""
}
],
"categoryName":"ROOT",
"categoryType":"TYPE_NODE",
"contentId":40000000,
"contentTitle":"ROOT",
"contentType":"",
"orderId":0,
"urlPictures":""
}
]
}
}
I must convert this hash in an array of objects, where every object has this structure:
{
"asNew":"N",
"categoryId":40000007,
"categoryName":"CATALOGO",
"categoryType":"TYPE_NODE",
"contentId":40000007,
"contentTitle":"CATALOGO",
"contentType":"CATEGORY_NODE",
"orderId":0,
"urlPictures":""
}
Basically every key "categoryList" has an array as value and this array has one or more hash with another key "categoryList". I must move all the objects in every categoryList array to another array without the categoryList 'key'.
edit
I add this method to Hash class
class Hash
def find_all_values_for(key)
result = []
result << self[key]
unless self.values.nil?
self.values.each do |values|
values = [values] unless values.is_a? Array
values.each do |value|
result += value.find_all_values_for(key) if value.is_a? Hash
end
end
end
result.flatten.compact
end
end
And I map the resulting array deleting the CategoryList key
h.map { |c| c.except!('categoryList') }
It works but meybe there is a more efficient way.
This code will return the requested Array of objects.
def flatten_categoryList(categoryList)
categoryList.inject([]) do |result, item|
result + flatten_categoryList(item.delete("categoryList")) << item
end
end
flatten_categoryList(h["resultObj"]["categoryList"])
I was unable to run yours to get a benchmark, so judging the solutions is currently subjective.

Resources