Related
So I have following array of hash:
my_array = [
{
"date" => "2022-12-01",
"pic" => "Jason",
"guard" => "Steven",
"front_desk" => "Emily"
},
{
"date" => "2022-12-02",
"pic" => "Gilbert",
"guard" => "Johnny",
"front_desk" => "Bella"
},
{
"date" => "2022-12-03",
"pic" => "Steven",
"guard" => "Gilbert",
"front_desk" => "Esmeralda"
}
]
My question is how do I change the structure of my array (grouping) by date in Ruby (Rails 7). Or in other word, I want to change my array into something like this:
my_array = [
{
"2022-12-01" => {
"pic" => "Jason",
"guard" => "Steven",
"front_desk" => "Emily"
{
},
{
"2022-12-02" => {
"pic" => "Gilbert",
"guard" => "Johnny",
"front_desk" => "Bella"
}
},
{
"2022-12-03" => {
"pic" => "Steven",
"guard" => "Gilbert",
"front_desk" => "Esmeralda"
}
}
]
Anyway, thanks in advance for the answer
I have tried using group_by method to group by its date, but it doesn't give the output I wanted
I've tried this method:
my_array.group_by { |element| element["date"] }.values
If you simply want a 1:1 mapping of your input objects to an output object of a new shape, then you just need to use Array#map:
my_array.map {|entry| {entry["date"] => entry.except("date")} }
(Hash#except comes from ActiveSupport, and is not standard Ruby, but since you're in Rails it should work just fine).
Both solutions assume the key "date" will be unique. If we cannot make this assumption safely, then each date should be mapped to an array of hashes.
my_array.each_with_object({}) do |x, hsh|
date = x["date"]
hsh[date] ||= []
hsh[date] << x.except("date")
end
Result:
{
"2022-12-01" => [
{"pic"=>"Jason", "guard"=>"Steven", "front_desk"=>"Emily"}
],
"2022-12-02" => [
{"pic"=>"Gilbert", "guard"=>"Johnny", "front_desk"=>"Bella"}
],
"2022-12-03" => [
{"pic"=>"Steven", "guard"=>"Gilbert", "front_desk"=>"Esmeralda"}
]
}
Or you may like:
my_array
.sort_by { |x| x["date"] }
.group_by { |x| x["date"] }
.transform_values { |x| x.except("date") }
I am following a blog to import data to Kibana when I followed those steps I am getting error in Logstash.
This is my configuration file:
input {
file {
path => "C:/SalesJan2009/SalesJan2009.csv"
type => "csv"
start_position => "beginning"
sincedb_path => "C:/SalesJan2009/sinceDb" }
}
filter {
csv {
separator => ","
columns => ["Transaction_date","Product","Price","Payment_Type","Name","City","State","Country","Account_Created","Last_Login","Latitude","Longitude"]
skip_empty_columns => "true"
}
mutate {
convert => [ "Product" => "string" ]
convert => [ "Price" => "float" ]
convert => [ "Payment_Type" => "string" ]
convert => [ "Name" => "string" ]
convert => [ "City" => "string" ]
convert => [ "State" => "string" ]
convert => [ "Country" => "string" ]
convert => [ "Longitude" => "float" ]
convert => [ "Latitude" => "float" ]
}
date
{
match => ["Transaction_date", "dd-MM-yyyyHH:mm:ss"]
match => ["Account_Created", "dd-MM-yyyyHH:mm:ss"]
match => ["Last_Login", "dd-MM-yyyyHH:mm:ss"]
}
}
output {
elasticsearch { hosts => ["http://localhost:9200"]
index => "salestansactions2009"
}
stdout {
codec => dots
}
}
Error is:
PS A:\elk\logstash\logstash-7.4.2\bin> ./logstash -f C:\SalesJan2009\testdata.conf Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to A:/elk/logstash/logstash-7.4.2/logs which is now configured via log4j2.properties
[2019-12-24T12:54:26,310][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-12-24T12:54:26,354][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-12-24T12:54:29,991][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, ,, ] at line 15, column 32 (byte 465) after filter { \r\n csv { \r\n\t separator => \",\" \r\n columns => [\"Transaction_date\",\"Product\",\"Price\",\"Payment_Type\",\"Name\",\"City\",\"State\",\"Country\",\"Account_Created\",\"Last_Login\",\"Latitude\",\"Longitude\"] \r\n\t skip_empty_columns => \"true\"\r\n\t } \r\n mutate { \r\n \tconvert => [ \"Product\" ", :backtrace=>["A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "A:/elk/logstash/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[2019-12-24T12:54:30,882][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-12-24T12:54:35,502][INFO ][logstash.runner ] Logstash shut down.
The syntax for the mutate-convert is wrong.
The error message says:
"Expected one of #, {, ,, ] at line 15, column 32 (byte 465) after filter { \r\n csv { \r\n\t separator => \",\" \r\n columns => [\"Transaction_date\",\"Product\",\"Price\",\"Payment_Type\",\"Name\",\"City\",\"State\",\"Country\",\"Account_Created\",\"Last_Login\",\"Latitude\",\"Longitude\"] \r\n\t skip_empty_columns => \"true\"\r\n\t } \r\n mutate { \r\n \tconvert => [ \"Product\" ", ...
So the error is after convert => [ "Product"
Take a look at the documentation
The value type is hash. So your usage of the square brackets ( [ ] ) is wrong. It should be:
input {
file {
path => "C:/SalesJan2009/SalesJan2009.csv"
type => "csv"
start_position => "beginning"
sincedb_path => "C:/SalesJan2009/sinceDb"
}
}
filter {
csv {
separator => ","
columns => ["Transaction_date","Product","Price","Payment_Type","Name","City","State","Country","Account_Created","Last_Login","Latitude","Longitude"]
skip_empty_columns => "true"
}
mutate {
convert => {
"Product" => "string"
"Price" => "float"
"Payment_Type" => "string"
"Name" => "string"
"City" => "string"
"State" => "string"
"Country" => "string"
"Longitude" => "float"
"Latitude" => "float"
}
}
date {
match => ["Transaction_date", "dd-MM-yyyyHH:mm:ss"]
match => ["Account_Created", "dd-MM-yyyyHH:mm:ss"]
match => ["Last_Login", "dd-MM-yyyyHH:mm:ss"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "salestansactions2009"
}
stdout {
codec => dots
}
}
Instead of this,
date {
match => ["Transaction_date", "dd-MM-yyyyHH:mm:ss"]
match => ["Account_Created", "dd-MM-yyyyHH:mm:ss"]
match => ["Last_Login", "dd-MM-yyyyHH:mm:ss"]
}
Try to configure below code,
date
{
match => ["Transaction_date", "dd-MM-yyyyHH:mm:ss"]
}
date
{
match => ["Account_Created", "dd-MM-yyyyHH:mm:ss"]
}
date
{
match => ["Last_Login", "dd-MM-yyyyHH:mm:ss"]
}
I have a set of nested hashes. I would like to add the string "Assembly" to the array value associated with [:dennis_ritche][:languages]
def adding_to_dennis
programmer_hash =
{
:grace_hopper => {
:known_for => "COBOL",
:languages => ["COBOL", "FORTRAN"]
},
:alan_kay => {
:known_for => "Object Orientation",
:languages => ["Smalltalk", "LISP"]
},
:dennis_ritchie => {
:known_for => "Unix",
:languages => ["C"]
}
}
programmer_hash[:dennis_ritchie][:languages] << "Assembly"
end
This is the error I get no implicit conversion of Symbol into Integer"
I think the problem you're seeing is you're manipulating the hash inside the method and as a result are inadvertently returning the wrong thing. This method returns an Array because that's the last operation performed (<< on Array return the modified Array).
To fix it define a method that does the manipulation:
def add_to_hash(hash, programmer = :dennis_ritchie, language = 'Assembly')
hash[programmer][:languages] << language
end
Make that independent of the definition:
programmer_hash =
{
:grace_hopper => {
:known_for => "COBOL",
:languages => ["COBOL", "FORTRAN"]
},
:alan_kay => {
:known_for => "Object Orientation",
:languages => ["Smalltalk", "LISP"]
},
:margaret_hamilton => {
:known_for => "Apollo Program",
:languages => ["Assembly"]
},
:dennis_ritchie => {
:known_for => "Unix",
:languages => ["C"]
}
}
Then call it to manipulate the hash:
add_to_hash(programmer_hash)
The programmer_hash structure is then updated.
I get following diffgram XML from a service:
<?xml version="1.0"?>
<xvcs:diffgram xmlns:xvcs="http://www.xvcs.org/">
<xvcs:update id="7" first-child-of="/opt/node/node[1]">
<xvcs:attr-update name="location" old-value="???" new-value="testlocation"/>
</xvcs:update>
<xvcs:update id="35" follows="/opt/node/node[2]">
<xvcs:attr-update name="URL" old-value="/" new-value="/testurl/"/>
</xvcs:update>
<xvcs:insert id="75" first-child-of="/opt">
<node node_id="/1234" location="new location" URL="/newurl"></node>
</xvcs:insert>
</xvcs:diffgram>
I'm parsing it with XML::Simple in this way:
my $diffgram_hashref = XMLin($diffgram->toString(1),
KeepRoot => 1,
ForceArray => 1,
);
$logger->debug( dump($diffgram_hashref) );
and get following result:
{
"xvcs:diffgram" => [
{
"xmlns:xvcs" => "http://www.xvcs.org/",
"xvcs:insert" => {
75 => {
"first-child-of" => "/opt",
"node" => [
{
node_id => "/1234",
location => "new location",
URL => "/newurl",
},
],
},
},
"xvcs:update" => {
7 => {
"first-child-of" => "/opt/node/node[1]",
"xvcs:attr-update" => {
location => { "new-value" => "testlocation", "old-value" => "???" },
},
},
35 => {
"follows" => "/opt/node/node[2]",
"xvcs:attr-update" => {
URL => { "new-value" => "/testurl/", "old-value" => "/" },
},
},
},
},
],
}
I tried several ForeArray / KeyAttr combinations but I did not achieve to get the diffgram statements (update, insert) as array in order to proceed them in correct order:
{
"xvcs:diffgram" => [
{
"xvcs:update" => {
7 => {
"first-child-of" => "/opt/node/node[1]",
"xvcs:attr-update" => {
location => { "new-value" => "testlocation", "old-value" => "???" },
},
}
}
},
{
"xvcs:update" => {
35 => {
"follows" => "/opt/node/node[2]",
"xvcs:attr-update" => {
URL => { "new-value" => "/testurl/", "old-value" => "/" },
},
},
}
},
{
"xvcs:insert" => {
75 => {
"first-child-of" => "/opt",
"node" => [
{
node_id => "/1234",
location => "new location",
URL => "/newurl",
},
],
},
},
}
]
}
Could someone help me out please?
This program does what you ask using the XML::Twig module. I've ignored the top-level hash key xvcs:diffgram as that hash has only a single element. The same applies to each hash within the array -- I would prefer to see the element tag as the value of one of the elements of the subsidiary hash, because as it stands you have an array of on-element hashes; however I have left this structure as you describe it.
I have also left the id, name and URL attributes as simple hash elements instead of treating them specially as your example does.
I have used Data::Dump only to show the structure that is built from the data.
use strict;
use warnings;
use XML::Twig;
my $twig = XML::Twig->new;
$twig->parse(\*DATA);
my #data;
for my $node ( $twig->root->children ) {
my $atts = $node->atts;
for my $child ($node->children) {
$atts->{$child->tag} = $child->atts;
}
push #data, { $node->tag => $atts };
}
use Data::Dump;
dd \#data;
__DATA__
<?xml version="1.0"?>
<xvcs:diffgram xmlns:xvcs="http://www.xvcs.org/">
<xvcs:update id="7" first-child-of="/opt/node/node[1]">
<xvcs:attr-update name="location" old-value="???" new-value="testlocation"/>
</xvcs:update>
<xvcs:update id="35" follows="/opt/node/node[2]">
<xvcs:attr-update name="URL" old-value="/" new-value="/testurl/"/>
</xvcs:update>
<xvcs:insert id="75" first-child-of="/opt">
<node node_id="/1234" location="new location" URL="/newurl"></node>
</xvcs:insert>
</xvcs:diffgram>
output
[
{
"xvcs:update" => {
"first-child-of" => "/opt/node/node[1]",
"id" => 7,
"xvcs:attr-update" => {
"name" => "location",
"new-value" => "testlocation",
"old-value" => "???",
},
},
},
{
"xvcs:update" => {
"follows" => "/opt/node/node[2]",
"id" => 35,
"xvcs:attr-update" => {
"name" => "URL",
"new-value" => "/testurl/",
"old-value" => "/",
},
},
},
{
"xvcs:insert" => {
"first-child-of" => "/opt",
"id" => 75,
"node" => {
location => "new location",
node_id => "/1234",
URL => "/newurl",
},
},
},
]
This is an incomplete solution based on comments so far. Hopefully it'll illustrate why Borodin and I are requesting what you're actually trying to get out of your parse.
use strict;
use warnings;
use XML::Twig;
my $twig = XML::Twig->new()->parse( \*DATA );
foreach my $thing ( $twig->root->children() ) {
print $thing ->tag, "\n";
foreach my $att ( keys %{ $thing->atts() } ) {
print "\t", $att, "=", $thing->att($att), "\n";
}
my $op = $thing->first_child;
print "\t\t", $op->name, "\n";
foreach my $att ( keys %{ $op->atts } ) {
print "\t\t\t", $att, "=", $op->att($att), "\n";
}
}
__DATA__
<?xml version="1.0"?>
<xvcs:diffgram xmlns:xvcs="http://www.xvcs.org/">
<xvcs:update id="7" first-child-of="/opt/node/node[1]">
<xvcs:attr-update name="location" old-value="???" new-value="testlocation"/>
</xvcs:update>
<xvcs:update id="35" follows="/opt/node/node[2]">
<xvcs:attr-update name="URL" old-value="/" new-value="/testurl/"/>
</xvcs:update>
<xvcs:insert id="75" first-child-of="/opt">
<node node_id="/1234" location="new location" URL="/newurl"></node>
</xvcs:insert>
</xvcs:diffgram>
This will print:
xvcs:update
first-child-of=/opt/node/node[1]
id=7
xvcs:attr-update
old-value=???
new-value=testlocation
name=location
xvcs:update
follows=/opt/node/node[2]
id=35
xvcs:attr-update
old-value=/
new-value=/testurl/
name=URL
xvcs:insert
first-child-of=/opt
id=75
node
URL=/newurl
location=new location
node_id=/1234
They key point being that turning your XML into an array of hashes is - probably - an XY problem. You're focussing on trying to do something one way, and the answer is - probably - don't do it that way.
I have recursively put together an array of hashes for perl, which looks something like this :
[
{
'Default' => {
'Elect' => { 'P' => 1 }
}
},
{
'Default' => {
'Elect' => { 'A' => 1 }
}
},
{
'Default' => {
'Elect' => { 'M' => 1 }
}
},
{
'Default' => {
'Elect' => { 'I' => 1 }
}
},
{
'Default' => {
'Picker' => { 'L' => 1 }
}
},
]
My aim is to make this more condensed and look like a single hash, as compared to array of hashes. Is there anyway in which i can make this array of hashes look like a hash:
{
'Default' =>{
'Elect' =>{
'P' => 1,
'A' => 1,
'M' => 1,
'I' => 1,
},
'Picker' => {
'L' => 1
}
}
}
Well, here is a simple recursive procedure to merge two hash references:
sub merge {
my ($xs, $ys) = #_;
while (my ($k, $v) = each %$ys) {
if ('HASH' eq ref $v) {
merge($xs->{$k} //= {}, $v);
}
else {
$xs->{$k} = $v;
}
}
}
Then:
my $data = ...; # your input data structure
my $acc = {};
merge($acc, $_) for #$data;
which produces the result you desire in $acc.
There is also the Hash::Merge module, with that:
use Hash::Merge 'merge';
my $acc = {};
$acc = merge $acc, $_ for #$data;