in erlang language I receive this kids of JSON data:
{"Time":"2020-08-16T15:28:55","BME680":{"Temperature":29.8,"Humidity":55.5,"Pressure":1003.5,"Gas":422.24},"PressureUnit":"hPa","TempUnit":"C"}
or
{"Time":"2020-08-16T11:39:49","SI7021":{"Temperature":32.4,"Humidity":99.9},"TempUnit":"C"}
I need to select only one value per each JSON data for example:
based on "BME680" value
{"Temperature":29.8,"Humidity":54.8,"Pressure":1005.0,"Gas":1513.60}
or
based on "SI7021" value
{"Temperature":32.4,"Humidity":99.9}
How can acive this task in erlang language ?
If it cold be simpler I need to extract the second value of each JSON data.
For decode JSON format into Erlang format data like proplists or into the maps, you can try to use 3rd party library jiffy, then you can use simple pattern matching, eg:
Maps:
1> JSON = "{\"Time\":\"2020-08-17T05:32:09\",\"BME680\":{\"Temperature\":29.6,\"Humidity\":54.6,\"Pressure\":1003.9,\"Gas\":1517.91},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}".
2> Map = jiffy:decode(JSON,[return_maps]).
3> #{<<"BME680">> := BME680} = Map.
4> BME680.
#{<<"Gas">> => 1517.91,<<"Humidity">> => 54.6, <<"Pressure">> => 1003.9,<<"Temperature">> => 29.6}
Proplists:
1> JSON = "{\"Time\":\"2020-08-17T05:32:09\",\"BME680\":{\"Temperature\":29.6,\"Humidity\":54.6,\"Pressure\":1003.9,\"Gas\":1517.91},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}".
2> {Proplists} = jiffy:decode(JSON).
3> [BME680] = [V || {K, V} <- Proplists, K == <<"BME680">>].
4> BME680.
{[{<<"Temperature">>,29.6},{<<"Humidity">>,54.6},{<<"Pressure">>,1003.9},{<<"Gas">>,1517.91}]}
You can use 2 modules from mochiweb project:
https://github.com/mochi/mochiweb/blob/master/src/mochijson2.erl
https://raw.githubusercontent.com/mochi/mochiweb/master/src/mochinum.erl
Put them in some folder (f.e. test) and run erl shell and compile this files:
alexei#MacBook-Pro test % erl
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
Eshell V11.0 (abort with ^G)
1> c("mochijson2.erl").
{ok,mochijson2}
2> c("mochinum.erl").
{ok,mochinum}
Decode json-as-text to json-as-erlang-term:
3> Body = "{\"Time\":\"2020-08-16T15:28:55\",\"BME680\":{\"Temperature\":29.8,\"Humidity\":55.5,\"Pressure\":1003.5,\"Gas\":422.24},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}".
"{\"Time\":\"2020-08-16T15:28:55\",\"BME680\":{\"Temperature\":29.8,\"Humidity\":55.5,\"Pressure\":1003.5,\"Gas\":422.24},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}"
4> {struct, Map} = mochijson2:decode(Body).
{struct,[{<<"Time">>,<<"2020-08-16T15:28:55">>},
{<<"BME680">>,
{struct,[{<<"Temperature">>,29.8},
{<<"Humidity">>,55.5},
{<<"Pressure">>,1003.5},
{<<"Gas">>,422.24}]}},
{<<"PressureUnit">>,<<"hPa">>},
{<<"TempUnit">>,<<"C">>}]}
Retrive the items you need:
5> {struct, Obj1} = proplists:get_value(<<"BME680">>, Map).
{struct,[{<<"Temperature">>,29.8},
{<<"Humidity">>,55.5},
{<<"Pressure">>,1003.5},
{<<"Gas">>,422.24}]}
6> Item1 = proplists:lookup(<<"Temperature">>, Obj1).
{<<"Temperature">>,29.8}
7> Item2 = proplists:lookup(<<"Humidity">>, Obj1).
{<<"Humidity">>,55.5}
And encode erlang term to text:
8> List = [Item1, Item2].
[{<<"Temperature">>,29.8},{<<"Humidity">>,55.5}]
9> iolist_to_binary(mochijson2:encode({struct, List})).
<<"{\"Temperature\":29.8,\"Humidity\":55.5}">>
There is a very short documentation for mochijson2, but it is easy to learn just practice in shell.
You need to familiarize yourself with rebar3, which is erlang's package manager. Then, you can use a package like jsx, which will allow you to convert between binary types and maps. First, you will need to convert whatever it is that you call "JSON" to a binary type, perhaps using tuple_to_list() then list_to_binary(), then you can use the jsx package to convert the binary to a map, which will enable you to extract the target value from the map.
Here's an example:
my.erl:
json_to_map(Bin) ->
jsx:decode(
%%<<"{\"data\": [1, 2, 3]}">>,
Bin,
[return_maps]
).
In the shell:
~/erlang_programs/myapp$ rebar3 compile
===> Verifying dependencies...
===> Compiling myapp
src/my.erl:2: Warning: export_all flag enabled - all functions will be exported
~/erlang_programs/myapp$ rebar3 shell
===> Verifying dependencies...
===> Compiling myapp
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [kernel-poll:false]
Eshell V9.3 (abort with ^G)
1> Bin1 = <<"{\"A\": 100, \"B\":{\"x\": 4,\"y\": [1,2,3]}}">>.
<<"{\"A\": 100, \"B\":{\"x\": 4,\"y\": [1,2,3]}}">>
2> Bin2 = <<"{\"A\":200, \"C\":{\"a\": 6, \"b\": [3,4,5]}}">>.
<<"{\"A\":200, \"C\":{\"a\": 6, \"b\": [3,4,5]}}">>
3> M1 = my:json_to_map(Bin1).
#{<<"A">> => 100,
<<"B">> => #{<<"x">> => 4,<<"y">> => [1,2,3]}}
4> M2 = my:json_to_map(Bin2).
#{<<"A">> => 200,
<<"C">> => #{<<"a">> => 6,<<"b">> => [3,4,5]}}
5> [V1] = maps:values( maps:without([<<"A">>], M1)).
[#{<<"x">> => 4,<<"y">> => [1,2,3]}]
6> V1.
#{<<"x">> => 4,<<"y">> => [1,2,3]}
7> [V2] = maps:values( maps:without([<<"A">>], M2)).
[#{<<"a">> => 6,<<"b">> => [3,4,5]}]
8> V2.
#{<<"a">> => 6,<<"b">> => [3,4,5]}
9>
Hi following your advice is it correct my below code if "Body" is equal to:
"{\"Time\":\"2020-08-17T05:32:09\",\"BME680\":{\"Temperature\":29.6,\"Humidity\":54.6,\"Pressure\":1003.9,\"Gas\":1517.91},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}"
my code
....
eBody = jiffy:decode(<<Body>>),
Map = #eBody,
#{"BME680" := BME680} = Map,
newbody = jiffy:encode(BME680),
HTTPOptions = [],
Options = [],
R = httpc:request(Method, {URL, Header, Type, newbody},HTTPOptions, Options),
....
I'm newbie in erlang.
I followed the tutorial from https://medium.com/#chvanikoff/phoenix-react-love-story-reph-1-c68512cfe18 and developed an application but with different versions.
elixir - 1.3.4
phoenix - 1.2.1
poison - 2.0
distillery - 0.10
std_json_io - 0.1
The application ran successfully when running locally.
Bur when created a mix release(MIX_ENV=prod mix release --env=prod --verbose) and ran rel/utopia/bin/utopia console(the otp application name was :utopia), I ran into error
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)
14:18:21.857 request_id=idqhtoim2nb3lfeguq22627a92jqoal6 [info] GET /
panic: write |1: broken pipe
goroutine 3 [running]:
runtime.panic(0x4a49e0, 0xc21001f480)
/usr/local/Cellar/go/1.2.2/libexec/src/pkg/runtime/panic.c:266 +0xb6
log.(*Logger).Panicf(0xc210020190, 0x4de060, 0x3, 0x7f0924c84e38, 0x1, ...)
/usr/local/Cellar/go/1.2.2/libexec/src/pkg/log/log.go:200 +0xbd
main.fatal_if(0x4c2680, 0xc210039ab0)
/Users/alco/extra/goworkspace/src/goon/util.go:38 +0x17e
main.inLoop2(0x7f0924e0c388, 0xc2100396f0, 0xc2100213c0, 0x7f0924e0c310, 0xc210000000, ...)
/Users/alco/extra/goworkspace/src/goon/io.go:100 +0x5ce
created by main.wrapStdin2
/Users/alco/extra/goworkspace/src/goon/io.go:25 +0x15a
goroutine 1 [chan receive]:
main.proto_2_0(0x7ffce6670101, 0x4e3e20, 0x3, 0x4de5a0, 0x1, ...)
/Users/alco/extra/goworkspace/src/goon/proto_2_0.go:58 +0x3a3
main.main()
/Users/alco/extra/goworkspace/src/goon/main.go:51 +0x3b6
14:18:21.858 request_id=idqhtoim2nb3lfeguq22627a92jqoal6 [info] Sent 500 in 1ms
14:18:21.859 [error] #PID<0.1493.0> running Utopia.Endpoint terminated
Server: 127.0.0.1:8080 (http)
Request: GET /
** (exit) an exception was raised:
** (Protocol.UndefinedError) protocol String.Chars not implemented for {#PID<0.1467.0>, :result, %Porcelain.Result{err: nil, out: {:send, #PID<0.1466.0>}, status: 2}}
(elixir) lib/string/chars.ex:3: String.Chars.impl_for!/1
(elixir) lib/string/chars.ex:17: String.Chars.to_string/1
(utopia) lib/utopia/react_io.ex:2: Utopia.ReactIO.json_call!/2
(utopia) web/controllers/page_controller.ex:12: Utopia.PageController.index/2
(utopia) web/controllers/page_controller.ex:1: Utopia.PageController.action/2
(utopia) web/controllers/page_controller.ex:1: Utopia.PageController.phoenix_controller_pipeline/2
(utopia) lib/utopia/endpoint.ex:1: Utopia.Endpoint.instrument/4
(utopia) lib/phoenix/router.ex:261: Utopia.Router.dispatch/2
goon got panicked and hence the porcelain. Someone please provide a solution.
Related issues: https://github.com/alco/porcelain/issues/13
EDIT: My page_controller.ex
defmodule Utopia.PageController do
use Utopia.Web, :controller
def index(conn, _params) do
visitors = Utopia.Tracking.Visitors.state
initial_state = %{"visitors" => visitors}
props = %{
"location" => conn.request_path,
"initial_state" => initial_state
}
result = Utopia.ReactIO.json_call!(%{
component: "./priv/static/server/js/utopia.js",
props: props
})
render(conn, "index.html", html: result["html"], props: initial_state)
end
end
And again, asking for help. But, before I start, here will be a lot of text, so please sorry for that.
I have about 500~ IP addresses with devices 2x categories in .xlsx book
I want:
telnet to device. Check device (by authentication prompt) type 1 or type 2.
If device is type 1 - get it firmware version in 2x partitions
write in excel file:
column 1 - IP address
column 2 - device type
column 3 - firmware version
column 4 - firmware version in reserve partition.
If type 2 - write in excel file:
column 1 - IP address
column 2 - device type
If device is down, or device type 3(unknown) - write in excel file:
column 1 - IP address
column 2 - result (EOF, TIMEOUT)
What I have done: I'm able to telnet to device, check device type, write in excel with 2 columns (in 1 column IP addresses, in 2 column is device type, or EOF/TIMEOUT results)
And, I'm writing full logs from session to files in format IP_ADDRESS.txt to future diagnosis.
What I can't understand to do? I can't understand how to get firmware version, and put it on 3,4 columns.
I can't understand how to work with current log session in real time, so I've decided to copy logs from main file (IP_ADDRESS.txt) to temp.txt to work with it.
I can't understand how to extract information I needed.
The file output example:
Trying 10.40.81.167...
Connected to 10.40.81.167.
Escape character is '^]'.
####################################
# #
# RADIUS authorization disabled #
# Enter local login/password #
# #
####################################
bt6000 login: admin
Password:
Please, fill controller information at first time (Ctrl+C to abort):
^C
Controller information filling canceled.
^Cadmin#bt6000# firmware info
Active boot partition: 1
Partition 0 (reserved):
Firmware: Energomera-2.3.1
Version: 10117
Partition 1 (active):
Firmware: Energomera-2.3.1_01.04.15c
Version: 10404M
Kernel version: 2.6.38.8 #2 Mon Mar 2 20:41:26 MSK 2015
STM32:
Version: bt6000 10083
Part Number: BT6024
Updated: 27.04.2015 16:43:50
admin#bt6000#
I need values - after "Energomera" words, like 2.3.1 for reserved partition, and 2.3.1_01.04.15c for active partition.
I've tried to work with string numbers and excract string, but there was not any kind of good result at all.
Full code of my script below.
import pexpect
import pxssh
import sys #hz module
import re #Parser module
import os #hz module
import getopt
import glob #hz module
import xlrd #Excel read module
import xlwt #Excel write module
import telnetlib #telnet module
import shutil
#open excel book
rb = xlrd.open_workbook('/samba/allaccess/Energomera_Eltek_list.xlsx')
#select work sheet
sheet = rb.sheet_by_name('IPs')
#rows number in sheet
num_rows = sheet.nrows
#cols number in sheet
num_cols = sheet.ncols
#creating massive with IP addresses inside
ip_addr_list = [sheet.row_values(rawnum)[0] for rawnum in range(sheet.nrows)]
#create excel workbook with write permissions (xlwt module)
wb = xlwt.Workbook()
#create sheet IP LIST with cell overwrite rights
ws = wb.add_sheet('IP LIST', cell_overwrite_ok=True)
#create counter
i = 0
#authorization details
port = "23" #telnet port
user = "admin" #telnet username
password = "12345" #telnet password
#firmware ask function
def fw_info():
print('asking for firmware')
px.sendline('firmware info')
px.expect('bt6000#')
#firmware update function
def fw_send():
print('sending firmware')
px.sendline('tftp server 172.27.2.21')
px.expect('bt6000')
px.sendline('firmware download tftp firmware.ext2')
px.expect('Updating')
px.sendline('y')
px.send(chr(13))
ws.write(i, 0, host)
ws.write(i, 1, 'Energomera')
#if eltek found - skip, write result in book
def eltek_found():
print(host, "is Eltek. Skipping")
ws.write(i, 0, host)
ws.write(i, 1, 'Eltek')
#if 23 port telnet conn. refused - skip, write result in book
def conn_refuse():
print(host, "connection refused")
ws.write(i, 0, host)
ws.write(i, 1, 'Connection refused')
#auth function
def auth():
print(host, "is up! Energomera found. Starting auth process")
px.sendline(user)
px.expect('assword')
px.sendline(password)
#start working with ip addresses in ip_addr_list massive
for host in ip_addr_list:
#spawn pexpect connection
px = pexpect.spawn('telnet ' + host)
px.timeout = 35
#create log file with in IP.txt format (10.1.1.1.txt, for example)
fout = open('/samba/allaccess/Energomera_Eltek/{0}.txt'.format(host),"wb")
#push pexpect logfile_read output to log file
px.logfile_read = fout
try:
index = px.expect (['bt6000', 'sername', 'refused'])
#if device tell us bt6000 - authorize
if index == 0:
auth()
index1 = px.expect(['#', 'lease'])
#if "#" - ask fw version immediatly
if index1 == 0:
print('seems to controller ID already set')
fw_info()
#if "Please" - press 2 times Ctrl+C, then ask fw version
elif index1 == 1:
print('trying control C controller ID')
px.send(chr(3))
px.send(chr(3))
px.expect('bt6000')
fw_info()
#firmware update start (temporarily off)
# fw_send()
#Eltek found - func start
elif index == 1:
eltek_found()
#Conn refused - func start
elif index == 2:
conn_refuse()
#print output to console (test purposes)
print(px.before)
px.send(chr(13))
#Copy from current log file to temp.txt for editing
shutil.copy2('/samba/allaccess/Energomera_Eltek/{0}.txt'.format(host), '/home/bark/expect/temp.txt')
#EOF result - skip host, write result to excel
except pexpect.EOF:
print(host, "EOF")
ws.write(i, 0, host)
ws.write(i, 1, 'EOF')
#print output to console (test purposes)
print(px.before)
#Timeout result - skip host, write result to excel
except pexpect.TIMEOUT:
print(host, "TIMEOUT")
ws.write(i, 0, host)
ws.write(i, 1, 'TIMEOUT')
#print output to console (test purposes)
print(px.before)
#Copy from current log file to temp.txt for editing
shutil.copy2('/samba/allaccess/Energomera_Eltek/{0}.txt'.format(host), '/home/bark/expect/temp.txt')
#count +1 to correct output for Excel
i += 1
#workbook save
wb.save('/samba/allaccess/Energomera_Eltek_result.xls')
Have you have any suggestions or ideas, guys, how I can do this?
Any help is greatly appreciated.
You can use regular expressions
example:
>>> import re
>>>
>>> str = """
... Trying 10.40.81.167...
...
... Connected to 10.40.81.167.
...
... Escape character is '^]'.
...
...
...
... ####################################
... # #
... # RADIUS authorization disabled #
... # Enter local login/password #
... # #
... ####################################
... bt6000 login: admin
... Password:
... Please, fill controller information at first time (Ctrl+C to abort):
... ^C
... Controller information filling canceled.
... ^Cadmin#bt6000# firmware info
... Active boot partition: 1
... Partition 0 (reserved):
... Firmware: Energomera-2.3.1
... Version: 10117
... Partition 1 (active):
... Firmware: Energomera-2.3.1_01.04.15c
... Version: 10404M
... Kernel version: 2.6.38.8 #2 Mon Mar 2 20:41:26 MSK 2015
... STM32:
... Version: bt6000 10083
... Part Number: BT6024
... Updated: 27.04.2015 16:43:50
... admin#bt6000#
... """
>>> re.findall(r"Firmware:.*?([0-9].*)\s", str)
['2.3.1', '2.3.1_01.04.15c']
>>> reserved_firmware = re.search(r"reserved.*\s*Firmware:.*?([0-9].*)\s", str).group(1)
>>> reserved_firmware
'2.3.1'
>>> active_firmware = re.search(r"active.*\s*Firmware:.*?([0-9].*)\s", str).group(1)
>>> active_firmware
'2.3.1_01.04.15c'
>>>