erlang JSON data - how to select only one value - arrays

in erlang language I receive this kids of JSON data:
{"Time":"2020-08-16T15:28:55","BME680":{"Temperature":29.8,"Humidity":55.5,"Pressure":1003.5,"Gas":422.24},"PressureUnit":"hPa","TempUnit":"C"}
or
{"Time":"2020-08-16T11:39:49","SI7021":{"Temperature":32.4,"Humidity":99.9},"TempUnit":"C"}
I need to select only one value per each JSON data for example:
based on "BME680" value
{"Temperature":29.8,"Humidity":54.8,"Pressure":1005.0,"Gas":1513.60}
or
based on "SI7021" value
{"Temperature":32.4,"Humidity":99.9}
How can acive this task in erlang language ?
If it cold be simpler I need to extract the second value of each JSON data.

For decode JSON format into Erlang format data like proplists or into the maps, you can try to use 3rd party library jiffy, then you can use simple pattern matching, eg:
Maps:
1> JSON = "{\"Time\":\"2020-08-17T05:32:09\",\"BME680\":{\"Temperature\":29.6,\"Humidity\":54.6,\"Pressure\":1003.9,\"Gas\":1517.91},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}".
2> Map = jiffy:decode(JSON,[return_maps]).
3> #{<<"BME680">> := BME680} = Map.
4> BME680.
#{<<"Gas">> => 1517.91,<<"Humidity">> => 54.6, <<"Pressure">> => 1003.9,<<"Temperature">> => 29.6}
Proplists:
1> JSON = "{\"Time\":\"2020-08-17T05:32:09\",\"BME680\":{\"Temperature\":29.6,\"Humidity\":54.6,\"Pressure\":1003.9,\"Gas\":1517.91},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}".
2> {Proplists} = jiffy:decode(JSON).
3> [BME680] = [V || {K, V} <- Proplists, K == <<"BME680">>].
4> BME680.
{[{<<"Temperature">>,29.6},{<<"Humidity">>,54.6},{<<"Pressure">>,1003.9},{<<"Gas">>,1517.91}]}

You can use 2 modules from mochiweb project:
https://github.com/mochi/mochiweb/blob/master/src/mochijson2.erl
https://raw.githubusercontent.com/mochi/mochiweb/master/src/mochinum.erl
Put them in some folder (f.e. test) and run erl shell and compile this files:
alexei#MacBook-Pro test % erl
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
Eshell V11.0 (abort with ^G)
1> c("mochijson2.erl").
{ok,mochijson2}
2> c("mochinum.erl").
{ok,mochinum}
Decode json-as-text to json-as-erlang-term:
3> Body = "{\"Time\":\"2020-08-16T15:28:55\",\"BME680\":{\"Temperature\":29.8,\"Humidity\":55.5,\"Pressure\":1003.5,\"Gas\":422.24},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}".
"{\"Time\":\"2020-08-16T15:28:55\",\"BME680\":{\"Temperature\":29.8,\"Humidity\":55.5,\"Pressure\":1003.5,\"Gas\":422.24},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}"
4> {struct, Map} = mochijson2:decode(Body).
{struct,[{<<"Time">>,<<"2020-08-16T15:28:55">>},
{<<"BME680">>,
{struct,[{<<"Temperature">>,29.8},
{<<"Humidity">>,55.5},
{<<"Pressure">>,1003.5},
{<<"Gas">>,422.24}]}},
{<<"PressureUnit">>,<<"hPa">>},
{<<"TempUnit">>,<<"C">>}]}
Retrive the items you need:
5> {struct, Obj1} = proplists:get_value(<<"BME680">>, Map).
{struct,[{<<"Temperature">>,29.8},
{<<"Humidity">>,55.5},
{<<"Pressure">>,1003.5},
{<<"Gas">>,422.24}]}
6> Item1 = proplists:lookup(<<"Temperature">>, Obj1).
{<<"Temperature">>,29.8}
7> Item2 = proplists:lookup(<<"Humidity">>, Obj1).
{<<"Humidity">>,55.5}
And encode erlang term to text:
8> List = [Item1, Item2].
[{<<"Temperature">>,29.8},{<<"Humidity">>,55.5}]
9> iolist_to_binary(mochijson2:encode({struct, List})).
<<"{\"Temperature\":29.8,\"Humidity\":55.5}">>
There is a very short documentation for mochijson2, but it is easy to learn just practice in shell.

You need to familiarize yourself with rebar3, which is erlang's package manager. Then, you can use a package like jsx, which will allow you to convert between binary types and maps. First, you will need to convert whatever it is that you call "JSON" to a binary type, perhaps using tuple_to_list() then list_to_binary(), then you can use the jsx package to convert the binary to a map, which will enable you to extract the target value from the map.
Here's an example:
my.erl:
json_to_map(Bin) ->
jsx:decode(
%%<<"{\"data\": [1, 2, 3]}">>,
Bin,
[return_maps]
).
In the shell:
~/erlang_programs/myapp$ rebar3 compile
===> Verifying dependencies...
===> Compiling myapp
src/my.erl:2: Warning: export_all flag enabled - all functions will be exported
~/erlang_programs/myapp$ rebar3 shell
===> Verifying dependencies...
===> Compiling myapp
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [kernel-poll:false]
Eshell V9.3 (abort with ^G)
1> Bin1 = <<"{\"A\": 100, \"B\":{\"x\": 4,\"y\": [1,2,3]}}">>.
<<"{\"A\": 100, \"B\":{\"x\": 4,\"y\": [1,2,3]}}">>
2> Bin2 = <<"{\"A\":200, \"C\":{\"a\": 6, \"b\": [3,4,5]}}">>.
<<"{\"A\":200, \"C\":{\"a\": 6, \"b\": [3,4,5]}}">>
3> M1 = my:json_to_map(Bin1).
#{<<"A">> => 100,
<<"B">> => #{<<"x">> => 4,<<"y">> => [1,2,3]}}
4> M2 = my:json_to_map(Bin2).
#{<<"A">> => 200,
<<"C">> => #{<<"a">> => 6,<<"b">> => [3,4,5]}}
5> [V1] = maps:values( maps:without([<<"A">>], M1)).
[#{<<"x">> => 4,<<"y">> => [1,2,3]}]
6> V1.
#{<<"x">> => 4,<<"y">> => [1,2,3]}
7> [V2] = maps:values( maps:without([<<"A">>], M2)).
[#{<<"a">> => 6,<<"b">> => [3,4,5]}]
8> V2.
#{<<"a">> => 6,<<"b">> => [3,4,5]}
9>

Hi following your advice is it correct my below code if "Body" is equal to:
"{\"Time\":\"2020-08-17T05:32:09\",\"BME680\":{\"Temperature\":29.6,\"Humidity\":54.6,\"Pressure\":1003.9,\"Gas\":1517.91},\"PressureUnit\":\"hPa\",\"TempUnit\":\"C\"}"
my code
....
eBody = jiffy:decode(<<Body>>),
Map = #eBody,
#{"BME680" := BME680} = Map,
newbody = jiffy:encode(BME680),
HTTPOptions = [],
Options = [],
R = httpc:request(Method, {URL, Header, Type, newbody},HTTPOptions, Options),
....
I'm newbie in erlang.

Related

Generate all combinations of the SUM in Ruby but only using specific amount of numbers

I am currently pulling in F1 prices from an Api, placing them into an Array. and determining what combination is less than or equal to 20. Using the below successfully:
require 'net/http'
require 'json'
#url = 'HIDDEN URL AS HAS NO RELEVANCE'
#uri = URI(#url)
#response = Net::HTTP.get(#uri)
#fantasy = JSON.parse(#response)
arr= [[#fantasy.first["Mercedes"].to_f, #fantasy.first["Ferrari"].to_f], [#fantasy.first["Hamilton"].to_f, #fantasy.first["Verstappen"].to_f]]
target = 20
#array = arr[0].product(*arr[1..-1]).select { |a| a.reduce(:+) <= target }
Where:
#fantasy = [{"Mercedes" => "4", "Ferrari" => "6.2", "Hamilton" => "7.1", "Verstappen" => "3"}]
This is successfully outputting:
[[4.0, 7.1], [4.0, 3.0], [6.2, 7.1], [6.2, 3.0]]
Eventually this will contain all F1 teams on the left side and all F1 drivers on the right (making an F1 fantasy teambuilder). But the idea is that only 1 constructor is needed and 5 drivers for the combination that should be equal or less than 20.
Is there a way to define this? To only use 1 Team (Mercedes, Ferrari etc) and 5 drivers (Hamilton, Verstappen etc) in the calculation? Obviously do not have 5 drivers included yet as just testing. So that my output would be:
[[4.0, 7.1, 3.0], [6.2, 7.1, 3.0]]
Where the constructor forms the 'base' for the calculation and then it can have any 5 of the driver calls?
My final question is, considering what I am trying to do, is this the best way to put my API into an array? As in to manually place #fantasy.first["Mercedes"].to_f inside my array brackets?
Thanks!
Not sure if I understand the question, but does this help?
arr = #fantasy.first.values.map(&:to_f)
target = 20
p result = arr.combination(2).select{|combi| combi.sum <= target}

How do i copy an array from a lua file

I Want to copy an array from a text file and make another array equal it
so
local mapData = {
grass = {
cam = "hud",
x = 171,
image = "valley/grass",
y = 168,
animated = true
}
}
This is an array that is in Data.lua
i want to copy this array and make it equal another array
local savedMapData = {}
savedMapData = io.open('Data.lua', 'r')
Thank you.
It depends on Lua Version what you can do further.
But i like questions about file operations.
Because filehandlers in Lua are Objects with methods.
The datatype is userdata.
That means it has methods that can directly be used on itself.
Like the methods for the datatype string.
Therefore its easy going to do lazy things like...
-- Example open > reading > loading > converting > defining
-- In one Line - That is possible with methods on datatype
-- Lua 5.4
local savedMapData = load('return {' .. io.open('Data.lua'):read('a'):gsub('^.*%{', ''):gsub('%}.*$', '') .. '}')()
for k, v in pairs(savedMapData) do print(k, '=>', v) end
Output should be...
cam => hud
animated => true
image => valley/grass
y => 168
x => 171
If you need it in the grass table then do...
local savedMapData = load('return {grass = {' .. io.open('Data.lua'):read('a'):gsub('^.*%{', ''):gsub('%}.*$', '') .. '}}')()
The Chain of above methods do...
io.open('Data.lua') - Creates Filehandler (userdata) in read only mode
(userdata):read('a') - Reading whole File into one (string)
(string):gsub('^.*%{', '') - Replace from begining to first { with nothing
(string):gsub('%}.*$', '') - Replace from End to first } with nothing

Python, face_recognition convert string to array

I want to convert a variable to a string and then to an array that I can use to compare, but i dont know how to do that.
my code:
import face_recognition
import numpy as np
a = face_recognition.load_image_file('C:\\Users\zivsi\OneDrive\תמונות\סרט צילום\WIN_20191115_10_32_24_Pro.jpg') # my picture 1
b = face_recognition.load_image_file('C:\\Users\zivsi\OneDrive\תמונות\סרט צילום\WIN_20191115_09_48_56_Pro.jpg') # my picture 2
c = face_recognition.load_image_file(
'C:\\Users\zivsi\OneDrive\תמונות\סרט צילום\WIN_20191115_09_48_52_Pro.jpg') # my picture 3
d = face_recognition.load_image_file('C:\\Users\zivsi\OneDrive\תמונות\סרט צילום\ziv sion.jpg') # my picture 4
e = face_recognition.load_image_file(
'C:\\Users\zivsi\OneDrive\תמונות\סרט צילום\WIN_20191120_17_46_40_Pro.jpg') # my picture 5
f = face_recognition.load_image_file(
'C:\\Users\zivsi\OneDrive\תמונות\סרט צילום\WIN_20191117_16_19_11_Pro.jpg') # my picture 6
a = face_recognition.face_encodings(a)[0]
b = face_recognition.face_encodings(b)[0]
c = face_recognition.face_encodings(c)[0]
d = face_recognition.face_encodings(d)[0]
e = face_recognition.face_encodings(e)[0]
f = face_recognition.face_encodings(f)[0]
Here I tried to convert the variable to a string
str_variable = str(a)
array_variable = np.array(str_variable)
my_face = a, b, c, d, e, f, array_variable
while True:
new = input('path: ')
print('Recognizing...')
unknown = face_recognition.load_image_file(new)
unknown_encodings = face_recognition.face_encodings(unknown)[0]
The program cannot use the variable:
results = face_recognition.compare_faces(array_variable, unknown_encodings, tolerance=0.4)
print(results)
recognize_times = int(results.count(True))
if (3 <= recognize_times):
print('hello boss!')
my_face = *my_face, unknown_encodings
please help me
The error shown:
Traceback (most recent call last):
File "C:/Users/zivsi/PycharmProjects/AI/pytt.py", line 37, in <module>
results = face_recognition.compare_faces(my_face, unknown_encodings, tolerance=0.4)
File "C:\Users\zivsi\AppData\Local\Programs\Python\Python36\lib\site-
packages\face_recognition\api.py", line 222, in compare_faces
return list(face_distance(known_face_encodings, face_encoding_to_check) <= tolerance)
File "C:\Users\zivsi\AppData\Local\Programs\Python\Python36\lib\site-packages\face_recognition\api.py", line 72, in face_distance
return np.linalg.norm(face_encodings - face_to_compare, axis=1)
ValueError: operands could not be broadcast together with shapes (7,) (128,)
First of all, the array_variable should actually be a list of the known encodings and not a numpy array.
Also you do not need str.
Now, in your case, if the input images i.e., a,b,c,d,f,e do NOT have the same dimensions, the error will persist. You can not compare images that have different sizes using this function. The reason is that the comparison is based on the distance and distance is defined on vectors of the same length.
Here is a working simple example using the photos from https://github.com/ageitgey/face_recognition/tree/master/examples:
import face_recognition
import numpy as np
from PIL import Image, ImageDraw
from IPython.display import display
# Load a sample picture and learn how to recognize it.
obama_image = face_recognition.load_image_file("obama.jpg")
obama_face_encoding = face_recognition.face_encodings(obama_image)[0]
# Load a second sample picture and learn how to recognize it.
biden_image = face_recognition.load_image_file("biden.jpg")
biden_face_encoding = face_recognition.face_encodings(biden_image)[0]
array_variable = [obama_face_encoding,biden_face_encoding] # list of known encodings
# compare the list with the biden_face_encoding
results = face_recognition.compare_faces(array_variable, biden_face_encoding, tolerance=0.4)
print(results)
[False, True] # True means match, False mismatch
# False: coming from obama_face_encoding VS biden_face_encoding
# True: coming from biden_face_encoding VS biden_face_encoding
To run it go here: https://beta.deepnote.com/project/09705740-31c0-4d9a-8890-269ff1c3dfaf#
Documentation: https://face-recognition.readthedocs.io/en/latest/face_recognition.html
EDIT
To save the known encodings you can use numpy.save
np.save('encodings',biden_face_encoding) # save
load_again = np.load('encodings.npy') # load again

Turning an Array into a Data Frame

I would like to put my summary statistics into a table using the kable function, but I cannot because it comes up as an array.
```{r setup options, include = FALSE}
knitr::opts_chunk$set(fig.width = 8, fig.height = 5, echo = TRUE)
library(mosaic)
library(knitr)
```
```{r}
sum = summary(SwimRecords$time) # generic data set included with mosaic package
kable(sum) # I want this to be printed into a table
```
Any suggestions?
You can do so easily with the broom package which is built to "tidy" these stats-related objects:
#install.packages(broom)
broom::tidy(sum)

Scala read only certain parts of file

I'm trying to read an input file in Scala that I know the structure of, however I only need every 9th entry. So far I have managed to read the whole thing using:
val lines = sc.textFile("hdfs://moonshot-ha-nameservice/" + args(0))
val fields = lines.map(line => line.split(","))
The issue, this leaves me with an array that is huge (we're talking 20GB of data). Not only have I seen myself forced to write some very ugly code in order to convert between RDD[Array[String]] and Array[String] but it's essentially made my code useless.
I've tried different approaches and mixes between using
.map()
.flatMap() and
.reduceByKey()
however nothing actually put my collected "cells" into the format that I need them to be.
Here's what is supposed to happen: Reading a folder of text files from our server, the code should read each "line" of text in the format:
*---------*
| NASDAQ: |
*---------*
exchange, stock_symbol, date, stock_price_open, stock_price_high, stock_price_low, stock_price_close, stock_volume, stock_price_adj_close
and only keep a hold of the stock_symbol as that is the identifier I'm counting. So far my attempts have been to turn the entire thing into an array only collect every 9th index from the first one into a collected_cells var. Issue is, based on my calculations and real life results, that code would take 335 days to run (no joke).
Here's my current code for reference:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SparkNum {
def main(args: Array[String]) {
// Do some Scala voodoo
val sc = new SparkContext(new SparkConf().setAppName("Spark Numerical"))
// Set input file as per HDFS structure + input args
val lines = sc.textFile("hdfs://moonshot-ha-nameservice/" + args(0))
val fields = lines.map(line => line.split(","))
var collected_cells:Array[String] = new Array[String](0)
//println("[MESSAGE] Length of CC: " + collected_cells.length)
val divider:Long = 9
val array_length = fields.count / divider
val casted_length = array_length.toInt
val indexedFields = fields.zipWithIndex
val indexKey = indexedFields.map{case (k,v) => (v,k)}
println("[MESSAGE] Number of lines: " + array_length)
println("[MESSAGE] Casted lenght of: " + casted_length)
for( i <- 1 to casted_length ) {
println("[URGENT DEBUG] Processin line " + i + " of " + casted_length)
var index = 9 * i - 8
println("[URGENT DEBUG] Index defined to be " + index)
collected_cells :+ indexKey.lookup(index)
}
println("[MESSAGE] collected_cells size: " + collected_cells.length)
val single_cells = collected_cells.flatMap(collected_cells => collected_cells);
val counted_cells = single_cells.map(cell => (cell, 1).reduceByKey{case (x, y) => x + y})
// val result = counted_cells.reduceByKey((a,b) => (a+b))
// val inmem = counted_cells.persist()
//
// // Collect driver into file to be put into user archive
// inmem.saveAsTextFile("path to server location")
// ==> Not necessary to save the result as processing time is recorded, not output
}
}
The bottom part is currently commented out as I tried to debug it, but it acts as pseudo-code for me to know what I need done. I may want to point out that I am next to not at all familiar with Scala and hence things like the _ notation confuse the life out of me.
Thanks for your time.
There are some concepts that need clarification in the question:
When we execute this code:
val lines = sc.textFile("hdfs://moonshot-ha-nameservice/" + args(0))
val fields = lines.map(line => line.split(","))
That does not result in a huge array of the size of the data. That expression represents a transformation of the base data. It can be further transformed until we reduce the data to the information set we desire.
In this case, we want the stock_symbol field of a record encoded a csv:
exchange, stock_symbol, date, stock_price_open, stock_price_high, stock_price_low, stock_price_close, stock_volume, stock_price_adj_close
I'm also going to assume that the data file contains a banner like this:
*---------*
| NASDAQ: |
*---------*
The first thing we're going to do is to remove anything that looks like this banner. In fact, I'm going to assume that the first field is the name of a stock exchange that start with an alphanumeric character. We will do this before we do any splitting, resulting in:
val lines = sc.textFile("hdfs://moonshot-ha-nameservice/" + args(0))
val validLines = lines.filter(line => !line.isEmpty && line.head.isLetter)
val fields = validLines.map(line => line.split(","))
It helps to write the types of the variables, to have peace of mind that we have the data types that we expect. As we progress in our Scala skills that might become less important. Let's rewrite the expression above with types:
val lines: RDD[String] = sc.textFile("hdfs://moonshot-ha-nameservice/" + args(0))
val validLines: RDD[String] = lines.filter(line => !line.isEmpty && line.head.isLetter)
val fields: RDD[Array[String]] = validLines.map(line => line.split(","))
We are interested in the stock_symbol field, which positionally is the element #1 in a 0-based array:
val stockSymbols:RDD[String] = fields.map(record => record(1))
If we want to count the symbols, all that's left is to issue a count:
val totalSymbolCount = stockSymbols.count()
That's not very helpful because we have one entry for every record. Slightly more interesting questions would be:
How many different stock symbols we have?
val uniqueStockSymbols = stockSymbols.distinct.count()
How many records for each symbol do we have?
val countBySymbol = stockSymbols.map(s => (s,1)).reduceByKey(_+_)
In Spark 2.0, CSV support for Dataframes and Datasets is available out of the box
Given that our data does not have a header row with the field names (what's usual in large datasets), we will need to provide the column names:
val stockDF = sparkSession.read.csv("/tmp/quotes_clean.csv").toDF("exchange", "symbol", "date", "open", "close", "volume", "price")
We can answer our questions very easy now:
val uniqueSymbols = stockDF.select("symbol").distinct().count
val recordsPerSymbol = stockDF.groupBy($"symbol").agg(count($"symbol"))

Resources