I am in a Julia project and I am trying to write code for saving Matrix data as .json file.
The problem I am facing is when I read the json string from the file and parse it, the Matrix is changed into "vector of vectors".
In [1]:
n=10
mat = zeros(n,n)
data = Dict(
"n" => n,
"mat" => mat
)
using JSON
output_text = JSON.json(data)
out = open("jsonTest.json","w")
println(out,output_text)
close(out)
In [2]:
data
Out[2]:
Dict{String, Any} with 2 entries:
"mat" => [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.…
"n" => 10
In [3]:
op = JSON.parsefile("jsonTest.json")
Out[3]:
Dict{String, Any} with 2 entries:
"mat" => Any[Any[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], Any[0.0, …
"n" => 10
According to this question, one simple solution is using hcat function to re-convert "vector of vectors" to "matrix" when reading the json.
It works, but it is a bit messy(and might time-consuming) to take care of each matrix in the json. Is there simpler way?
Any information would be appreciated.
Since JSON (the spec) does not know about Matrix, you need to either do nested vectors (as you're right now), or reshape.
I personally prefer reshape because its memory layout is the same as an concrete Matrix in Julia and reshap has no allocation and less overhead over hcat
julia> a = rand(2,3)
2×3 Matrix{Float64}:
0.534246 0.282277 0.140581
0.841056 0.443697 0.427142
julia> as = JSON3.write(a)
"[0.5342463881378705,0.8410557102859995,0.2822771326129221,0.44369703601566,0.1405805564055571,0.4271417199755423]"
julia> reshape(JSON3.read(as), (2,3))
2×3 reshape(::JSON3.Array{Float64, Base.CodeUnits{UInt8, String}, Vector{UInt64}}, 2, 3) with eltype Float64:
0.534246 0.282277 0.140581
0.841056 0.443697 0.427142
Related
I have different time series ts_values turned to list and I want to predict the next itemss using an ARIMA model but it seems not to take care about zeroes:
row['shop_id']: 5 row['item_id']: 5037
[2599.0, 2599.0, 3998.0, 3998.0, 1299.0, 1499.0, 1499.0, 2997.5, 749.5, 0.0, 0.0, 0.0, 0.0]
predicted: 2599.019975890905
-------------------
row['shop_id']: 5 row['item_id']: 5320
predicted: 0
-------------------
row['shop_id']: 5 row['item_id']: 5233
[2697.0, 1198.0, 599.0, 2997.0, 1199.0, 0.0]
predicted: 2697.000099353263
-------------------
row['shop_id']: 5 row['item_id']: 5232
predicted: 0
-------------------
row['shop_id']: 5 row['item_id']: 5268
predicted: 0
-------------------
row['shop_id']: 5 row['item_id']: 5039
[5198.0, 6597.0, 2599.0, 5197.0, 749.5, 1499.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
predicted: 5198.0926378541535
So I wondered what did I did wrong there.
Here is my code:
import statsmodels.tsa.arima.model as smt
ranges = range(1, 5)
for difference in ranges:
# try:
tmp_model = smt.ARIMA(ts_values, order=(0, 1, 0), trend='t').fit()
tmp_aic = tmp_model.aic
if tmp_aic < best_aic:
best_aic = tmp_aic
best_difference = difference
best_model = tmp_model
# except Exception as e:
# print(e)
# continue
if best_model is not None:
y_hat = best_model.forecast()[0]
I know difference is of no use there. It was used for the coefficient of ARIMA. But I've been told that as far as my lists would be of a max size of 32 I should use a simple time forecasting (0,1,0).
Edit: version 0.12.2 of Statsmodels has been released, so your original code should now work.
Unfortunately, this is a known bug with ARIMA in v0.12.1. Until v0.12.2 is released, one option is to use SARIMAX instead:
from statsmodels.tsa.statespace import sarimax as smt
...
tmp_model = smt.SARIMAX(ts_values, order=(0, 1, 0), trend='t').fit()
I am using the eigs() function (from the Arpack package) to find the eigenvalues of a sparse matrix (eigen() doesn't work for spare matrices). Apparently, eigs() fails to find all the eigenvalues, even in a very simple case:
using Arpack
M = spdiagm(0 => [1,2,3])
eigs(M, nev = 3)
The output of the last line is a vector contacting only 2 eigenvalues, "2" and "3" ("1" is missing). What am I missing? Is there a different function for computing the eigenvalues of a sparse matrix (the actual sparse matrix is much large than the above M).
It actually shows at warning:
julia> eigs(Matrix(M), nev = 3);
┌ Warning: Adjusting nev from 3 to 2
└ # Arpack c:\JuliaPkg\Julia1.5.0\packages\Arpack\o35I5\src\Arpack.jl:82
Looking at the source code this can return a maximum of LinearAlgebra.checksquare(M) - 1 values.
What you could try to do is to use a BandedMatrix instead which is also sparse:
julia> m=BandedMatrix(0=>1:3)
3×3 BandedMatrix{Int64,Array{Int64,2},Base.OneTo{Int64}}:
1 ⋅ ⋅
⋅ 2 ⋅
⋅ ⋅ 3
julia> eigen(m)
Eigen{Float64,Float64,Array{Float64,2},Array{Float64,1}}
values:
3-element Array{Float64,1}:
1.0
2.0
3.0
vectors:
3×3 Array{Float64,2}:
1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0
eigs uses an iterative method that is only practical or encouraged for large sparse matrices and when you only want a small number of eigenvalues (small compared to the size of the matrix).
The error message you got in eigen is a little unfortunate. Preferably, it would have also mentioned that if your array is small, you should just collect it into a dense matrix, e.g.
julia> using SparseArrays, LinearAlgebra
julia> M = spdiagm(0 => [1,2,3])
3×3 SparseMatrixCSC{Int64,Int64} with 3 stored entries:
[1, 1] = 1
[2, 2] = 2
[3, 3] = 3
julia> eigen(collect(M))
Eigen{Float64,Float64,Array{Float64,2},Array{Float64,1}}
values:
3-element Array{Float64,1}:
1.0
2.0
3.0
vectors:
3×3 Array{Float64,2}:
1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0
If your matrix is too big to fit into memory and you must use a sparse matrix, then I suspect you don't need all the eigenvalues.
I am using Zygote in Julia, and I have some code that is supposed to loop through two objects of the same dimensions and, for every item of length less than <= 1, randomly choose between that part of/for either the first or the second object. The code may better explain what I am attempting to do:
function rangeThrough(parent1, parent2)
child = map(x -> if length(x[1]) > 1 rangeThrough(x[1], x[2]) else rand([x[1] x[2]]) end, zip(parent1, parent2))
return child
end
aa = [Float32[0. 0. 0. 0.], Float32[0. 0. 0.], Float32[0.]]
bb = [Float32[1. 1. 1. 1.], Float32[1. 1. 1.], Float32[1.]]
Zygote.Params(rangeThrough(aa, bb))
The output that I get for the last line of code is something like the following:
Params([Float32[1.0 1.0 1.0 0.0], Float32[0.0 0.0 1.0], 0.0])
However, what I am trying to achieve is:
Params([Float32[1.0 1.0 1.0 0.0], Float32[0.0 0.0 1.0], Float32[0.0]])
I have tried a few things but wasn't able to get it working yet. I am not sure why it is that the last number is not contained in a Float32[] array. What could I do differently to correct this?
Edit:
By simplifying the code to only accept one list as input, like so:
function tempRangeThrough(parent1)
child = map(x -> if length(x) > 1 tempRangeThrough(x) else rand([x]) end, parent1)
return child
end
Zygote.Params(tempRangeThrough(aa))
I get the intended output:
Params([Float32[0.0 0.0 0.0 0.0], Float32[0.0 0.0 0.0], Float32[0.0]])
So, I believe this is related to the properties of zip(), but I'm not positive how to fix this.
You might want something like this:
julia> rangeThrough(p1, p2) = [length(a) > 1 ? a : rand((a,b)) for (a, b) in zip(p1, p2)]
rangeThrough (generic function with 1 method)
julia> rangeThrough(aa, bb)
3-element Array{Array{Float32,N} where N,1}:
[0.0 0.0 0.0 0.0]
[0.0 0.0 0.0]
[1.0]
I didn't really get what behaviour you seek exactly, but wanted to mention that map can take n-ary functions, too -- so you don't need zip here. And combining with a do-block, the function could look as follows:
julia> range_through(parent1, parent2) = map(parent1, parent2) do x, y
if length(x) > 1
return x
else
return rand((x, y))
end
end
range_through (generic function with 1 method)
julia> range_through(aa, bb)
3-element Array{Array{Float32,N} where N,1}:
[0.0 0.0 0.0 0.0]
[0.0 0.0 0.0]
[1.0]
I was using find with a 3D matrix A like this:
julia> find(A.==1)
2-element Array{Int64,1}:
1
234
4567
Julia gives me the location as an index instead of as an array of indices. For example, it returns 234 instead of (1,2,1).
I looked at this question, but my matrix is very large and has a shape of (360,360,360). I can't use the method suggested there.
I tried to study its index pattern and transform it using a function that I coded:
function cmf_p(matrix)
for a=1:length(matrix);
aa=matrix[a]
rd_u_m=ceil(aa/(360^2))
rd_d_m=floor(aa/(360^2)-1)
rd_d_t=(aa-rd_d_m*360)/360^2
rd_d_p=aa-rd_d_m*360^2-floor(rd_d_t)*360
println(rd_u_m);
println(ceil(rd_d_t)*360);
println(ceil(aa-rd_d_m*360^2-floor(rd_d_t)*360))
end
end
But it gives me the wrong result.
How can I use the index and transform it to the location I want?
You are looking for ind2sub:
julia> A = eye(3)
3x3 Array{Float64,2}:
1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0
julia> inds = find(A.==1.0)
3-element Array{Int64,1}:
1
5
9
julia> [ind2sub(size(A), ind) for ind in inds]
3-element Array{Any,1}:
(1,1)
(2,2)
(3,3)
I have a 4x4 array of zeros.
julia> X = zeros(4,4)
4x4 Array{Float64,2}:
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
I have an 2xN array containing indices of elements in X that I want to assign a new value.
julia> ind = [1 1; 2 2; 3 3]
3x2 Array{Int64,2}:
1 1
2 2
3 3
What is the simplest way to assign a value to all elements in X whose indices are rows in ind? (something like X[ind] = 2.0).
julia> X
2.0 0.0 0.0 0.0
0.0 2.0 0.0 0.0
0.0 0.0 2.0 0.0
0.0 0.0 0.0 0.0
I'm not sure there is a non-looping way to do this. What's wrong with this?
for i=[1:size(ind)[1]]
a, b = ind[i, :]
X[a, b] = 2.0
end
user3467349's answer is correct, but inefficient, because it allocates an Array for the indices. Also, the notation [a:b] is deprecated as of Julia 0.4. Instead, you can use:
for i = 1:size(ind, 1)
a, b = ind[i, :]
X[a, b] = 2.0
end