I have a data structure that I have loaded in from json that resembles the below
json_in =
[ Dict("customer" => "cust1", "transactions" => 1:10^6)
, Dict("customer" => "cust2", "transactions" => 1:10^6)
, Dict("customer" => "cust3", "transactions" => 1:10^6)]
I know of two methods to collapse the transactions into one array
#time methodA = reduce(vcat,[cust["transactions"] for cust in json_in])
#time methodB = vcat(json_in[1]["transactions"],json_in[2]["transactions"],json_in[3]["transactions"])
However the timing of methodA is ~0.22s vs ~0.02s for methodB on my computer. I intend to perform this thousands of times so 10x quicker performance is a big deal.
I see methodB is not very robust as it can only deal with 3 Dicts (customers) so even though it's performant it doesn't generalise.
What would be the most efficient way to concatenate arrays that are elements in an array of Dict efficiently?
As #Gnimuc states in his comment, you should not benchmark in global scope, and benchmarks are best done using BenchmarkTools.jl - here are the timings done right:
julia> methodA(json_in) = reduce(vcat,[cust["transactions"] for cust in json_in])
method1 (generic function with 1 method)
julia> methodB(json_in) = vcat(json_in[1]["transactions"],json_in[2]["transactions"],json_in[3]["transactions"])
method2 (generic function with 1 method)
#Gnimuc's syntax from his comment
julia> methodC(json_in) = mapreduce(x->x["transactions"], vcat, json_in)
method3 (generic function with 1 method)
julia> using BenchmarkTools
julia> #benchmark methodA(json_in)
BenchmarkTools.Trial:
memory estimate: 38.15 MiB
allocs estimate: 15
--------------
minimum time: 10.584 ms (3.10% GC)
median time: 14.781 ms (32.02% GC)
mean time: 15.112 ms (32.19% GC)
maximum time: 69.341 ms (85.28% GC)
--------------
samples: 331
evals/sample: 1
julia> #benchmark methodB(json_in)
BenchmarkTools.Trial:
memory estimate: 22.89 MiB
allocs estimate: 2
--------------
minimum time: 5.921 ms (5.92% GC)
median time: 8.402 ms (32.48% GC)
mean time: 8.701 ms (33.46% GC)
maximum time: 69.268 ms (91.09% GC)
--------------
samples: 574
evals/sample: 1
julia> #benchmark methodC(json_in)
BenchmarkTools.Trial:
memory estimate: 38.15 MiB
allocs estimate: 12
--------------
minimum time: 10.599 ms (3.37% GC)
median time: 14.843 ms (32.12% GC)
mean time: 15.228 ms (32.24% GC)
maximum time: 71.954 ms (85.95% GC)
--------------
samples: 328
evals/sample: 1
Method B is still like twice as fast. That is exactly because it is more specialized, on an array with exactly three elements.
An alternative solution that might work well here is to use a MappedArray, which creates a lazy view into the original array:
using MappedArrays
method4(json_in) = mappedarray(x->x["transactions"], json_in)
Of course this doesn't concatenate the arrays, but you can concatenate views using the CatView package:
using CatViews
julia> method5(json_in) = reduce(CatView, mappedarray(x->x["transactions"], json_in))
method5 (generic function with 1 method)
julia> #benchmark method5(json_in)
BenchmarkTools.Trial:
memory estimate: 1.73 KiB
allocs estimate: 46
--------------
minimum time: 23.320 μs (0.00% GC)
median time: 23.916 μs (0.00% GC)
mean time: 25.466 μs (0.00% GC)
maximum time: 179.092 μs (0.00% GC)
--------------
samples: 10000
evals/sample: 1
Because it doesn't allocate it is like 300x faster than method B (but it's possible it's slower to use the result because of nonlocality - worth benchmarking).
Thanks for the help, after some research I came up with this idea to inline expand the code using macros, see code below, and it performs pretty well on the benchmarks (on Juliabox.com 21Sep2017)
macro inline_vcat(a)
quote
astr = $(string(a))
s = reduce(string, string(astr,"[",aa,"][\"transactions\"],") for aa in 1:length($a))
string("vcat(", s[1:(end-1)],")")
end
end
methodE(json_in) = (#inline_vcat json_in) |> parse |> eval
using BenchmarkTools
#benchmark methodE(json_in)
One shortcoming of this method is that if there are a large (~1million) customers in the JSON then the code generated will be long and parsing it would take a long time I assume well. Hence it's probably not a good idea for large datasets.
Related
Consider the following Vector:
numbers = Int32[1,2,3,4,5,6,7,8,9,10]
If I want to create a 2x5 matrix with the result:
1 2 3 4 5
6 7 8 9 10
I can't use reshape(numbers,2,5) or else I'll get:
1 3 5 7 9
2 4 6 8 10
Using slice or view(), you can extract the top row and bottom row, convert them to a matrix row, and then use vcat().
I'm not saying using slice or view() is the only or best way of doing it, perhaps there is a faster way using reshape(), I just haven't figured it out.
numbers = Int32[1,2,3,4,5,6,7,8,9,10]
println("Using Slice:")
#time numbers_slice_matrix_top = permutedims(numbers[1:5])
#time numbers_slice_matrix_bottom = permutedims(numbers[6:10])
#time vcat(numbers_slice_matrix_top,numbers_slice_matrix_bottom)
println("Using view():")
#time numbers_view_matrix_top = permutedims(view(numbers,1:5))
#time numbers_view_matrix_bottom = permutedims(view(numbers,6:10))
#time vcat(numbers_view_matrix_top,numbers_view_matrix_bottom)
Output:
Using Slice:
0.026763 seconds (5.48 k allocations: 329.155 KiB, 99.78% compilation time)
0.000015 seconds (3 allocations: 208 bytes)
0.301833 seconds (177.09 k allocations: 10.976 MiB, 93.30% compilation time)
Using view():
0.103084 seconds (72.25 k allocations: 4.370 MiB, 99.90% compilation time)
0.000011 seconds (2 allocations: 112 bytes)
0.503787 seconds (246.63 k allocations: 14.537 MiB, 99.85% compilation time)
Why is slice faster? In a few rare cases view() was faster, but not by much.
From view() documentation:
For example, if x is an array and v = #view x[1:10], then v acts like
a 10-element array, but its data is actually accessing the first 10
elements of x. Writing to a view, e.g. v[3] = 2, writes directly to
the underlying array x (in this case modifying x[3]).
I don't know enough, but from my understanding, because view() has to convert the Vector to a matrix row (the original Vector) through another array (the view()), it's slower. Using slice we create a copy and don't have to worry about manipulating the original Vector.
Your results actually show that view is faster not slicing. The point is that only the second tests is measuring the time to run the code while in the tests 1 and 3 you are measuring the time to compile the code.
This is a common misunderstanding how to run benchmarks in Julia. The point is that when a Julia function is run for the first time it needs to be compiled to an assembly code. Normally in production codes compile times do not matter because you compile only once for a fraction of a second and then run computations for many minutes, hours or days.
More than that - your code is using a global variable so in such a microbenchmark you are also measuring "how long does it take to resolve a global variable type" which is slow in Julia and not used in a production code.
Here is the correct way to run the benchmark using BenchmarkTools:
julia> #btime vcat(permutedims($numbers[1:5]),permutedims($numbers[6:10]));
202.326 ns (7 allocations: 448 bytes)
julia> #btime vcat(permutedims(view($numbers,1:5)),permutedims(view($numbers,6:10)));
88.736 ns (1 allocation: 96 bytes)
Note the interpolation symbol $ that makes numbers a type stable variable.
reshape(numbers, 5, 2)' can be used also to create the desired 2x5 matrix.
If I have a vector of sets, say,
vec_of_sets = [Set(vec1), Set(vec2), ..., Set(vecp)]
how do I obtain a set equal to the union of sets in the vector? That is, how can I write the following efficiently?
S1 = Set(vec1);
union!(S1, Set(vec2))
union!(S1, Set(vec3))
...
union!(S1, Set(vecp))
I don't really know where to start!
Thanks in advance.
Edit: I have tried a solution using generating functions but it doesn't work:
union(j for j in vec_of_sets)
The best and fastest approach is:
Set(Iterators.flatten(vec_of_sets))
It is around twice as fast as other possible approaches proposed at the other post and has makes than half memory allocations.
Here are some benchmarks:
julia> v = [Set(1:3), Set(2:6), Set(4:8)];
julia> #btime Set(Iterators.flatten($v));
270.492 ns (4 allocations: 400 bytes)
julia> #btime reduce(union, $v);
550.000 ns (11 allocations: 1.25 KiB)
julia> #btime union($v...);
506.250 ns (11 allocations: 944 bytes)
julia> #btime union((j for j in $v)...);
699.286 ns (15 allocations: 1.03 KiB)
I guess you should use reduce:
reduce(union, vec_of_sets)
but you could also use splatting (with ...):
union(vec_of_sets...)
FWIW, you could have used splitting with your attempt, too:
union((j for j in vec_of_sets)...)
I want to turn a array of arrays into a matrix. To illustrate; let the array of arrays be:
[ [1,2,3], [4,5,6], [7,8,9]]
I would like to turn this into the 3x3 matrix:
[1 2 3
4 5 6
7 8 9]
How would you do this in Julia?
There are several ways of doing this. For instance, something along the lines of vcat(transpose.(a)...) will work as a one-liner
julia> a = [[1,2,3], [4,5,6], [7,8,9]]
3-element Vector{Vector{Int64}}:
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
julia> vcat(transpose.(a)...)
3×3 Matrix{Int64}:
1 2 3
4 5 6
7 8 9
though note that
Since your inner arrays are column-vectors as written, you need to transpose them all before you can vertically concatenate (aka vcat) them (either that or horizontally concatenate and then transpose the whole result after, i.e., transpose(hcat(a...))), and
The splatting operator ... which makes this one-liner work will not be very efficient when applied to Arrays in general, and especially not when applied to larger arrays-of-arrays.
Performance-wise for larger arrays-of-arrays, it will likely actually be hard to beat preallocating a result of the right size and then simply filling with a loop, e.g.
result = similar(first(a), length(a), length(first(a)))
for i=1:length(a)
result[i,:] = a[i] # Aside: `=` is actually slightly faster than `.=` here, though either will have the same practical result in this case
end
Some quick benchmarks for reference:
julia> using BenchmarkTools
julia> #benchmark vcat(transpose.($a)...)
BechmarkTools.Trial: 10000 samples with 405 evaluations.
Range (min … max): 241.289 ns … 3.994 μs ┊ GC (min … max): 0.00% … 92.59%
Time (median): 262.836 ns ┊ GC (median): 0.00%
Time (mean ± σ): 289.105 ns ± 125.940 ns ┊ GC (mean ± σ): 2.06% ± 4.61%
▁▆▇█▇▆▅▅▅▄▄▄▄▃▂▂▂▃▃▂▂▁▁▁▂▄▃▁▁ ▁ ▁ ▂
████████████████████████████████▇▆▅▆▆▄▆▆▆▄▄▃▅▅▃▄▆▄▁▃▃▃▅▄▁▃▅██ █
241 ns Histogram: log(frequency) by time 534 ns <
Memory estimate: 320 bytes, allocs estimate: 5.
julia> #benchmark for i=1:length($a)
$result[i,:] = $a[i]
end
BechmarkTools.Trial: 10000 samples with 993 evaluations.
Range (min … max): 33.966 ns … 124.918 ns ┊ GC (min … max): 0.00% … 0.00%
Time (median): 36.710 ns ┊ GC (median): 0.00%
Time (mean ± σ): 39.795 ns ± 7.566 ns ┊ GC (mean ± σ): 0.00% ± 0.00%
▄▄██▄▅▃ ▅▃ ▄▁▂ ▂▁▂▅▂▁ ▄▂▁ ▂
██████████████▇██████▆█▇▆███▆▇███▇▆▆▅▆▅▅▄▄▅▄▆▆▆▄▁▃▄▁▃▄▅▅▃▁▄█ █
34 ns Histogram: log(frequency) by time 77.7 ns <
Memory estimate: 0 bytes, allocs estimate: 0.
In general, filling column-by-column (if possible) will be faster than filling row-by-row as we have done here, since Julia is column-major.
Expanding on #cbk's answer, another (slightly more efficient) one-liner is
julia> transpose(reduce(hcat, a))
3×3 transpose(::Matrix{Int64}) with eltype Int64:
1 2 3
4 5 6
7 8 9
[1 2 3; 4 5 6; 7 8 9]
# or
reshape(1:9, 3, 3)' # remember that ' makes the transpose of a Matrix
I have a function, which returns a two dimensional Array:
2-element Array{Float64,1}:
0.809919
2.00754
I now want to efficiently sample over it and store all the results in an array with 2 rows and n columns. The problem is that I get a Vector of vectors. How could I flatten it or construct it?
A toy example is the following:
julia> [rand(2) for i=1:3]
3-element Array{Array{Float64,1},1}:
[0.906644, 0.614673]
[0.426492, 0.67645]
[0.473704, 0.726284]
julia> [rand(2)' for i=1:3]
3-element Array{RowVector{Float64,Array{Float64,1}},1}:
[0.403384 0.431918]
[0.410625 0.546614]
[0.224933 0.118778]
And I would like to have the result in a form like this:
julia> [rand(2) rand(2) rand(2)]
2×3 Array{Float64,2}:
0.360833 0.205969 0.209643
0.507417 0.317295 0.588516
Actually my dream would be:
julia> [rand(2) rand(2) rand(2)]'
3×2 Array{Float64,2}:
0.0320955 0.821869
0.358808 0.26685
0.230355 0.31273
Any ideas? I know that I could construct it via a for loop, but was looking for a more efficient way.
Thanks!
RecursiveArrayTools.jl has a VectorOfArray type which dispatches in the way you'd want:
julia> using RecursiveArrayTools
julia> A = [rand(2) for i=1:3]
3-element Array{Array{Float64,1},1}:
[0.957228, 0.104218]
[0.293985, 0.83882]
[0.788157, 0.454772]
julia> VectorOfArray(A)'
3×2 Array{Float64,2}:
0.957228 0.104218
0.293985 0.83882
0.788157 0.454772
As for timing:
julia> #benchmark VectorOfArray(A)'
BenchmarkTools.Trial:
memory estimate: 144 bytes
allocs estimate: 2
--------------
minimum time: 100.658 ns (0.00% GC)
median time: 111.740 ns (0.00% GC)
mean time: 127.159 ns (3.29% GC)
maximum time: 1.360 μs (82.71% GC)
--------------
samples: 10000
evals/sample: 951
VectorOfArray itself is almost no overhead, and the ' uses the Cartesian indexing to be fast.
Something along these lines
using BenchmarkTools
function createSample!(vec::AbstractVector)
vec .= randn(length(vec))
return vec
end
function createSamples!(A::Matrix)
for row in indices(A, 1)
createSample!(view(A, row, :))
end
return A
end
A = zeros(10, 2)
#benchmark createSamples!(A)
might help. The timing on my laptop gives:
Main> #benchmark createSamples!(A)
BenchmarkTools.Trial:
memory estimate: 1.41 KiB
allocs estimate: 20
--------------
minimum time: 539.104 ns (0.00% GC)
median time: 581.194 ns (0.00% GC)
mean time: 694.601 ns (13.34% GC)
maximum time: 10.324 μs (90.10% GC)
--------------
samples: 10000
evals/sample: 193
For example, [1 1 ; 2 2 ; 3 3] becomes
[1 1
1 1
1 1
2 2
2 2
2 2
3 3
3 3
3 3]
I am using this:
expander(orig,mult::Int) = orig[ceil(Int,(1:size(orig,1)*mult)/mult),:]; in Julia and the following in Matlab:
function expanded = expander(original,multiplier)
expanded = original(ceil((1:size(original,1)*multiplier)/multiplier),:);
end
Another matlab only way to do it is this:
expanded = kron(original,ones(multiplier,1));
I would prefer a superfast julia option if it exists.
This doesn't prove that kron is fastest, but I compared its time to how long it would just take to populate a similarly sized Array with ones, and kron did quite well:
original = [1 1 ; 2 2 ; 3 3];
multiplier = 3*10^6;
#time begin
for idx = 1:100
expanded = kron(original,ones(multiplier));
end
end
## 9.199143 seconds (600 allocations: 15.646 GB, 9.05% gc time)
#time begin
for idx = 1:100
myones = [ones(multiplier*size(original,1)) ones(multiplier*size(original,1))];
end
end
## 12.746123 seconds (800 allocations: 26.822 GB, 14.86% gc time)
Update In response to comments by David Sanders, here are tests wrapped in a function. The reason I did the tests globally, which I know isn't normal best practice, is because it seemed quite plausible to me that the objects might get created globally.
function kron_test(original, multiplier)
for idx = 1:100
expanded = kron(original,ones(multiplier));
end
end
function ones_test(original, multiplier)
for idx = 1:100
myones = [ones(multiplier*size(original,1)) ones(multiplier*size(original,1))];
end
end
## times given after first function call to compile
#time kron_test(original, multiplier); ## 11.107632 seconds (604 allocations: 15.646 GB, 23.98% gc time)
#time ones_test(original, multiplier); ## 15.849761 seconds (604 allocations: 26.822 GB, 33.50% gc time)
Personally, I'd just use repeat:
repeat(original, inner=(multiplier, 1))
Unlike kron, it's very readable and understandable. Unfortunately it is quite a bit slower. Even so, I'd only use kron if you've identified it as a performance bottleneck. While it's faster for computers to execute, it's much slower for humans to understand what's going on… and the performance of repeat should eventually get better (it's issue #15553).
Edit: As of Julia 1.2, repeat has indeed gotten significantly faster. It now rivals kron:
julia> #btime kron($original,ones($multiplier));
81.039 ms (6 allocations: 160.22 MiB)
julia> #btime repeat($original, inner=($multiplier, 1));
84.087 ms (27 allocations: 137.33 MiB)
You could do
a = [1 1 ; 2 2 ; 3 3]
a = a' #matrices are in column major order in julia, should be faster this way
a = repmat(a,1,n)
a = sortcols(a)
Unfortunatelly I have no clue wheter this method is "superfast" but it's relatively simple and intuitive