Consider this function
using Distributions
using StatsBase
test_vector = sample([1,2,3], 100000000)
function test_1( test_vector)
rand_vector = randn(3333)
sum = 0.0
for t in 1:1000
if t in test_vector
sum = sum +rand_vector[t]
else
sum = sum - rand_vector[t]
end
end
end
I applied #profview to understand performance and it turns out most of the time is spent on if t in test_vector. Is there a way to speed up this part of the program? I thought about excluding test_vector from 1:1000 and run two loops, but this creates memory allocation. Can I get a hint?
P.S. I intend to let the user pass in any test_vector. I'm using sample to create a test_vector just for illustration.
If the vector is large, changing the check for element membership will be faster if you create a Set for the check:
using Distributions
using StatsBase
using BenchmarkTools
test_vector = sample([1,2,3], 1000000)
function test_1(test_vector)
rand_vector = randn(3333)
sum1 = 0.0
for t in 1:1000
if t in test_vector
sum1 = sum1 + rand_vector[t]
else
sum1 = sum1 - rand_vector[t]
end
end
return sum1
end
function test_1_set(test_vector)
rand_vector = randn(3333)
test_set = Set(test_vector)
sum2 = 0.0
for t in 1:1000
if t in test_set
sum2 += rand_vector[t]
else
sum2 -= rand_vector[t]
end
end
return sum2
end
#btime test_1(test_vector)
#btime test_1_set(test_vector)
677.818 ms (3 allocations: 26.12 KiB)
8.795 ms (10 allocations: 18.03 MiB)
Bill's version is the fastest if you don't count the construction of the set:
julia> test_vector = rand(1:3, 10000);
julia> rand_vector = randn(3333);
julia> range = 1:1000;
julia> #btime test_1($(Set(test_vector)), $rand_vector, $range)
3.692 μs (0 allocations: 0 bytes)
14.82533505498519
However, the set creation itself is more costly, especially in terms of memory:
julia> #btime test_1(Set($test_vector), $rand_vector, $range)
52.731 μs (7 allocations: 144.59 KiB)
14.82533505498519
Here's a variant more optimized for memory:
julia> function test_2(xs, ys, range)
range = Set(range)
positive = intersect(range, xs)
negative = setdiff!(range, positive)
return sum(ys[i] for i in positive) - sum(ys[i] for i in negative)
end
test_2 (generic function with 1 method)
julia> #btime test_2($test_vector, $rand_vector, $range)
96.020 μs (11 allocations: 18.98 KiB)
14.825335054985187
Use a Set. They have O(1) lookup.
Related
I've used Julia for some time, but still know little about it, especially for the paralell computing.
I want to obtain a new array with the existing data, as the array is very large, I want to do it in a parallel way, and the code is written as follows:
ψ1 = Array{Complex}(undef, D)
ψ = rand(Complex{Float64},D)
Threads.#threads for k = 1:D
ψ1[k] = #views sum(ψ[GetBasis(k - 1, N)])
end
I run it with julia -t 4, But it turn out to be very slow, compared to the non parallel code as follows,
ψ1 = [#views sum(ψ[GetBasis(k - 1, N)]) for k=1:D]
I have no idea about why this happens, and GetBasis() is just a function to generate a Array, Array{Int64,1}(N).
I would like to ask how I could improve the first code, or, is there some way I can modify the second code to also run it in a parallel way? As the array can be very large, and I want to find a way to speed it up...
Thanks a lot, and look forward to your replies!
The complete code can be found as follows
function GetBasis(n, N)
xxN = collect(0:N-1)
BI = BitFlip.(n,xxN).+1
end
function BitFlip(n, i)
n = n ⊻ (1 << i)
end
N=24
D=2^N
ψ1 = Array{Complex}(undef, D)
ψ = rand(Complex{Float64},D)
Threads.#threads for k = 1:D
ψ1[k] = #views sum(ψ[GetBasis(k - 1, N)])
end
ψ2 = [#views sum(ψ[GetBasis(k - 1, N)]) for k=1:D]
ψ1 = Array{Complex}(undef, D)
This is an abstract type, which is very slow.
You should define the type of the complex components, e.g. Complex{Float64}.
Alternatively, use similar.
When all is done properly the multithreaded code is faster indeed.
function GetBasis(n, N)
BI = BitFlip.(n,0:N-1).+1
end
function BitFlip(n, i)
n = n ⊻ (1 << i)
end
const N=24
const D=2^N
const ψ = rand(Complex{Float64},D)
const ψ1 = Vector{Complex{Float64}}(undef, D)
using BenchmarkTools
Threads.nthreads() # should return 4 or more
# set JULIA_NUM_THREADS environment variable
Now testing:
julia> GC.gc()
julia> #btime ψ2 = [#views sum(ψ[GetBasis(k - 1, N)]) for k=1:D];
5.591 s (16777218 allocations: 4.50 GiB)
julia> GC.gc()
julia> #btime Threads.#threads for k = 1:D
#inbounds ψ1[k] = #views sum(ψ[GetBasis(k - 1, N)])
end
2.293 s (16777237 allocations: 4.25 GiB)
Note the amount of memory used by this code - you need to run garbage collector before running the benchamrk and the test will be less meaningful when you have less than 16GB RAM in your machine.
I have to code a very simple for loop in Julia, which I reproduce below:
result=fill([],6,1)
E=rand(5,5)
D=3.27
k=2
for s in [0.5,0.75,1,1.25,1.5,2]
result[??]=exp.(-(E.^2/D)/(2*s*k))
end
At each iteration, I want that the i-th element of result is filled with the result of the function, which uses the i-th element of the array [0.5,0.75,1,1.25,1.5,2]. So I don't know what to put inside of the brackets [??].
So far, I tried
for (index, value) in enumerate([0.5,0.75,1,1.25,1.5,2])
result["$index"]=exp.(-(E.^2/D)/(2* "$value" *k))
end
but it doesn't work. Any hint?
You're currently initialising the results to be a one dimensional array, but they're actually two dimension. So you need to switch the results as follows
result = fill(Array{Float64}(undef,0,0),6,1)
You shouldn't need to do any conversion of the types and the following will just work.
for (index, value) in enumerate([0.5,0.75,1,1.25,1.5,2])
result[index]=exp.(-(E.^2/D)/(2*value*k))
end
Rather than initialising the results, you can just map across the values as well which becomes a bit easier to read.
result = map(x -> exp.(-(E.^2/D)/(2*x*k)), [0.5, 0.75, 1, 1.25, 1.5, 2])
Some comments on performance
using BenchmarkTools
function t1()
result=fill(Array{Float64}(undef,0,0),6,1)
E=rand(5,5)
D=3.27
k=2
for (index, value) in enumerate([0.5,0.75,1,1.25,1.5,2])
result[index]=exp.(-(E.^2/D)/(2*value*k))
end
end
function t2()
E=rand(5,5)
D=3.27
k=2
result = map(x -> exp.(-(E.^2/D)/(2*x*k)), [0.5, 0.75, 1, 1.25, 1.5, 2])
end
#btime t1() # 4.904 μs (49 allocations: 9.66 KiB)
#btime t2() # 4.812 μs (50 allocations: 9.64 KiB)
As you can see, no real difference in the performance. If you want to improve performance then it's easiest to try and pull the constants out of the inner loop.
function t3()
E=rand(5,5)
D=3.27
k=2
f = -(E.^2/D)/(2*k)
result = map(x -> exp.(f/x), [0.5, 0.75, 1, 1.25, 1.5, 2])
end
#btime t3() # 3.168 μs (31 allocations: 5.53 KiB)
Assuming result should be a vector of matrices:
els = [0.5,0.75,1,1.25,1.5,2]
result=Vector{Matrix{Float64}}(undef, length(els))
E=rand(5,5)
D=3.27
k=2
for s in 1:length(els)
result[s]=exp.(-(E.^2/D)/(2*s*k))
end
I have the following line of code in Julia:
X=[(i,i^2) for i in 1:100 if i^2%5==0]
Basically, it returns a list of tuples (i,i^2) from i=1 to 100 if the remainder of i^2 and 5 is zero. What I want to do is, in the array comprehension, break out of the for loop if i^2 becomes larger than 1000. However, if I implement
X=[(i,i^2) for i in 1:100 if i^2%5==0 else break end]
I get the error: syntax: expected "]".
Is there any way to easily break out of this for loop inside the array? I've tried looking online, but nothing came up.
It's a "fake" for-loop, so you can't break it. Take a look at the lowered code below:
julia> foo() = [(i,i^2) for i in 1:100 if i^2%5==0]
foo (generic function with 1 method)
julia> #code_lowered foo()
LambdaInfo template for foo() at REPL[0]:1
:(begin
nothing
#1 = $(Expr(:new, :(Main.##1#3)))
SSAValue(0) = #1
#2 = $(Expr(:new, :(Main.##2#4)))
SSAValue(1) = #2
SSAValue(2) = (Main.colon)(1,100)
SSAValue(3) = (Base.Filter)(SSAValue(1),SSAValue(2))
SSAValue(4) = (Base.Generator)(SSAValue(0),SSAValue(3))
return (Base.collect)(SSAValue(4))
end)
The output shows that array comprehension is implemented via Base.Generator which takes an iterator as input. It only supports the [if cond(x)::Bool] "guard" for now, so there is no way to use break here.
For your specific case, a workaround is to use isqrt:
julia> X=[(i,i^2) for i in 1:isqrt(1000) if i^2%5==0]
6-element Array{Tuple{Int64,Int64},1}:
(5,25)
(10,100)
(15,225)
(20,400)
(25,625)
(30,900)
I don't think so. You could always just
tmp(i) = (j = i^2; j > 1000 ? false : j%5==0)
X=[(i,i^2) for i in 1:100 if tmp(i)]
Using a for loop is considered idiomatic in Julia and could be more readable in this instance. Also, it could be faster.
Specifically:
julia> using BenchmarkTools
julia> tmp(i) = (j = i^2; j > 1000 ? false : j%5==0)
julia> X1 = [(i,i^2) for i in 1:100 if tmp(i)];
julia> #btime [(i,i^2) for i in 1:100 if tmp(i)];
471.883 ns (7 allocations: 528 bytes)
julia> X2 = [(i,i^2) for i in 1:isqrt(1000) if i^2%5==0];
julia> #btime [(i,i^2) for i in 1:isqrt(1000) if i^2%5==0];
281.435 ns (7 allocations: 528 bytes)
julia> function goodsquares()
res = Vector{Tuple{Int,Int}}()
for i=1:100
if i^2%5==0 && i^2<=1000
push!(res,(i,i^2))
elseif i^2>1000
break
end
end
return res
end
julia> X3 = goodsquares();
julia> #btime goodsquares();
129.123 ns (3 allocations: 304 bytes)
So, another 2x improvement is nothing to disregard and the long function gives plenty of room for illuminating comments.
In julia we can check if an array contains a value, like so:
> 6 in [4,6,5]
true
However this returns false, when attempting to check for a sub-array in a specific order:
> [4,6] in [4,6,5]
false
What is the correct syntax to verify if a specific sub-array exists in an array?
I think it is worth mentioning that in Julia 1.0 you have the function issubset
> issubset([4,6], [4,6,5])
true
You can also quite conveniently call it using the \subseteq latex symbol
> [4,6] ⊆ [4,6,5]
true
This looks pretty optimized to me:
> using Random
> x, y = randperm(10^3)[1:10^2], randperm(10^3);
> #btime issubset(x, y);
16.153 μs (12 allocations: 45.96 KiB)
It takes a little bit of code to make a function that performs well, but this is much faster than the issubvec version above:
function subset2(x,y)
lenx = length(x)
first = x[1]
if lenx == 1
return findnext(y, first, 1) != 0
end
leny = length(y)
lim = length(y) - length(x) + 1
cur = 1
while (cur = findnext(y, first, cur)) != 0
cur > lim && break
beg = cur
#inbounds for i = 2:lenx
y[beg += 1] != x[i] && (beg = 0 ; break)
end
beg != 0 && return true
cur += 1
end
false
end
Note: it would also be much more useful if the function actually returned the position of the beginning of the subarray if found, or 0 if not, similarly to the findfirst/findnext functions.
Timing information (the second one is using my subset2 function):
0.005273 seconds (65.70 k allocations: 4.073 MB)
0.000086 seconds (4 allocations: 160 bytes)
For the third condition i.e. vector [4,6] appears as a sub-vector of 4,6,5 the following function is suggested:
issubvec(v,big) =
any([v == slice(big,i:(i+length(v)-1)) for i=1:(length(big)-length(v)+1)])
For the second condition, that is, give a boolean for each element in els vectors which appears in set vector, the following is suggested:
function vecin(els,set)
res = zeros(Bool,size(els))
res[findin(els,set)]=true
res
end
With the vector in the OP, these result in:
julia> vecin([4,6],[4,6,5])
2-element Array{Bool,1}:
true
true
julia> issubvec([4,6],[4,6,5])
true
note that you can now vectorize in with a dot:
julia> in([4,6,5]).([4, 6])
2-element BitArray{1}:
true
true
and chain with all to get your answer:
julia> all(in([4,6,5]).([4, 6]))
true
I used this recently to find subsequences in arrays of integers. It's not as good or as fast as #scott's subset2(x,y)... but it returns the indices.
function findsequence(arr::Array{Int64}, seq::Array{Int64})
indices = Int64[]
i = 1
n = length(seq)
if n == 1
while true
occurrence = findnext(arr, seq[1], i)
if occurrence == 0
break
else
push!(indices, occurrence)
i = occurrence +1
end
end
else
while true
occurrence = Base._searchindex(arr, seq, i)
if occurrence == 0
break
else
push!(indices, occurrence)
i = occurrence +1
end
end
end
return indices
end
julia> #time findsequence(rand(1:9, 1000), [2,3])
0.000036 seconds (29 allocations: 8.766 KB)
16-element Array{Int64,1}:
80
118
138
158
234
243
409
470
539
589
619
629
645
666
762
856
Here is a more up-to-date implementation using findall
function issubsequence(A, B)
B1inA = findall(isequal(B[1]), A) # indices of the first element of b occuring in a
matchFirstIndex = [] # Saves the first index in A of the occurances
for i in B1inA
if length(A[i:end]) < length(B) continue end
if A[i:i + length(B) - 1] == B
push!(matchFirstIndex, i)
end
end
return matchFirstIndex
end
I get a similar runtime to #daycaster
#time issubsequence(rand(1:9, 1000), [2,3])
0.000038 seconds (111 allocations: 20.844 KiB)
7-element Vector{Any}:
57
427
616
644
771
855
982
There is no standard Julia library function to determine if a particular sequence occurs as a subsequence of another. Probably this is because this is actually a fairly trickly problem (known as the String-searching algorithm) and quite how to do it depends on whether you'll be searching repeatedly, whether you want to do multiple matches, have multiple patterns, want fuzzy matches, etc.
The other answers here give reasonable answers but some are old and Julia has improved, and I wanted to offer a slightly more idiomatic solution.
function issubarray(needle, haystack)
getView(vec, i, len) = view(vec, i:i+len-1)
ithview(i) = getView(haystack, i, length(needle))
return any(i -> ithview(i) == needle, 1:length(haystack)-length(needle)+1)
end
This is lightening fast and requires almost no memory - Julia's view is lightweight and efficient. And, as always with Julia, the solution is generally to simply define more functions.
Generally having 1-based array for Julia is a good decision, but sometimes it is desirable to have Fortran-like array with indices that span some subranges of ℤ:
julia> x = FArray(Float64, -1:1,-7:7,-128:512)
where it would be useful:
in the codes accompanying the book Numerical Solution of Hyperbolic Partial Differential Equations by prof. John A. Trangenstein these negative indices are used intensively for ghost cells for boundary conditions.
The same is true for Clawpack (stands for “Conservation Laws Package”) by prof. Randall J. LeVeque http://depts.washington.edu/clawpack/ and there are many other codes where such indices would be natural.
So such auxiliary class would be useful for speedy translation of such codes.
I just started to implement such an auxiliary type but as I'm quite new to Julia your help would be greatly appreciated.
I started with:
type FArray
ranges
array::Array
function FArray(T, r::Range1{Int}...)
dims = map((x) -> length(x), r)
array = Array(T, dims)
new(r, array)
end
end
Output:
julia> include ("FortranArray.jl")
julia> x = FArray(Float64, -1:1,-7:7,-128:512)
FArray((-1:1,-7:7,-128:512),3x15x641 Array{Float64,3}:
[:, :, 1] =
6.90321e-310 2.6821e-316 1.96042e-316 0.0 0.0 0.0 9.84474e-317 … 1.83233e-316 2.63285e-316 0.0 9.61618e-317 0.0
6.90321e-310 6.32404e-322 2.63285e-316 0.0 0.0 0.0 2.63292e-316 2.67975e-316
...
[:, :, 2] =
...
As I'm completely new to Julia any recommendations especially that lead to more efficient would be greatly appreciated.
[Edit]
The topic has been discussed here:
https://groups.google.com/forum/#!topic/julia-dev/NOF6MA6tb9Y
During the discussion two ways to have Julia arrays with arbitrary base were elaborated:
SubArray-based, sample usage is with a helper function:
function farray(T, r::Range1{Int64}...)
dims = map((x) -> length(x), r)
array = Array(T, dims)
sub_indices = map((x) -> -minimum(x) + 2 : maximum(x), r)
sub(array, sub_indices)
end
julia> y[-1,-7,-128] = 777
777
julia> y[-1,-7,-128] + 33
810.0
julia> y[-2,-7,-128]
ERROR: BoundsError()
in getindex at subarray.jl:200
julia> y[2,-7,-128]
2.3977385e-316
Please note, that bounds are not checked fully more details are here:
https://github.com/JuliaLang/julia/issues/4044
At the moment SubArray has performance issues but eventually its performance might be improved, see also:
https://github.com/JuliaLang/julia/issues/5117
https://github.com/JuliaLang/julia/issues/3496
Another approach that has better performance at the moment, besides checks both bounds:
type FArray{T<:Number, N, A<:AbstractArray} <: AbstractArray
ranges
offsets::NTuple{N,Int}
array::A
function FArray(r::Range1{Int}...)
dims = map((x) -> length(x), r)
array = Array(T, dims)
offs = map((x) -> 1 - minimum(x), r)
new(r, offs, array)
end
end
FArray(T, r::Range1{Int}...) = FArray{T, length(r,), Array{T, length(r,)}}(r...)
getindex{T<:Number}(FA::FArray{T}, i1::Int) = FA.array[i1+FA.offsets[1]]
getindex{T<:Number}(FA::FArray{T}, i1::Int, i2::Int) = FA.array[i1+FA.offsets[1], i2+FA.offsets[2]]
getindex{T<:Number}(FA::FArray{T}, i1::Int, i2::Int, i3::Int) = FA.array[i1+FA.offsets[1], i2+FA.offsets[2], i3+FA.offsets[3]]
setindex!{T}(FA::FArray{T}, x, i1::Int) = arrayset(FA.array, convert(T,x), i1+FA.offsets[1])
setindex!{T}(FA::FArray{T}, x, i1::Int, i2::Int) = arrayset(FA.array, convert(T,x), i1+FA.offsets[1], i2+FA.offsets[2])
setindex!{T}(FA::FArray{T}, x, i1::Int, i2::Int, i3::Int) = arrayset(FA.array, convert(T,x), i1+FA.offsets[1], i2+FA.offsets[2], i3+FA.offsets[3])
getindex and setindex! methods for FArray were inspired by base/array.jl code.
Use cases:
julia> y = FArray(Float64, -1:1,-7:7,-128:512);
julia> y[-1,-7,-128] = 777
777
julia> y[-1,-7,-128] + 33
810.0
julia> y[-1,2,3]
0.0
julia> y[-2,-7,-128]
ERROR: BoundsError()
in getindex at FortranArray.jl:27
julia> y[2,-7,-128]
ERROR: BoundsError()
in getindex at FortranArray.jl:27
There are now two packages that provide this kind of functionality. For arrays with arbitrary start indices, see https://github.com/alsam/OffsetArrays.jl. For even more flexibility see https://github.com/mbauman/AxisArrays.jl, where indices do not have to be integers.