What is the most efficient way to create a view on array using, for example, sliding window=2
Let's say we have:
x = collect(1:1:6)
# 1 2 3 4 5 6
And I want to create a view like this:
# 1 2
# 2 3
# 3 4
# 4 5
# 5 6
So far I found only this option, but not sure if it's an optimal one:
y = Array{Float32, 2}(undef, nslides, window)
#inbounds for i in 1:window
y[:, i] = #view x[i:end-(window-i)]
end
One solution with a package (well, with my package) is this:
julia> using Tullio
julia> x = 1:6; window = 2;
julia> #tullio y[r,c] := x[r+c-1] (c in 1:window)
5×2 Matrix{Int64}:
1 2
2 3
3 4
4 5
5 6
The one liner is:
view.(Ref(x), (:).(1:length(x)-1,2:length(x)))
Testing:
julia> x=collect(1:6);
julia> view.(Ref(x), (:).(1:length(x)-1,2:length(x)))
5-element Array{SubArray{Int64,1,Array{Int64,1},Tuple{UnitRange{Int64}},true},1}:
[1, 2]
[2, 3]
[3, 4]
[4, 5]
[5, 6]
Explanation:
creation of views is vectorized by the dot operator .
we do not want to vectorize on elements of x so use Ref(x) instead
(:) is just a shorter form for UnitRange and again we use the dot operator . to vectorize
I used 2 as the Window size but of course you can write view.(Ref(x), (:).(1:length(x)-(window-1),window:length(x)))
EDIT:
If you want rather a library function this would work for you:
julia> using ImageFiltering
julia> mapwindow(collect, x, 0:1,border=Inner())
5-element OffsetArray(::Array{Array{Int64,1},1}, 1:5) with eltype Array{Int64,1} with indices 1:5:
[1, 2]
[2, 3]
[3, 4]
[4, 5]
[5, 6]
Of course you could put them the function that you want to run on the sliding window rather than just collect.
Related
I am trying to access elements of the previous columns than first column.
For example in python if i try to access elements before the first column 0 it will return an empty array.
import numpy as np
x = np.matrix([[ 1, 2, 10],
[ 1, 4, 2],
[ 2, 3, 2],
[ 2, 3, 6]])
a = x[:, 0]
print('\n a is \n', a)
b = x[:, :0]
print('\n b is \n',b)
Output: -->
a is
[[1]
[1]
[2]
[2]]
b is
[]
But julia does not behave in such manner.
My trial in julia:
array = [1 2 3;3 4 5;2 3 4;1 2 3]
a = array[:, 1]
b = array[:, 1-1]
Output :-->
BoundsError: attempt to access 4×3 Array{Int64,2} at index [1:4, 0]
Stacktrace:
[1] throw_boundserror(::Array{Int64,2}, ::Tuple{Base.Slice{Base.OneTo{Int64}},Int64}) at .\abstractarray.jl:541
[2] checkbounds at .\abstractarray.jl:506 [inlined]
[3] _getindex at .\multidimensional.jl:742 [inlined]
[4] getindex(::Array{Int64,2}, ::Function, ::Int64) at .\abstractarray.jl:1060
[5] top-level scope at In[1166]:1
[6] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1091
However, to replicate such operation I tried to implement try and catch method and use it over the loop to see what happens when i try accessing different columns. The output is always an empty array.
for i in 1:3
try
b = array[:, :i-1]
catch
b = []
end
println("\b b in $i is $b")
end
Output :--->
b in 1 is Any[]
b in 2 is Any[]
b in 3 is Any[]
May I know how can I access row elements from first_column - 1 to last_column - 1, where first_column - 1 returns an empty array but rest returns data from the matrix.
The answer to my question is provided by #ettersi on julia discourse, I am posting it here for reference. Please follow the link mentioned in the comment to access the discourse page.
ANSWER!
julia> array = [1 2 3;3 4 5;2 3 4;1 2 3]
4×3 Matrix{Int64}:
1 2 3
3 4 5
2 3 4
1 2 3
julia> array[:, 1:0]
4×0 Matrix{Int64}
I have an array of arrays A which is an N-element Array{Array{Int64,1},1} of integers. I am trying to find the largest array in A using Julia.
For example:
A = [[1, 2], [3, 4], [5, 6, 7], [1, 2, 5, 8]]
In Python I would simply do: max(A, key=len) but in Julia I don't know how to do it.
What I did is this:
L = []
for a in A
push!(L, length(a))
end
A[findmax(L)[2]]
Thanks!
#Colin has provided a compact, convenient answer. However, if speed matters (op asked for most efficient way) this should be close to optimum
function findlongest(A)
idx = 0
len = 0
#inbounds for i in 1:length(A)
l = length(A[i])
l > len && (idx = i; len=l)
end
return A[idx]
end
Note that this implementation would (presumably) be a really bad idea in Python :)
Quick benchmark:
julia> using BenchmarkTools
julia> A = [[1,2], [1,2,3,4,5,6], [1,2,3]]
3-element Array{Array{Int64,1},1}:
[1, 2]
[1, 2, 3, 4, 5, 6]
[1, 2, 3]
julia> #btime findlongest(A);
26.880 ns (0 allocations: 0 bytes)
julia> #btime A[indmax(length.(A))];
9.813 μs (25 allocations: 1.14 KiB)
That's a ~365 times speedup for this example.
EDIT: Better benchmark (suggested in the comments)
julia> #btime findlongest($A);
9.813 ns (0 allocations: 0 bytes)
julia> #btime $A[indmax(length.($A))];
41.813 ns (1 allocation: 112 bytes)
The $ signs avoid setup allocations and times. Speedup ~4.
Quick explanation
for loops are fast in Julia, so why not use them
avoid allocation (length.(A) allocates a new array of integers)
a && b is shortcut for "if a then b"
#inbounds avoids bound checks for A[i]
UPDATE: For v1+ you'll need to replace indmax in this answer with argmax.
EDIT: Note, it is also worth checking out the other answer by #crstnbr
Consider the following example code:
julia> A = [[1,2], [1,2,3,4,5,6], [1,2,3]]
3-element Array{Array{Int64,1},1}:
[1, 2]
[1, 2, 3, 4, 5, 6]
[1, 2, 3]
julia> length(A)
3
julia> length.(A)
3-element Array{Int64,1}:
2
6
3
julia> indmax(length.(A))
2
julia> A[indmax(length.(A))]
6-element Array{Int64,1}:
1
2
3
4
5
6
The first call to length gets the length of the outer vector in A, which is not what we want. In the second call, I use the broadcasting operator . so that I instead get the length of each of the inner vectors. In the indmax line, I'm finding the index of largest value in length.(A), ie the index of the longest inner vector. If you instead want to return the longest inner vector, you can just index into A using the result of the indmax line.
indmax is no longer defined in Julia (at least 1.3).
Use argmax instead.
>>> A = [[1,2], [1,2,3]]
2-element Array{Array{Int64,1},1}:
[1, 2]
[1, 2, 3]
>>> length.(A)
2-element Array{Int64,1}:
2
3
>>> argmax(length.(A))
2
>>> A[argmax(length.(A))]
3-element Array{Int64,1}:
1
2
3
In Julia vec reshapes multidimensional arrays into one-dimension arrays.
However it doesn't work for arrays of arrays or arrays of tuples.
A part from using array comprehension, is there another way to flatten arrays of arrays/tuples? Or arrays of arrays/tuples of arrays/tuples? Or ...
Iterators.flatten(x) creates a generator that iterates over each element of x. It can handle some of the cases you describe, eg
julia> collect(Iterators.flatten([(1,2,3),[4,5],6]))
6-element Array{Any,1}:
1
2
3
4
5
6
If you have arrays of arrays of arrays and tuples, you should probably reconsider your data structure because it doesn't sound type stable. However, you can use multiple calls to flatten, eg
julia> collect(Iterators.flatten([(1,2,[3,3,3,3]),[4,5],6]))
6-element Array{Any,1}:
1
2
[3, 3, 3, 3]
4
5
6
julia> collect(Iterators.flatten(Iterators.flatten([(1,2,[3,3,3,3]),[4,5],6])))
9-element Array{Any,1}:
1
2
3
3
3
3
4
5
6
Note how all of my example return an Array{Any,1}. That is a bad sign for performance, because it means the compiler could not determine a single concrete type for the elements of the output array. I chose these example because the way I read your question it sounded like you may have type unstable containers already.
In order to flatten an array of arrays, you can simply use vcat() like this:
julia> A = [[1,2,3],[4,5], [6,7]]
Vector{Int64}[3]
Int64[3]
Int64[2]
Int64[2]
julia> flat = vcat(A...)
Int64[7]
1
2
3
4
5
6
7
The simplest way is to apply the ellipsis ... twice.
A = [[1,2,3],[4,5], [6,7]]
flat = [(A...)...]
println(flat)
The output would be
[1, 2, 3, 4, 5, 6, 7].
If you use VectorOfArray from RecursiveArrayTools.jl, it uses the indexing fallback to provide convert(Array,A) for a VectorOfArray A.
julia> using RecursiveArrayTools
julia> A = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
3-element Array{Array{Int64,1},1}:
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
julia> VA = VectorOfArray(A)
3-element Array{Array{Int64,1},1}:
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
First of it acts as a lazy wrapper for doing the indexing without conversion:
julia> VA[1,3]
7
Note that columns are the separate arrays so that way it's still "column-major" (i.e. efficient to index down columns). But then it has a straight conversion:
julia> convert(Array,VA)
3×3 Array{Int64,2}:
1 4 7
2 5 8
3 6 9
The other way to handle this conversion is to do something like hcat(A...), but that's slow if you have a lot of arrays you're splatting!
Now, you may think: what about writing a function that pre-allocates the matrix, then loops through and fills it? That's almost what convert on the VectorOfArray works, except the fallback that convert uses here utilizes Tim Holy's Cartesian machinery. At one point, I wrote that function:
function vecvec_to_mat(vecvec)
mat = Matrix{eltype(eltype(vecvec))}(length(vecvec),length(vecvec[1]))
for i in 1:length(vecvec)
mat[i,:] .= vecvec[i]
end
mat
end
but I have since gotten rid of it because the fallback was much faster. So, YMMV but that's a few ways to solve your problem.
for Julia 0.7x:
for Arrays:
flat(arr::Array) = mapreduce(x -> isa(x, Array) ? flat(x) : x,
append!, arr,init=[])
for Tuples:
flat(arr::Tuple) = mapreduce(x -> isa(x, Tuple) ? flat(x) : x,
append!, arr,init=[])
Works for arbitrary depth.
see: https://rosettacode.org/wiki/Flatten_a_list#Julia
Code for Array/Tuple:
function flatten(arr)
rst = Any[]
grep(v) = for x in v
if isa(x, Tuple) || isa(x, Array)
grep(x)
else push!(rst, x) end
end
grep(arr)
rst
end
For example, I can create an array that contains a function.
julia> a(x) = x + 1
>> a (generic function with 1 method)
julia> [a]
>> 1-element Array{#a,1}:
a
But I can't seem to add the function to an empty array:
julia> append!([],a)
>> ERROR: MethodError: no method matching length(::#a)
Closest candidates are:
length(::SimpleVector) at essentials.jl:168
length(::Base.MethodList) at reflection.jl:256
length(::MethodTable) at reflection.jl:322
...
in _append!(::Array{Any,1}, ::Base.HasLength, ::Function) at .\collections.jl:25
in append!(::Array{Any,1}, ::Function) at .\collections.jl:21
What I ulimately want to do is store the pre-defined functions so that I can ultimately map them over a value. E.g.:
x = 0.0
for each fn in vec
x = x + fn(x)
end
append! is for appending one collection on to another.
You are looking for push!, to add an element to a collection.
Your code should be push!([], a)
See the docs:
julia>?append!
search: append!
append!(collection, collection2) -> collection.
Add the elements of collection2 to the end of collection.
julia> append!([1],[2,3])
3-element Array{Int64,1}:
1
2
3
julia> append!([1, 2, 3], [4, 5, 6])
6-element Array{Int64,1}:
1
2
3
4
5
6
Use push! to add individual items to collection which are not already themselves in another collection. The result is of the preceding example is equivalent to push!([1, 2, 3], 4, 5, 6).
vs:
julia>?push!
search: push! pushdisplay
push!(collection, items...) -> collection
Insert one or more items at the end of collection.
julia> push!([1, 2, 3], 4, 5, 6)
6-element Array{Int64,1}:
1
2
3
4
5
6
Use append! to add all the elements of another collection to collection. The result of the preceding example is equivalent to append!([1, 2, 3], [4, 5, 6]).
I need to copy a part of a 3D array.
I have the indexes of start and end of the copy.
For example 2D array:
[[2 2 3 4 5]
[2 3 3 4 5]
[2 3 4 4 5]
[2 3 4 5 5]
[2 3 4 5 6]]
starting index, end index are:
mini = [2, 1]
maxi = [4, 3]
So the result should be:
[[3 4 4]
[3 4 5]]
I can write:
result = matrix[mini[0]:maxi[0], mini[1]:maxi[1]]
Is there a way to do it generally ? for 3Dim or NDim arrays ?
The trick here is realizing what the indexing syntax is under the hood. This:
result = matrix[mini[0]:maxi[0], mini[1]:maxi[1]]
Is shorthand in python (not just numpy) for:
indices = slice(mini[0], maxi[0]), slice(mini[1], maxi[1])
result = matrix[indices]
So we just need to generate indices dynamically:
lower = [2, 1, ...]
upper = [4, 3, ...]
indices = tuple(np.s_[l:u] for l, u in zip(lower, upper))
result = matrix_nd[indices]
np.s_[a:b] is a shorthand for slice(a, b). Here we build a tuple containing as many slices as you have values in lower and upper
What you are looking for is the slice object, see that example:
matrix = np.random.rand(4,5)
mini = [2, 1]
maxi = [4, 3]
slices=[slice(b,e) for b, e in zip(mini,maxi)]
print(slices)
print(matrix[slices])
print(matrix[mini[0]:maxi[0], mini[1]:maxi[1]])