Array range complement - arrays

Is there a way to overwrite [] to have complement of range in array?
julia> a=[1:8...]
8-element Array{Int64,1}:
1
2
3
4
5
6
7
8
julia> a[-1] == a[2:8]
julia> a[-(1:3)] == a[4:8]
julia> a[-end] == a[1:7]

I haven't looked into the internals of indexing before, but at a first glance, the following might work without breaking too much:
immutable Not{T}
idx::T
end
if :to_indices in names(Base)
# 0.6
import Base: to_indices, uncolon, tail, _maybetail
#inline to_indices(A, inds, I::Tuple{Not, Vararg{Any}}) =
(setdiff(uncolon(inds, (:, tail(I)...)), I[1].idx), to_indices(A, _maybetail(inds), tail(I))...)
else
# 0.5
import Base: getindex, _getindex
not_index(a::AbstractArray, I, i::Int) = I
not_index(a::AbstractArray, I::Not, i::Int) = setdiff(indices(a, i), I.idx)
getindex(a::AbstractArray, I::Not) = getindex(a, setdiff(linearindices(a), I.idx))
_getindex(::Base.LinearIndexing, a::AbstractArray, I::Vararg{Union{Real, AbstractArray, Colon, Not}}) =
Base._getindex(Base.linearindexing(a), a, (not_index(a, idx, i) for (i,idx) in enumerate(I))...)
end
For example:
julia> a = reshape(1:9, (3, 3))
3×3 Base.ReshapedArray{Int64,2,UnitRange{Int64},Tuple{}}:
1 4 7
2 5 8
3 6 9
julia> a[Not(2:8)]
2-element Array{Int64,1}:
1
9
julia> a[Not(1:2), :]
1×3 Array{Int64,2}:
3 6 9
julia> a[Not(end), end]
2-element Array{Int64,1}:
7
8
I didn't care for performance and also did no extensive testing, so things can certainly be improved.
Edit:
I replaced the code for 0.6 with Matt B. version from his github comment linked in the comments.
Thanks to his great design of the array indexing implementation for 0.6, only a single function needs to be extended to get complement indexing for getindex, setindex and view, e.g.,
julia> view(a, Not(2:8))
2-element SubArray{Int64,1,UnitRange{Int64},Tuple{Array{Int64,1}},false}:
1
9
# collect because ranges are immutable
julia> b = collect(a); b[Not(2), Not(2)] = 10; b
3×3 Array{Int64,2}:
10 4 10
2 5 8
10 6 10

Directly overwriting [](i.e. getindex) is prone to break many indexing-related things in Base, but we can write an array wrapper to work around it. We only need to define the following three methods to get your specific test cases passed:
immutable ComplementVector{T} <: AbstractArray{T,1}
data::Vector{T}
end
Base.size(A:: ComplementVector) = size(A.data)
Base.getindex(A:: ComplementVector, i::Integer) = i > 0 ? A.data[i] : A.data[setdiff(1:end, (-i))]
Base.getindex(A:: ComplementVector, I::StepRange) = all(x->x>0, I) ? A.data[I] : A.data[setdiff(1:end, -I)]
julia> a = ComplementVector([1:8...])
julia> a[-1] == a[2:8]
true
julia> a[-(1:3)] == a[4:8]
true
julia> a[-end] == a[1:7]
true
If you would like to extend ComplementVector further more, please read the doc about Interfaces.
Update:
For safety sake, we'd better not extend AbstractArray as #Fengyang Wang suggested in the comment blow:
immutable ComplementVector{T}
data::Vector{T}
end
Base.endof(A::ComplementVector) = length(A.data)
Base.getindex(A::ComplementVector, i::Integer) = i > 0 ? A.data[i] : A.data[setdiff(1:end, (-i))]
Base.getindex(A::ComplementVector, I::OrdinalRange) = all(x->x>0, I) ? A.data[I] : A.data[setdiff(1:end, -I)]

Related

Julia strange answer from indexing an array

I'm attempting to index an array based on conditions in the array itself and corresponding locations in other arrays. I've produced a MWE but this will be hopefully used in a much larger example, within a loop to automate across scenarios, dare I say... using parallelisation?
MWE:
# create 3d array
a, b, c = [8;8;6;5;5;6], [8;8;7;6;6;6], [8;2;7;7;6;6]
d = transpose(cat(a,b,c, dims = 2))
e, f, g = [3;2;5;1;4;1], [4;3;1;1;1;2], [5;1;2;1;2;3]
h = transpose(cat(e,f,g, dims = 2))
wrkarr = cat(d,h,dims = 3)
# create temp array for indexes
temp = wrkarr[size(wrkarr,1), :, 1]
# calculate indexes
temp[(wrkarr[size(wrkarr,1),:,1] .== 8) .& (wrkarr[size(wrkarr,1),:,2]) .>= 4] .= 22
In my case it doesn't change anything in the temp array, when I would expect the first element to be changed from 8 to 22. Both of the individual conditional tests produce a vector [1,0,0,0,0,0] so why won't the .& test produce the same? Thx. J
First, and as an aside, please note that you can use arr[end,i] to refer to the element arr with the last index in the 1st axis, and index i in the second axis.
Using this notation, your condition can be rewritten as:
julia> (wrkarr[end,:,1] .== 8) .& (wrkarr[end,:,2]) .>=4
6-element BitArray{1}:
0
0
0
0
0
0
It might be a bit easier to see here that there is a parenthesis issue. I think you actually wanted to write:
julia> (wrkarr[end,:,1] .== 8) .& (wrkarr[end,:,2] .>=4)
6-element BitArray{1}:
1
0
0
0
0
0
Making this change does what (I think) you want. (Also note that I added #views below in order to avoid allocations and speed things up a little)
julia> idx = (#view(wrkarr[end,:,1]) .== 8) .& (#view(wrkarr[end,:,2]) .>=4);
julia> temp[idx] .= 22
1-element view(::Array{Int64,1}, [1]) with eltype Int64:
22
julia> temp
6-element Array{Int64,1}:
22
2
7
7
6
6
EDIT: as mentioned in comments, other solutions could be considered:
using findall to generate a vector of indices matching the condition
using a simple for loop
Here is a benchmark of all three solutions, in order to see how they compare in terms of readability and performance.
TLDR: the for loop seems to be much more efficient in this case, and allocates less.
# create 3d array
d = [8 8 6 5 5 6;
8 8 7 6 6 6;
8 2 7 7 6 6]
h = [3 2 5 1 4 1;
4 3 1 1 1 2;
5 1 2 1 2 3]
wrkarr = cat(d,h,dims = 3)
using BenchmarkTools
Option 1 logical indexing
julia> function version1!(temp, wrkarr)
idx = (#view(wrkarr[end,:,1]) .== 8) .& (#view(wrkarr[end,:,2]) .>= 4)
temp[idx] .= 22
end
version1! (generic function with 1 method)
julia> temp = wrkarr[end, :, 1]; #btime version1!($temp, $wrkarr); temp
182.646 ns (3 allocations: 224 bytes)
6-element Array{Int64,1}:
22
2
7
7
6
6
Option 2 vector of indices
julia> function version2!(temp, wrkarr)
idx = findall(i -> (wrkarr[end,i,1] == 8) & (wrkarr[end,i,2] >=4), axes(wrkarr,2))
temp[idx] .= 22
end
version2! (generic function with 1 method)
julia> temp = wrkarr[end, :, 1]; #btime version2!($temp, $wrkarr); temp
134.395 ns (3 allocations: 208 bytes)
6-element Array{Int64,1}:
22
2
7
7
6
6
Option 3 for loop
julia> function version3!(temp, wrkarr)
#inbounds for i in axes(wrkarr, 2)
if wrkarr[end, i, 1] == 8 && wrkarr[end, i, 2] >= 4
temp[i] = 22
end
end
end
version3! (generic function with 1 method)
julia> temp = wrkarr[end, :, 1]; #btime version3!($temp, $wrkarr); temp
21.820 ns (0 allocations: 0 bytes)
6-element Array{Int64,1}:
22
2
7
7
6
6
Based on all the clever stuff given to me above I pushed it a bit further and this solution appears to work and dispenses with the need for an index array, simply does the substitution right back into the original array. Is there anything wrong with that? Looking at #François Févotte's answer the loop function is the fastest, and this could be very important when scaled up to proper size. Now for my next challenge I want to use the same function to loop through sets of numbers to replace the 8, 4, 22, for example for the same wrkarr:
== 8 &>= 4 -> 22
== 7 &>= 2 -> 8
== 6 &>= 3 -> 7
Any takers or suggestions? When I get that to work I want to see if it can be parallelized across these individual sets of numbers, i.e. each substitution can be done independently, not dependent on any other. Thx a bunch!
function version4(wrkarr)
#inbounds for i in axes(wrkarr, 2)
if wrkarr[end, i, 1] == 8 && wrkarr[end, i, 2] >= 4
wrkarr[end,i, 1] = 22
end
end
end
wrkarr[end, :, 1]; #btime version4($wrkarr); wrkarr
2.237 ns (0 allocations: 0 bytes)
3×6×2 Array{Int64,3}:
[:, :, 1] =
8 8 6 5 5 6
8 8 7 6 6 6
22 2 7 7 6 6
[:, :, 2] =
3 2 5 1 4 1
4 3 1 1 1 2
5 1 2 1 2 3

Vectorized multiplication: Multiply two vectors in Julia, element-wise

I decided to dive into Julia and hitting the wall; fast.
I am trying to replicate a simple operation, which would like as follows in python numpy
a = numpy.array([1,2,3])
b = numpy.array([1,2,3])
a*b
[output]: [1,4,9]
In other words "[1,4,9]" is the output I expect.
I tried in Julia the following:
a = [1,2,3]
b = [1,2,3]
a*b
[output]: MethodError: no method matching *(::Array{Int64,1}, ::Array{Int64,1})
or after trying to wise up:
a = [1,2,3]
b = [1,2,3]'
a*b
[output]: 3×3 Array{Int64,2}:
1 2 3
2 4 6
3 6 9
I am aware that this seems like a basic question, but my Googling does not seem my finest today and/or stackoverflow could use this question & answer ;)
Thanks for any help and pointers!
Best
Julia needs a . in front of the operator or function call to indicate you want elementwise multiplication and not an operation on the vector as a unit. This is called broadcasting the array:
julia> a = [1,2,3]
3-element Array{Int64,1}:
1
2
3
julia> b = [1,2,3]
3-element Array{Int64,1}:
1
2
3
julia> a .* b
3-element Array{Int64,1}:
1
4
9
I just found a solution, although surely not optimal as it will generate a dot product and then select the diagonals.... to much calc!\
use LinearAlgebra
a = [1,2,3]
b = [1,2,3]
c = a * b'
diag(c)
I am pretty sure that there is a better solution.

Converting Array of CartesianIndex to 2D-Matrix in Julia

let's say we have an array of cartesian indices in Julia
julia> typeof(indx)
Array{CartesianIndex{2},1}
Now we want to plot them as a scatter-plot using PyPlot. so we should convert the indx-Array of Cartesian to a 2D-Matrix so we can plot it like this:
PyPlot.scatter(indx[:, 1], indx[:, 2])
How can i convert an Array of type Array{CartesianIndex{2},1} to a 2D-Matrix of type Array{Int,2}
By the way here is a code snippet how to produce a dummy Array of cartesianindex:
A = rand(1:10, 5, 5)
indx = findall(a -> a .> 5, A)
typeof(indx) # this is an Array{CartesianIndex{2},1}
Thanks
An easy and generic way is
julia> as_ints(a::AbstractArray{CartesianIndex{L}}) where L = reshape(reinterpret(Int, a), (L, size(a)...))
as_ints (generic function with 1 method)
julia> as_ints(indx)
2×9 reshape(reinterpret(Int64, ::Array{CartesianIndex{2},1}), 2, 9) with eltype Int64:
1 3 4 1 2 4 1 1 4
2 2 2 3 3 3 4 5 5
This works for any dimensionality, making the first dimension the index into the CartesianIndex.
One possible way is hcat(getindex.(indx, 1), getindex.(indx,2))
julia> #btime hcat(getindex.($indx, 1), getindex.($indx,2))
167.372 ns (6 allocations: 656 bytes)
10×2 Array{Int64,2}:
4 1
3 2
4 2
1 3
4 3
5 3
2 4
5 4
1 5
4 5
However, note that you don't need to - and therefore probably shouldn't - bring your indices to 2D-Matrix form. You could simply do
PyPlot.scatter(getindex.(indx, 1), getindex.(indx, 2))

MATLAB-style replacement of array values that meet certain condition in Julia [duplicate]

In Octave, I can do
octave:1> A = [1 2; 3 4]
A =
1 2
3 4
octave:2> A(A>1) -= 1
A =
1 1
2 3
but in Julia, the equivalent syntax does not work.
julia> A = [1 2; 3 4]
2x2 Array{Int64,2}:
1 2
3 4
julia> A[A>1] -= 1
ERROR: `isless` has no method matching isless(::Int64, ::Array{Int64,2})
in > at operators.jl:33
How do you conditionally assign values to certain array or matrix elements in Julia?
Your problem isn't with the assignment, per se, it's that A > 1 itself doesn't work. You can use the elementwise A .> 1 instead:
julia> A = [1 2; 3 4];
julia> A .> 1
2×2 BitArray{2}:
false true
true true
julia> A[A .> 1] .-= 1000;
julia> A
2×2 Array{Int64,2}:
1 -998
-997 -996
Update:
Note that in modern Julia (>= 0.7), we need to use . to say that we want to broadcast the action (here, subtracting by the scalar 1000) to match the size of the filtered target on the left. (At the time this question was originally asked, we needed the dot in A .> 1 but not in .-=.)
In Julia v1.0 you can use the replace! function instead of logical indexing, with considerable speedups:
julia> B = rand(0:20, 8, 2);
julia> #btime (A[A .> 10] .= 10) setup=(A=copy($B))
595.784 ns (11 allocations: 4.61 KiB)
julia> #btime replace!(x -> x>10 ? 10 : x, A) setup=(A=copy($B))
13.530 ns ns (0 allocations: 0 bytes)
For larger matrices, the difference hovers around 10x speedup.
The reason for the speedup is that the logical indexing solution relies on creating an intermediate array, while replace! avoids this.
A slightly terser way of writing it is
replace!(x -> min(x, 10), A)
There doesn't seem to be any speedup using min, though.
And here's another solution that is almost as fast:
A .= min.(A, 10)
and that also avoids allocations.
To make it work in Julia 1.0 one need to change = to .=. In other words:
julia> a = [1 2 3 4]
julia> a[a .> 1] .= 1
julia> a
1×4 Array{Int64,2}:
1 1 1 1
Otherwise you will get something like
ERROR: MethodError: no method matching setindex_shape_check(::Int64, ::Int64)

Julia's Negative/Complement Indexing like R

Is there a functionality in Julia that's similar to R's negative indexing? In R, the code would be similar to:
x = 1:10
inds = c(1, 5, 7)
x[-inds]
[1] 2 3 4 6 8 9 10
I've found this to be extremely useful in numerous situations, especially for things such as sampling indices to create a testing/training set, but also for subindexing an array to exclude certain rows. So I am hoping there's something simple in Julia that can do the same.
This is similar to #Colin T Bower's answer and also only uses base Julia. Afraid it is not as elegant as your R example.
julia> minus(indx, x) = setdiff(1:length(x), indx)
minus (generic function with 1 method)
julia> x = collect(1:10)
10-element Array{Int64,1}:
1
2
3
4
5
6
7
8
9
10
julia> inds = [1, 5, 7]
3-element Array{Int64,1}:
1
5
7
julia> x[minus(inds, x)]
7-element Array{Int64,1}:
2
3
4
6
8
9
10
Not a feature of the base language, but see for example the package here: https://github.com/mbauman/InvertedIndices.jl

Resources