Equivalent np.nanquantile in Julia? - arrays

Is there a np.nanquantile equivalent in Julia? I have a 2D array and calculate a quantile along one axis but the array contains NaN-values. My current code block:
quantiles = Array{Float32}(undef, size(array, 1), 2)
p=0.1
quantiles[:, 1] = mapslices(x -> quantile(x, p), array, dims = 2)
quantiles[:, 2] = mapslices(x -> quantile(x, 1 - p), array, dims = 2)

The simplest thing to do is to use the following:
x -> quantile(filter(!isnan, x), p)
e.g.
julia> array = [1 NaN 3 4
NaN 2 3 4]
2×4 Matrix{Float64}:
1.0 NaN 3.0 4.0
NaN 2.0 3.0 4.0
julia> mapslices(x -> quantile(filter(!isnan, x), 0.5), array, dims = 2)
2×1 Matrix{Float64}:
3.0
3.0

Related

MATLAB-style replacement of array values that meet certain condition in Julia [duplicate]

In Octave, I can do
octave:1> A = [1 2; 3 4]
A =
1 2
3 4
octave:2> A(A>1) -= 1
A =
1 1
2 3
but in Julia, the equivalent syntax does not work.
julia> A = [1 2; 3 4]
2x2 Array{Int64,2}:
1 2
3 4
julia> A[A>1] -= 1
ERROR: `isless` has no method matching isless(::Int64, ::Array{Int64,2})
in > at operators.jl:33
How do you conditionally assign values to certain array or matrix elements in Julia?
Your problem isn't with the assignment, per se, it's that A > 1 itself doesn't work. You can use the elementwise A .> 1 instead:
julia> A = [1 2; 3 4];
julia> A .> 1
2×2 BitArray{2}:
false true
true true
julia> A[A .> 1] .-= 1000;
julia> A
2×2 Array{Int64,2}:
1 -998
-997 -996
Update:
Note that in modern Julia (>= 0.7), we need to use . to say that we want to broadcast the action (here, subtracting by the scalar 1000) to match the size of the filtered target on the left. (At the time this question was originally asked, we needed the dot in A .> 1 but not in .-=.)
In Julia v1.0 you can use the replace! function instead of logical indexing, with considerable speedups:
julia> B = rand(0:20, 8, 2);
julia> #btime (A[A .> 10] .= 10) setup=(A=copy($B))
595.784 ns (11 allocations: 4.61 KiB)
julia> #btime replace!(x -> x>10 ? 10 : x, A) setup=(A=copy($B))
13.530 ns ns (0 allocations: 0 bytes)
For larger matrices, the difference hovers around 10x speedup.
The reason for the speedup is that the logical indexing solution relies on creating an intermediate array, while replace! avoids this.
A slightly terser way of writing it is
replace!(x -> min(x, 10), A)
There doesn't seem to be any speedup using min, though.
And here's another solution that is almost as fast:
A .= min.(A, 10)
and that also avoids allocations.
To make it work in Julia 1.0 one need to change = to .=. In other words:
julia> a = [1 2 3 4]
julia> a[a .> 1] .= 1
julia> a
1×4 Array{Int64,2}:
1 1 1 1
Otherwise you will get something like
ERROR: MethodError: no method matching setindex_shape_check(::Int64, ::Int64)

Remove a value in an array and decrease the size of it

I have an array filled with some values. After running this code:
array = zeros(10)
for i in 1:10
array[i] = 2*i + 3
end
The array looks like:
10-element Array{Float64,1}:
5.0
7.0
9.0
11.0
13.0
15.0
17.0
19.0
21.0
23.0
I would like to obtain, for example, the following array by removing the third value:
9-element Array{Float64,1}:
5.0
7.0
11.0
13.0
15.0
17.0
19.0
21.0
23.0
How to do that?
EDIT
If I have an array (and not a vector), like here:
a = [1 2 3 4 5]
1×5 Array{Int64,2}:
1 2 3 4 5
The deleteat! proposed is not working:
a = deleteat!([1 2 3 4 5], 1)
ERROR: MethodError: no method matching deleteat!(::Array{Int64,2}, ::Int64)
You might have used a 2d row vector where a 1d column vector was required.
Note the difference between 1d column vector [1,2,3] and 2d row vector [1 2 3].
You can convert to a column vector with the vec() function.
Closest candidates are:
deleteat!(::Array{T,1} where T, ::Integer) at array.jl:875
deleteat!(::Array{T,1} where T, ::Any) at array.jl:913
deleteat!(::BitArray{1}, ::Integer) at bitarray.jl:961
I don't want a column vector. I would want:
1×4 Array{Int64,2}:
2 3 4 5
Is it possible ?
To make that clear: Vector{T} in Julia is just a synonym for Array{T, 1}, unless you're talking about something else... we call Arrays of all ranks arrays.
But this seems to be a Matlab-inherited misconception. In Julia, you construct a Matrix{T}, ie., an Array{T, 2}, by using spaces in the literal:
julia> a = [1 2 3 4 5]
1×5 Array{Int64,2}:
1 2 3 4 5
Deleting from a matrix does not make sense in general, since you can't trivially "fix the shape" in a rectangular layout.
A Vector or Array{T, 1} can be written using commas:
julia> a = [1, 2, 3, 4, 5]
5-element Array{Int64,1}:
1
2
3
4
5
And on this, deleteat! works:
julia> deleteat!(a, 1)
4-element Array{Int64,1}:
2
3
4
5
For completeness, there's also a third variant, the RowVector, which results of a transposition:
julia> a'
1×4 RowVector{Int64,Array{Int64,1}}:
2 3 4 5
From this you also can't delete.
Deleteat! is only defined for:
Fully implemented by:
Vector (a.k.a. 1-dimensional Array)
BitVector (a.k.a. 1-dimensional BitArray)
A Row Vector (2 Dimensions) won't work.
But ... there is a workaround by this trick:
julia> deleteat!(a[1,:], 1)' # mind the ' -> transposes it back to a row vector.
1×4 RowVector{Int64,Array{Int64,1}}:
2 3 4 5
Ofcourse this wouldn't work for an Array with 2 or more rows.

Array range complement

Is there a way to overwrite [] to have complement of range in array?
julia> a=[1:8...]
8-element Array{Int64,1}:
1
2
3
4
5
6
7
8
julia> a[-1] == a[2:8]
julia> a[-(1:3)] == a[4:8]
julia> a[-end] == a[1:7]
I haven't looked into the internals of indexing before, but at a first glance, the following might work without breaking too much:
immutable Not{T}
idx::T
end
if :to_indices in names(Base)
# 0.6
import Base: to_indices, uncolon, tail, _maybetail
#inline to_indices(A, inds, I::Tuple{Not, Vararg{Any}}) =
(setdiff(uncolon(inds, (:, tail(I)...)), I[1].idx), to_indices(A, _maybetail(inds), tail(I))...)
else
# 0.5
import Base: getindex, _getindex
not_index(a::AbstractArray, I, i::Int) = I
not_index(a::AbstractArray, I::Not, i::Int) = setdiff(indices(a, i), I.idx)
getindex(a::AbstractArray, I::Not) = getindex(a, setdiff(linearindices(a), I.idx))
_getindex(::Base.LinearIndexing, a::AbstractArray, I::Vararg{Union{Real, AbstractArray, Colon, Not}}) =
Base._getindex(Base.linearindexing(a), a, (not_index(a, idx, i) for (i,idx) in enumerate(I))...)
end
For example:
julia> a = reshape(1:9, (3, 3))
3×3 Base.ReshapedArray{Int64,2,UnitRange{Int64},Tuple{}}:
1 4 7
2 5 8
3 6 9
julia> a[Not(2:8)]
2-element Array{Int64,1}:
1
9
julia> a[Not(1:2), :]
1×3 Array{Int64,2}:
3 6 9
julia> a[Not(end), end]
2-element Array{Int64,1}:
7
8
I didn't care for performance and also did no extensive testing, so things can certainly be improved.
Edit:
I replaced the code for 0.6 with Matt B. version from his github comment linked in the comments.
Thanks to his great design of the array indexing implementation for 0.6, only a single function needs to be extended to get complement indexing for getindex, setindex and view, e.g.,
julia> view(a, Not(2:8))
2-element SubArray{Int64,1,UnitRange{Int64},Tuple{Array{Int64,1}},false}:
1
9
# collect because ranges are immutable
julia> b = collect(a); b[Not(2), Not(2)] = 10; b
3×3 Array{Int64,2}:
10 4 10
2 5 8
10 6 10
Directly overwriting [](i.e. getindex) is prone to break many indexing-related things in Base, but we can write an array wrapper to work around it. We only need to define the following three methods to get your specific test cases passed:
immutable ComplementVector{T} <: AbstractArray{T,1}
data::Vector{T}
end
Base.size(A:: ComplementVector) = size(A.data)
Base.getindex(A:: ComplementVector, i::Integer) = i > 0 ? A.data[i] : A.data[setdiff(1:end, (-i))]
Base.getindex(A:: ComplementVector, I::StepRange) = all(x->x>0, I) ? A.data[I] : A.data[setdiff(1:end, -I)]
julia> a = ComplementVector([1:8...])
julia> a[-1] == a[2:8]
true
julia> a[-(1:3)] == a[4:8]
true
julia> a[-end] == a[1:7]
true
If you would like to extend ComplementVector further more, please read the doc about Interfaces.
Update:
For safety sake, we'd better not extend AbstractArray as #Fengyang Wang suggested in the comment blow:
immutable ComplementVector{T}
data::Vector{T}
end
Base.endof(A::ComplementVector) = length(A.data)
Base.getindex(A::ComplementVector, i::Integer) = i > 0 ? A.data[i] : A.data[setdiff(1:end, (-i))]
Base.getindex(A::ComplementVector, I::OrdinalRange) = all(x->x>0, I) ? A.data[I] : A.data[setdiff(1:end, -I)]

How to convert "1-D int array" to "int" in Julia (Jupyter notebook)

There is a similar question in C How to convert int array to int?; however, I cannot find such question in Julia.
My question is the following:
There is an error about v[1] = x'*y
So I try to find the reason and it show that:
However, x^Ty=2 so if you directly type 2, it is Int64 and there is no error as shown in the following:
So how to transfrom Array{Int64,1} to Int 64,1?
The reason under the hood is that Julia-0.5 still doesn't take vector transposes seriously, actually, x' is a 1x2 matrix:
julia> x'
1×2 Array{Int64,2}:
1 1
Apparently, you would like to get the dot product of x and y, but technically speaking x'*y is not the right syntax, you should use dot(x,y) or \cdot[tab]:
julia> x ⋅ y
2
This issue has already been fixed on Julia-0.6 master by introducing a new type RowVector:
julia> x'
1×2 RowVector{Int64,Array{Int64,1}}:
1 1
julia> x'*y
2
You have to define the corresponding convert method:
Base.convert{T}(Float64, x::Array{T, 1}) = float(x[1])
Or in general
Base.convert{T,K}(::Type{K}, x::Array{T, 1}) = Base.convert(K, x[1])
Example:
v = zeros(Complex{Float64}, 3)
x = [1, 1]
y = [1, 1]
v1 = x'*y
v[1] = v1
v[2] = 45
v[3] = 100
v
Result:
3-element Array{Complex{Float64},1}:
2.0+0.0im
45.0+0.0im
100.0+0.0im

How to multiply multi-dimensional arrays/matrices in Julia

Multiplying two multi-dimensional arrays, say, a 1-dimensional with a 3-dimensional array:
[1 2] * reshape(1:8,2,2,2)
gives me the error message:
LoadError: MethodError: `*` has no method matching *(::Array{Int64,2}, ::Array{Int64,3})
Closest candidates are:
*(::Any, ::Any, !Matched::Any, !Matched::Any...)
*{TA,TB}(::Union{DenseArray{TA,1},DenseArray{TA,2},SubArray{TA,1,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD},SubArray{TA,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, !Matched::Base.LinAlg.AbstractTriangular{TB,S<:AbstractArray{T,2}})
*{TA,TQ,N}(::Union{DenseArray{TA,N},SubArray{TA,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, !Matched::Union{Base.LinAlg.QRCompactWYQ{TQ,M<:AbstractArray{T,2}},Base.LinAlg.QRPackedQ{TQ,S<:AbstractArray{T,2}}})
...
while loading In[167], in expression starting on line 1
in Ac_mul_B at operators.jl:157
using the math definition of multi-dimensional matrix algebra for a (1 by 2) * (2 by 2 by 2) multiplication of matrices/arrays.
A somewhat more general example can be A*B = C meaning sum_k A_{i,j,k} B_{k,l,m} = C_{i,j,l,m}, where A is a 3-index matrix, or tensor if you like, B is a 3-index one, and the resulting C is a four index matrix/tensor, but, in general, there can be any number of dimensions and a dimension can have any size (within reason). See more info on the definition of a matrix product or tensor contraction.
What is the right syntax of this multiplication in Julia?
You can use reshape to convert the multi-dimensional arrays into matrices,
multiply them, and convert the result back to a multi-dimensional array.
A = [1 2]
B = reshape(1:8,2,2,2)
reshape( reshape(A,2,1)' * reshape(B,2,4), 2, 2 )
(In this example, since A is already a matrix, there is actually no need to reshape it.)
Matrices multiplication: use *
Matrices element-wise multiplication: use .*
If you want to multiply two matrices A*B, the size of both must much: e.g.
size of A (a1,a2),
size of B (b1, b2);
in order to make the multiplication feasible a2 must be the same as b1
In your case:
julia> A=[1 2]
1x2 Array{Int64,2}:
1 2
julia> size(A) #remember that size(A') is (2,1)
(1,2)
julia> B=reshape(1:8,2,2,2)
2x2x2 Array{Int64,3}:
[:, :, 1] =
1 3
2 4
[:, :, 2] =
5 7
6 8
julia> size(B)
(2,2,2)
julia> size(B[:,:,1])
(2,2)
julia> A*B[:,:,1]
1x2 Array{Int64,2}:
5 11
julia> A*B[:,:,2]
1x2 Array{Int64,2}:
17 23
Edit: Multiplication in "one shot" (separated for clarity):
julia> A*B[:,:,1]
1x2 Array{Int64,2}:
5 11
julia> A*B[:,:,2]
1x2 Array{Int64,2}:
17 23
julia> C=A*B[:,:]
1x4 Array{Int64,2}:
5 11 17 23
julia> size(C)
(1,4)
julia> D1 = reshape(C, 2,2)
2x2 Array{Int64,2}:
5 17
11 23
julia> D2 = reshape(C, 2,2)'
2x2 Array{Int64,2}:
5 11
17 23
Truly in one shot:
julia> D = reshape(A*B[:,:], 2,2)
2x2 Array{Int64,2}:
5 17
11 23

Resources