I'm unclear on how the outer function provided to walk should work.
The example in documentation referenced here:
https://docs.hylang.org/en/stable/contrib/walk.html
Suggests the outer function can be first which would return the first element of the collection produced by mapping over with inner.
However when I try to aggregate the result in outer (eg sum or first) I get an error as per below - complaining that int is not iterable - looking at the source code I suspect this is because of (type form) in the macro definition:
((type form) (outer (HyExpression (map inner form))))
Can anyone confirm, and advise if there is a way of having outer return a different type to the input form? i.e. can (walk inc sum [1 2 3 4 5]) be made to provide the sum of the list [2 3 4 5 6] as I'd expect?
=> (walk inc identity [1 2 3 4 5])
[2, 3, 4, 5, 6]
=> (walk inc accumulate [1 2 3 4 5])
[2, 5, 9, 14, 20]
=> (walk inc sum [1 2 3 4 5])
Traceback (most recent call last):
File "stdin-75eb4a20707c49e8c921e986e7d6164d36b7e4b2", line 1, in <module>
(walk inc sum [1 2 3 4 5])
File "/home/phil/.local/lib/python3.6/site-packages/hy/contrib/walk.hy", line 22, in walk
((type form) (outer (HyExpression (map inner form))))]
TypeError: 'int' object is not iterable
=> (walk inc first [1 2 3 4 5])
Traceback (most recent call last):
File "stdin-710dcc990bf071fe1a9a4c5501831c41867f1879", line 1, in <module>
(walk inc first [1 2 3 4 5])
File "/home/phil/.local/lib/python3.6/site-packages/hy/contrib/walk.hy", line 22, in walk
((type form) (outer (HyExpression (map inner form))))]
TypeError: 'int' object is not iterable
=>
It's a bug. Say (sum (map inc [1 2 3 4 5])) instead.
Related
I am trying to access elements of the previous columns than first column.
For example in python if i try to access elements before the first column 0 it will return an empty array.
import numpy as np
x = np.matrix([[ 1, 2, 10],
[ 1, 4, 2],
[ 2, 3, 2],
[ 2, 3, 6]])
a = x[:, 0]
print('\n a is \n', a)
b = x[:, :0]
print('\n b is \n',b)
Output: -->
a is
[[1]
[1]
[2]
[2]]
b is
[]
But julia does not behave in such manner.
My trial in julia:
array = [1 2 3;3 4 5;2 3 4;1 2 3]
a = array[:, 1]
b = array[:, 1-1]
Output :-->
BoundsError: attempt to access 4×3 Array{Int64,2} at index [1:4, 0]
Stacktrace:
[1] throw_boundserror(::Array{Int64,2}, ::Tuple{Base.Slice{Base.OneTo{Int64}},Int64}) at .\abstractarray.jl:541
[2] checkbounds at .\abstractarray.jl:506 [inlined]
[3] _getindex at .\multidimensional.jl:742 [inlined]
[4] getindex(::Array{Int64,2}, ::Function, ::Int64) at .\abstractarray.jl:1060
[5] top-level scope at In[1166]:1
[6] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1091
However, to replicate such operation I tried to implement try and catch method and use it over the loop to see what happens when i try accessing different columns. The output is always an empty array.
for i in 1:3
try
b = array[:, :i-1]
catch
b = []
end
println("\b b in $i is $b")
end
Output :--->
b in 1 is Any[]
b in 2 is Any[]
b in 3 is Any[]
May I know how can I access row elements from first_column - 1 to last_column - 1, where first_column - 1 returns an empty array but rest returns data from the matrix.
The answer to my question is provided by #ettersi on julia discourse, I am posting it here for reference. Please follow the link mentioned in the comment to access the discourse page.
ANSWER!
julia> array = [1 2 3;3 4 5;2 3 4;1 2 3]
4×3 Matrix{Int64}:
1 2 3
3 4 5
2 3 4
1 2 3
julia> array[:, 1:0]
4×0 Matrix{Int64}
My old friend the 3d array called Pop, I want to remove columns (d2), across all of d3 when the value in the end of d1 == 1, so I have this code:
Pop[end, :, 1] .!=1
I thought adding #view in front of this would write the changes back to Pop, outwith producing an additional copy in memory. This code works fine
#view(Pop[ :, Pop[end, :, 1] .!=1, :])
but it does not alter the original 3d array called Pop. I could do
Pop = #view(Pop[ :, Pop[end, :, 1] .!=1, :])
but I believe this will create another copy in memory which I'm trying to avoid. What simple syntax have I missed? Thx. J
#view does not modify the size of the original array, it provides a "view" into it, (e.g., omitting some columns in your case). I don't think there is anything wrong with
Pop = view(Pop, :, Pop[end, :, 1] .≠ 1, :)
since now Pop is a view into your old, full Pop, but it behaves like an array, so you can modify its entries, e.g., you could then do things like
julia> using Random # using a fixed seed for reproducibility
julia> Random.seed!(0) ;
julia> Pop = rand(1:5, (2,4,2)) # original Pop
2×4×2 Array{Int64,3}:
[:, :, 1] =
4 3 5 5
1 1 3 5
[:, :, 2] =
2 2 3 1
2 5 1 1
julia> Pop[end,:,1] .≠ 1 # columns to keep
4-element BitArray{1}:
0
0
1
1
julia> Pop = view(Pop, :, Pop[end, :, 1] .≠ 1, :) # make it a view
2×2×2 view(::Array{Int64,3}, :, [3, 4], :) with eltype Int64:
[:, :, 1] =
5 5
3 5
[:, :, 2] =
3 1
1 1
julia> Pop[end,:,1] .= 1 ; # use your new view to manipulate data
julia> Pop # view of the modified Pop
2×2×2 view(::Array{Int64,3}, :, [3, 4], :) with eltype Int64:
[:, :, 1] =
5 5
1 1
[:, :, 2] =
3 1
1 1
julia> Pop.parent # original full Pop (now Pop.parent) has been modified
2×4×2 Array{Int64,3}:
[:, :, 1] =
4 3 5 5
1 1 1 1
[:, :, 2] =
2 2 3 1
2 5 1 1
What is the most efficient way to create a view on array using, for example, sliding window=2
Let's say we have:
x = collect(1:1:6)
# 1 2 3 4 5 6
And I want to create a view like this:
# 1 2
# 2 3
# 3 4
# 4 5
# 5 6
So far I found only this option, but not sure if it's an optimal one:
y = Array{Float32, 2}(undef, nslides, window)
#inbounds for i in 1:window
y[:, i] = #view x[i:end-(window-i)]
end
One solution with a package (well, with my package) is this:
julia> using Tullio
julia> x = 1:6; window = 2;
julia> #tullio y[r,c] := x[r+c-1] (c in 1:window)
5×2 Matrix{Int64}:
1 2
2 3
3 4
4 5
5 6
The one liner is:
view.(Ref(x), (:).(1:length(x)-1,2:length(x)))
Testing:
julia> x=collect(1:6);
julia> view.(Ref(x), (:).(1:length(x)-1,2:length(x)))
5-element Array{SubArray{Int64,1,Array{Int64,1},Tuple{UnitRange{Int64}},true},1}:
[1, 2]
[2, 3]
[3, 4]
[4, 5]
[5, 6]
Explanation:
creation of views is vectorized by the dot operator .
we do not want to vectorize on elements of x so use Ref(x) instead
(:) is just a shorter form for UnitRange and again we use the dot operator . to vectorize
I used 2 as the Window size but of course you can write view.(Ref(x), (:).(1:length(x)-(window-1),window:length(x)))
EDIT:
If you want rather a library function this would work for you:
julia> using ImageFiltering
julia> mapwindow(collect, x, 0:1,border=Inner())
5-element OffsetArray(::Array{Array{Int64,1},1}, 1:5) with eltype Array{Int64,1} with indices 1:5:
[1, 2]
[2, 3]
[3, 4]
[4, 5]
[5, 6]
Of course you could put them the function that you want to run on the sliding window rather than just collect.
This question already has answers here:
Convert pandas dataframe to NumPy array
(15 answers)
Closed 4 years ago.
I am using a pandas data frame to clean and process data. However, I need to then convert it into a numpy ndarray in order to use exploit matrix multiplication. I turn the data frame into a list of lists with the following:
x = df.tolist()
This returns the following structure:
[[1, 2], [3, 4], [5, 6], [7, 8] ...]
I then convert it into a numpy array like this:
x = np.array(x)
However, the following print:
print(type(x))
print(type(x[0]))
gives this result:
'numpy.ndarray'
'numpy.float64'
However, I need them both to be numpy arrays. If it's not from a pandas data frame and I just convert a hard-coded list of lists then they are both ndarrays. How do I get the list, and the lists in that list to be ndarrays when that list has been made from a data frame? Many thanks for reading, this has had me stumped for hours.
I think you need values:
df = pd.DataFrame({'C':[7,8,9,4,2,3],
'D':[1,3,5,7,1,0]})
print (df)
C D
0 7 1
1 8 3
2 9 5
3 4 7
4 2 1
5 3 0
x = df.values
print (x)
[[7 1]
[8 3]
[9 5]
[4 7]
[2 1]
[3 0]]
And then select by indexing:
print (x[:,0])
[7 8 9 4 2 3]
print (x[:,1])
[1 3 5 7 1 0]
print (type(x[:,0]))
<class 'numpy.ndarray'>
Also is possible transpose array:
x = df.values.T
print (x)
[[7 8 9 4 2 3]
[1 3 5 7 1 0]]
print (x[0])
[7 8 9 4 2 3]
print (x[1])
[1 3 5 7 1 0]
How about as_matrix:
x = df.as_matrix()
You may want to try df.get_values(), and eventually np.reshape it.
For example, I can create an array that contains a function.
julia> a(x) = x + 1
>> a (generic function with 1 method)
julia> [a]
>> 1-element Array{#a,1}:
a
But I can't seem to add the function to an empty array:
julia> append!([],a)
>> ERROR: MethodError: no method matching length(::#a)
Closest candidates are:
length(::SimpleVector) at essentials.jl:168
length(::Base.MethodList) at reflection.jl:256
length(::MethodTable) at reflection.jl:322
...
in _append!(::Array{Any,1}, ::Base.HasLength, ::Function) at .\collections.jl:25
in append!(::Array{Any,1}, ::Function) at .\collections.jl:21
What I ulimately want to do is store the pre-defined functions so that I can ultimately map them over a value. E.g.:
x = 0.0
for each fn in vec
x = x + fn(x)
end
append! is for appending one collection on to another.
You are looking for push!, to add an element to a collection.
Your code should be push!([], a)
See the docs:
julia>?append!
search: append!
append!(collection, collection2) -> collection.
Add the elements of collection2 to the end of collection.
julia> append!([1],[2,3])
3-element Array{Int64,1}:
1
2
3
julia> append!([1, 2, 3], [4, 5, 6])
6-element Array{Int64,1}:
1
2
3
4
5
6
Use push! to add individual items to collection which are not already themselves in another collection. The result is of the preceding example is equivalent to push!([1, 2, 3], 4, 5, 6).
vs:
julia>?push!
search: push! pushdisplay
push!(collection, items...) -> collection
Insert one or more items at the end of collection.
julia> push!([1, 2, 3], 4, 5, 6)
6-element Array{Int64,1}:
1
2
3
4
5
6
Use append! to add all the elements of another collection to collection. The result of the preceding example is equivalent to append!([1, 2, 3], [4, 5, 6]).