Related
I have large 1D arrays a and b, and an array of pointers I that separates them into subarrays. My a and b barely fit into RAM and are of different dtypes (one contains UInt32s, the other Rational{Int64}s), so I don’t want to join them into a 2D array, to avoid changing dtypes.
For each i in I[2:end], I wish to sort the subarray a[I[i-1],I[i]-1] and apply the same permutation to the corresponding subarray b[I[i-1],I[i]-1]. My attempt at this is:
function sort!(a,b)
p=sortperm(a);
a[:], b[:] = a[p], b[p]
end
Threads.#threads for i in I[2:end]
sort!( a[I[i-1], I[i]-1], b[I[i-1], I[i]-1] )
end
However, already on a small example, I see that sort! does not alter the view of a subarray:
a, b = rand(1:10,10), rand(-1000:1000,10) .//1
sort!(a,b); println(a,"\n",b) # works like it should
a, b = rand(1:10,10), rand(-1000:1000,10) .//1
sort!(a[1:5],b[1:5]); println(a,"\n",b) # does nothing!!!
Any help on how to create such function sort! (as efficient as possible) are welcome.
Background: I am dealing with data coming from sparse arrays:
using SparseArrays
n=10^6; x=sprand(n,n,1000/n); #random matrix with 1000 entries per column on average
x = SparseMatrixCSC(n,n,x.colptr,x.rowval,rand(-99:99,nnz(x)).//1); #chnging entries to rationals
U = randperm(n) #permutation of rows of matrix x
a, b, I = U[x.rowval], x.nzval, x.colptr;
Thus these a,b,I serve as good examples to my posted problem. What I am trying to do is sort the row indices (and corresponding matrix values) of entries in each column.
Note: I already asked this question on Julia discourse here, but received no replies nor comments. If I can improve on the quality of the question, don't hesitate to tell me.
The problem is that a[1:5] is not a view, it's just a copy. instead make the view like
function sort!(a,b)
p=sortperm(a);
a[:], b[:] = a[p], b[p]
end
Threads.#threads for i in I[2:end]
sort!(view(a, I[i-1]:I[i]-1), view(b, I[i-1]:I[i]-1))
end
is what you are looking for
ps.
the #view a[2:3], #view(a[2:3]) or the #views macro can help making thins more readable.
First of all, you shouldn't redefine Base.sort! like this. Now, sort! will shadow Base.sort! and you'll get errors if you call sort!(a).
Also, a[I[i-1], I[i]-1] and b[I[i-1], I[i]-1] are not slices, they are just single elements, so nothing should happen if you sort them either with views or not. And sorting arrays in a moving-window way like this is not correct.
What you want to do here, since your vectors are huge, is call p = partialsortperm(a[i:end], i:i+block_size-1) repeatedly in a loop, choosing a block_size that fits into memory, and modify both a and b according to p, then continue to the remaining part of a and find next p and repeat until nothing remains in a to be sorted. I'll leave the implementation as an exercise for you, but you can come back if you get stuck on something.
I have two beginner's questions:
(1) I want to reshape an array, but dimensions come from a vector which can be a variable. For example,
A = ones(120,1)
b = [2,3,4,5]
I can write
C = reshape(A,2,3,4,5)
But in case b can vary, I want something like
C = reshape(A,b)
This code works in Matlab. Is there an analog in Julia?
(2) I want to slice a high-dimensional array, while keeping the dimensions flexible. In the example above, I fix the last dimension:
C[:,:,:,1]
C[:,:,:,2]
etc. The problem is to find an efficient way: For an array of any dimensions, I can always fix the last dimension and extract values.
Any help will be highly appreciated!
(1) C = reshape(A,b...)
(2) EllipsisNotation.jl provides a .. operator, so C[..,1] does what you want.
And there is C[ntuple(x->:, ndims(C)-1)..., 1] for (2) if you don't want to install a package.
I have to solve Sudoku puzzles in the format of a vector containing 9 vectors (of length 9 each). Seeing as vectors are linked lists in Prolog, I figured the search would go faster if I transformed the puzzles in a 2D array format first.
Example puzzle:
puzzle(P) :- P =
[[_,_,8,7,_,_,_,_,6],
[4,_,_,_,_,9,_,_,_],
[_,_,_,5,4,6,9,_,_],
[_,_,_,_,_,3,_,5,_],
[_,_,3,_,_,7,6,_,_],
[_,_,_,_,_,_,_,8,9],
[_,7,_,4,_,2,_,_,5],
[8,_,_,9,_,5,_,2,3],
[2,_,9,3,_,8,7,6,_]].
I'm using ECLiPSe CLP to implement a solver. The best I've come up with so far is to write a domain like this:
domain(P):-
dim(P,[9,9]),
P[1..9,1..9] :: 1..9.
and a converter for the puzzle (parameter P is the given puzzle and Sudoku is the new defined grid with the 2D array). But I'm having trouble linking the values from the given initial puzzle to my 2D array.
convertVectorsToArray(Sudoku,P):-
( for(I,1,9),
param(Sudoku,P)
do
( for(J,1,9),
param(Sudoku,P,I)
do
Sudoku[I,J] is P[I,J]
)
).
Before this, I tried using array_list (http://eclipseclp.org/doc/bips/kernel/termmanip/array_list-2.html), but I kept getting type errors. How I did it before:
convertVectorsToArray(Sudoku,P):-
( for(I,1,9),
param(Sudoku,P)
do
( for(J,1,9),
param(Sudoku,P,I)
do
A is Sudoku[I],
array_list(A,P[I])
)
).
When my Sudoku finally outputs the example puzzle P in the following format:
Sudoku = []([](_Var1, _Var2, 8, 7, ..., 6), [](4, ...), ...)
then I'll be happy.
update
I tried again with the array_list; it almost works with the following code:
convertVectorsToArray(Sudoku,P):-
( for(I,1,9),
param(Sudoku,P)
do
X is Sudoku[I],
Y is P[I],
write(I),nl,
write(X),nl,
write(Y),nl,
array_list(X, Y)
).
The writes are there to see how the vectors/arrays look like. For some reason, it stops at the second iteration (instead of 9 times) and outputs the rest of the example puzzle as a vector of vectors. Only the first vector gets assigned correctly.
update2
While I'm sure the answer given by jschimpf is correct, I also figured out my own implementation:
convertVectorsToArray(Sudoku,[],_).
convertVectorsToArray(Sudoku,[Y|Rest],Count):-
X is Sudoku[Count],
array_list(X, Y),
NewCount is Count + 1,
convertVectorsToArray(Sudoku,Rest,NewCount).
Thanks for the added explanation on why it didn't work before though!
The easiest solution is to avoid the conversion altogether by writing your puzzle specification directly as a 2-D array. An ECLiPSe "array" is simply a structure with the functor '[]'/N, so you can write:
puzzle(P) :- P = [](
[](_,_,8,7,_,_,_,_,6),
[](4,_,_,_,_,9,_,_,_),
[](_,_,_,5,4,6,9,_,_),
[](_,_,_,_,_,3,_,5,_),
[](_,_,3,_,_,7,6,_,_),
[](_,_,_,_,_,_,_,8,9),
[](_,7,_,4,_,2,_,_,5),
[](8,_,_,9,_,5,_,2,3),
[](2,_,9,3,_,8,7,6,_)).
You can then use this 2-D array directly as the container for your domain variables:
sudoku(P) :-
puzzle(P),
P[1..9,1..9] :: 1..9,
...
However, if you want to keep your list-of-lists puzzle specification, and convert that to an array-of-arrays format, you can use array_list/2. But since that only works for 1-D arrays, you have to convert the nesting levels individually:
listoflists_to_matrix(Xss, Xzz) :-
% list of lists to list of arrays
( foreach(Xs,Xss), foreach(Xz,Xzs) do
array_list(Xz, Xs)
),
% list of arrays to array of arrays
array_list(Xzz, Xzs).
As for the reason your own code didn't work: this is due to the subscript notation P[I]. This
requires P to be an array (you were using it on lists)
works only in contexts where an arithmetic expression is expected, e.g. the right hand side of is/2, in arithmetic constraints, etc.
I have two arrays of 2x2 matrices, and I'd like to apply a function over each pair of 2x2 matrices. Here's a minimal example, multiplying each matrix in A by its corresponding matrix in B:
A <- array(1:20, c(5,2,2))
B <- array(1:20, c(5,2,2))
n <- nrow(A)
# Desired output: array with dimension 5x2x2 that contains
# the product of each pair of 2x2 matrices in A and B.
C <- aperm(sapply(1:n, function(i) A[i,,]%*%B[i,,], simplify="array"), c(3,1,2))
This takes two arrays, each with 5 2x2 matrices, and multiplies each pair of 2x2 matrices together, with the desired result in C.
My current code is this ugly last line, using sapply to loop through the first array dimension and pull out each 2x2 matrix separately from A and B. And then I need to permute the array dimensions with aperm() in order to have the same ordering as the original arrays (sapply(...,simplify="array") indexes each 2x2 matrix using the third dimension rather than the first one).
Is there a nicer way to do this? I hate that ugly function(i) in there, which is really just a way of faking a for loop. And the aperm() call makes this much less readable. What I have now works fine; I'm just searching for something that feels more like idiomatic R.
mapply() will take multiple lists or vectors, but it doesn't seem to work with arrays. aaply() from plyr is also close, but it doesn't take multiple inputs. The closest I've come is to use abind() with aaply() to pack A and B into one array work with 2 matrices at once, but this doesn't quite work (it only gets the first two entries; somewhere my indexing is off):
aaply(.data=abind(A,B,along=0), 1, function(ab) ab[1,,]%*%ab[2,,])
And this isn't exactly cleaner or clearer anyway!
I've tried to make this a minimal example, but my real use case requires a more complicated function of the matrix pairs (and I'd also love to scale this up to more than two arrays), so I'm looking for something that will generalize and scale.
D <- aaply(abind(A, B, along = 4), 1, function(x) x[,,1] %*% x[,,2])
This is a working solution using abind and aaply.
Sometimes a for loop is the easiest to follow. It also generalizes and scales:
n <- nrow(A)
C <- A
for(i in 1:n) C[i,,] <- A[i,,] %*% B[i,,]
R's infrastructure for lists is much better (it seems) than for arrays, so I could also approach it by converting the arrays into lists of matrices like this:
A <- alply(A, 1, function(a) matrix(a, ncol=2, nrow=2))
B <- alply(A, 1, function(a) matrix(a, ncol=2, nrow=2))
mapply(function(a,b) a%*%b, A, B, SIMPLIFY=FALSE)
I think this is more straightforward than what I have above, but I'd still love to hear better ideas.
I have the following (imperative) algorithm that I want to implement in Haskell:
Given a sequence of pairs [(e0,s0), (e1,s1), (e2,s2),...,(en,sn)], where both "e" and "s" parts are natural numbers not necessarily different, at each time step one element of this sequence is randomly selected, let's say (ei,si), and based in the values of (ei,si), a new element is built and added to the sequence.
How can I implement this efficiently in Haskell? The need for random access would make it bad for lists, while the need for appending one element at a time would make it bad for arrays, as far as I know.
Thanks in advance.
I suggest using either Data.Set or Data.Sequence, depending on what you're needing it for. The latter in particular provides you with logarithmic index lookup (as opposed to linear for lists) and O(1) appending on either end.
"while the need for appending one element at a time would make it bad for arrays" Algorithmically, it seems like you want a dynamic array (aka vector, array list, etc.), which has amortized O(1) time to append an element. I don't know of a Haskell implementation of it off-hand, and it is not a very "functional" data structure, but it is definitely possible to implement it in Haskell in some kind of state monad.
If you know approx how much total elements you will need then you can create an array of such size which is "sparse" at first and then as need you can put elements in it.
Something like below can be used to represent this new array:
data MyArray = MyArray (Array Int Int) Int
(where the last Int represent how many elements are used in the array)
If you really need stop-and-start resizing, you could think about using the simple-rope package along with a StringLike instance for something like Vector. In particular, this might accommodate scenarios where you start out with a large array and are interested in relatively small additions.
That said, adding individual elements into the chunks of the rope may still induce a lot of copying. You will need to try out your specific case, but you should be prepared to use a mutable vector as you may not need pure intermediate results.
If you can build your array in one shot and just need the indexing behavior you describe, something like the following may suffice,
import Data.Array.IArray
test :: Array Int (Int,Int)
test = accumArray (flip const) (0,0) (0,20) [(i, f i) | i <- [0..19]]
where f 0 = (1,0)
f i = let (e,s) = test ! (i `div` 2) in (e*2,s+1)
Taking a note from ivanm, I think Sets are the way to go for this.
import Data.Set as Set
import System.Random (RandomGen, getStdGen)
startSet :: Set (Int, Int)
startSet = Set.fromList [(1,2), (3,4)] -- etc. Whatever the initial set is
-- grow the set by randomly producing "n" elements.
growSet :: (RandomGen g) => g -> Set (Int, Int) -> Int -> (Set (Int, Int), g)
growSet g s n | n <= 0 = (s, g)
| otherwise = growSet g'' s' (n-1)
where s' = Set.insert (x,y) s
((x,_), g') = randElem s g
((_,y), g'') = randElem s g'
randElem :: (RandomGen g) => Set a -> g -> (a, g)
randElem = undefined
main = do
g <- getStdGen
let (grownSet,_) = growSet g startSet 2
print $ grownSet -- or whatever you want to do with it
This assumes that randElem is an efficient, definable method for selecting a random element from a Set. (I asked this SO question regarding efficient implementations of such a method). One thing I realized upon writing up this implementation is that it may not suit your needs, since Sets cannot contain duplicate elements, and my algorithm has no way to give extra weight to pairings that appear multiple times in the list.