I have the following (imperative) algorithm that I want to implement in Haskell:
Given a sequence of pairs [(e0,s0), (e1,s1), (e2,s2),...,(en,sn)], where both "e" and "s" parts are natural numbers not necessarily different, at each time step one element of this sequence is randomly selected, let's say (ei,si), and based in the values of (ei,si), a new element is built and added to the sequence.
How can I implement this efficiently in Haskell? The need for random access would make it bad for lists, while the need for appending one element at a time would make it bad for arrays, as far as I know.
Thanks in advance.
I suggest using either Data.Set or Data.Sequence, depending on what you're needing it for. The latter in particular provides you with logarithmic index lookup (as opposed to linear for lists) and O(1) appending on either end.
"while the need for appending one element at a time would make it bad for arrays" Algorithmically, it seems like you want a dynamic array (aka vector, array list, etc.), which has amortized O(1) time to append an element. I don't know of a Haskell implementation of it off-hand, and it is not a very "functional" data structure, but it is definitely possible to implement it in Haskell in some kind of state monad.
If you know approx how much total elements you will need then you can create an array of such size which is "sparse" at first and then as need you can put elements in it.
Something like below can be used to represent this new array:
data MyArray = MyArray (Array Int Int) Int
(where the last Int represent how many elements are used in the array)
If you really need stop-and-start resizing, you could think about using the simple-rope package along with a StringLike instance for something like Vector. In particular, this might accommodate scenarios where you start out with a large array and are interested in relatively small additions.
That said, adding individual elements into the chunks of the rope may still induce a lot of copying. You will need to try out your specific case, but you should be prepared to use a mutable vector as you may not need pure intermediate results.
If you can build your array in one shot and just need the indexing behavior you describe, something like the following may suffice,
import Data.Array.IArray
test :: Array Int (Int,Int)
test = accumArray (flip const) (0,0) (0,20) [(i, f i) | i <- [0..19]]
where f 0 = (1,0)
f i = let (e,s) = test ! (i `div` 2) in (e*2,s+1)
Taking a note from ivanm, I think Sets are the way to go for this.
import Data.Set as Set
import System.Random (RandomGen, getStdGen)
startSet :: Set (Int, Int)
startSet = Set.fromList [(1,2), (3,4)] -- etc. Whatever the initial set is
-- grow the set by randomly producing "n" elements.
growSet :: (RandomGen g) => g -> Set (Int, Int) -> Int -> (Set (Int, Int), g)
growSet g s n | n <= 0 = (s, g)
| otherwise = growSet g'' s' (n-1)
where s' = Set.insert (x,y) s
((x,_), g') = randElem s g
((_,y), g'') = randElem s g'
randElem :: (RandomGen g) => Set a -> g -> (a, g)
randElem = undefined
main = do
g <- getStdGen
let (grownSet,_) = growSet g startSet 2
print $ grownSet -- or whatever you want to do with it
This assumes that randElem is an efficient, definable method for selecting a random element from a Set. (I asked this SO question regarding efficient implementations of such a method). One thing I realized upon writing up this implementation is that it may not suit your needs, since Sets cannot contain duplicate elements, and my algorithm has no way to give extra weight to pairings that appear multiple times in the list.
Related
I have large 1D arrays a and b, and an array of pointers I that separates them into subarrays. My a and b barely fit into RAM and are of different dtypes (one contains UInt32s, the other Rational{Int64}s), so I don’t want to join them into a 2D array, to avoid changing dtypes.
For each i in I[2:end], I wish to sort the subarray a[I[i-1],I[i]-1] and apply the same permutation to the corresponding subarray b[I[i-1],I[i]-1]. My attempt at this is:
function sort!(a,b)
p=sortperm(a);
a[:], b[:] = a[p], b[p]
end
Threads.#threads for i in I[2:end]
sort!( a[I[i-1], I[i]-1], b[I[i-1], I[i]-1] )
end
However, already on a small example, I see that sort! does not alter the view of a subarray:
a, b = rand(1:10,10), rand(-1000:1000,10) .//1
sort!(a,b); println(a,"\n",b) # works like it should
a, b = rand(1:10,10), rand(-1000:1000,10) .//1
sort!(a[1:5],b[1:5]); println(a,"\n",b) # does nothing!!!
Any help on how to create such function sort! (as efficient as possible) are welcome.
Background: I am dealing with data coming from sparse arrays:
using SparseArrays
n=10^6; x=sprand(n,n,1000/n); #random matrix with 1000 entries per column on average
x = SparseMatrixCSC(n,n,x.colptr,x.rowval,rand(-99:99,nnz(x)).//1); #chnging entries to rationals
U = randperm(n) #permutation of rows of matrix x
a, b, I = U[x.rowval], x.nzval, x.colptr;
Thus these a,b,I serve as good examples to my posted problem. What I am trying to do is sort the row indices (and corresponding matrix values) of entries in each column.
Note: I already asked this question on Julia discourse here, but received no replies nor comments. If I can improve on the quality of the question, don't hesitate to tell me.
The problem is that a[1:5] is not a view, it's just a copy. instead make the view like
function sort!(a,b)
p=sortperm(a);
a[:], b[:] = a[p], b[p]
end
Threads.#threads for i in I[2:end]
sort!(view(a, I[i-1]:I[i]-1), view(b, I[i-1]:I[i]-1))
end
is what you are looking for
ps.
the #view a[2:3], #view(a[2:3]) or the #views macro can help making thins more readable.
First of all, you shouldn't redefine Base.sort! like this. Now, sort! will shadow Base.sort! and you'll get errors if you call sort!(a).
Also, a[I[i-1], I[i]-1] and b[I[i-1], I[i]-1] are not slices, they are just single elements, so nothing should happen if you sort them either with views or not. And sorting arrays in a moving-window way like this is not correct.
What you want to do here, since your vectors are huge, is call p = partialsortperm(a[i:end], i:i+block_size-1) repeatedly in a loop, choosing a block_size that fits into memory, and modify both a and b according to p, then continue to the remaining part of a and find next p and repeat until nothing remains in a to be sorted. I'll leave the implementation as an exercise for you, but you can come back if you get stuck on something.
I am very new to Haskell (and functional programming in general) and I am trying to write a function called
"profileDistance m1 m2" that takes two matrices as parameters and needs to calculate the sum of the differences between each element in each matrix... I might have not explained that very well. Let me show it instead.
The matrices are on the form of: [[(Char,Int)]]
where each matrix might look something like this:
m1 = [[('A',1),('A',2)],
[('B',3),('B',4)],
[('C',5),('C',6)]]
m2 = [[('A',7),('A',8)],
[('B',9),('B',10)],
[('C',11),('C',12)]]
(Note: I wrote the numbers in order in this example but they can be ANY numbers in any order. The chars in each row in each matrix will however match like shown in the example.)
The result (in the case above) would look something like (psuedo code):
result = ((snd m1['A'][0])-(snd m2['A'][0]))+((snd m1['A'][1])-(snd m2['A'][1]))+((snd m1['B'][0])-(snd m2['B'][0]))+((snd m1['B'][1])-(snd m2['B'][1]))+((snd m1['C'][0])-(snd m2['C'][0]))+((snd m1['C'][1])-(snd m2['C'][1]))
This would be easy to do in any language that has for-loops and is non-functional but I have no idea how to do this in Haskell. I have a feeling that functions like map, fold or sum would help me here (admittedly I am not a 100% sure on how fold works). I hope there is an easy way to do this... please help.
Here a proposal:
solution m1 m2 = sum $ zipWith diffSnd flatM1 flatM2
where
diffSnd t1 t2 = snd t1 - snd t2
flatM1 = concat m1
flatM2 = concat m2
I wrote it so that it's easier to understand the building blocks.
The basic idea is to iterate simultaneously on our two lists of pairs using zipWith. Here its type:
zipWith :: (a -> b -> c) -> [a] -> [b] -> [c]
It means it takes a function with type a -> b -> c, a list of a's and a list of b's, and it returns a list of c's. In other words, zipWith takes case of the iteration, you just have to specify what you want to do with every item the iteration yields, that in your case will be a pair of pairs (one from the first matrix, another one from the second).
The function passed to zipWith takes the snd element from each pair, and computes the difference. Looking back at zipWith signature you can deduce it will return a list of numbers. So the last thing we need to do is summing them, using the function sum.
There's one last problem. We actually do not have two lists of pairs to be passed to zipWith!, but two matrices. We need to "flatten" them in a list, preserving the order of the elements. That's exactly what concat does, hence the calls to that function in the definitions of flatM1 and flatM2.
I suggest you look into the implementation of every function I mentioned to have a better grasp of how iteration is expressed by mean of recursion. HTH
I was suprised to learn that Array and List were two different types in Elm:
Array
List
In my case, I have a List Int of length 2,000,000 and I need about 10,000 of them but I don't know in advance which ten thousand. That will be provided by another list. In pseudo-code:
x = [ 1,1,0,30,...,255,0,1 ]
y = [ 1,4,7,18,36,..., 1334823 , ... 1899876 ]
z = [ y[x[0]], y[x[1]], ... ]
I am using pseudocode because clearly this isn't Elm syntax (it might be legal JavaScript).
Can these array selections be done in List or Array or both?
List is a linked list which provides O(n) lookup time based on index. Getting an element by index requires traversing the list over n nodes. An index lookup function for List isn't available in the core library but you can use the elm-community/list-extra package which provides two functions for lookup (varying by parameter order): !! and getAt.
Array allows for O(log n) index lookup. Index lookups on Array can be done using Array.get. Arrays are represented as Relaxed Radix Balanced Trees.
Both are immutable (all values in Elm are immutable), so you have trade-offs depending on your situation. List is great when you make a lot of changes because you are merely updating linked list pointers, whereas Array is great for speedy lookup but has somewhat poorer performance for modifications, which you'll want to consider if you're making a lot of changes.
Something like this should work:
import Array
import Debug
fromJust : Maybe a -> a
fromJust x = case x of
Just y -> y
Nothing -> Debug.crash "error: fromJust Nothing"
selectFromList : List a -> List Int -> List a
selectFromList els idxs =
let arr = Array.fromList els
in List.map (\i -> fromJust (Array.get i arr)) idxs
It converts the input list to an array for fast indexing, then maps the list of indices to their corresponding values in the array. I took the fromJust function from this StackOverflow question.
Only use Array if you need to use Array.get.
In most cases you should use List because usually you can do everything you need with foldl, map, etc. without having to get items from an index, and List has better performance with these functions.
I have accepted an answer to the question below, but It seemed I misunderstood how Arrays in haskell worked. I thought they were just beefed up lists. Keep that in mind when reading the question below.
I've found that monolithic arrays in haskell are quite inefficient when using them for larger arrays.
I haven't been able to find a non-monolithic implementation of arrays in haskell. What I need is O(1) time look up on a multidimensional array.
Is there an implementation of of arrays that supports this?
EDIT: I seem to have misunderstood the term monolithic. The problem is that it seems like the arrays in haskell treats an array like a list. I might be wrong though.
EDIT2: Short example of inefficient code:
fibArray n = a where
bnds = (0,n)
a = array bnds [ (i, f i) | i <- range bnds ]
f 0 = 0
f 1 = 1
f i = a!(i-1) + a!(i-2)
this is an array of length n+1 where the i'th field holds the i'th fibonacci number. But since arrays in haskell has O(n) time lookup, it takes O(n²) time to compute.
You're confusing linked lists in Haskell with arrays.
Linked lists are the data types that use the following syntax:
[1,2,3,5]
defined as:
data [a] = [] | a : [a]
These are classical recursive data types, supporting O(n) indexing and O(1) prepend.
If you're looking for multidimensional data with O(1) lookup, instead you should use a true array or matrix data structure. Good candidates are:
Repa - fast, parallel, multidimensional arrays -- (Tutorial)
Vector - An efficient implementation of Int-indexed arrays (both mutable and immutable), with a powerful loop optimisation framework . (Tutorial)
HMatrix - Purely functional interface to basic linear algebra and other numerical computations, internally implemented using GSL, BLAS and LAPACK.
Arrays have O(1) indexing. The problem is that each element is calculated lazily. So this is what happens when you run this in ghci:
*Main> :set +s
*Main> let t = 100000
(0.00 secs, 556576 bytes)
*Main> let a = fibArray t
Loading package array-0.4.0.0 ... linking ... done.
(0.01 secs, 1033640 bytes)
*Main> a!t -- result omitted
(1.51 secs, 570473504 bytes)
*Main> a!t -- result omitted
(0.17 secs, 17954296 bytes)
*Main>
Note that lookup is very fast, after it's already been looked up once. The array function creates an array of pointers to thunks that will eventually be calculated to produce a value. The first time you evaluate a value, you pay this cost. Here are a first few expansions of the thunk for evaluating a!t:
a!t -> a!(t-1)+a!(t-2)-> a!(t-2)+a!(t-3)+a!(t-2) -> a!(t-3)+a!(t-4)+a!(t-3)+a!(t-2)
It's not the cost of the calculations per se that's expensive, rather it's the need to create and traverse this very large thunk.
I tried strictifying the values in the list passed to array, but that seemed to result in an endless loop.
One common way around this is to use a mutable array, such as an STArray. The elements can be updated as they're available during the array creation, and the end result is frozen and returned. In the vector package, the create and constructN functions provide easy ways to do this.
-- constructN :: Unbox a => Int -> (Vector a -> a) -> Vector a
import qualified Data.Vector.Unboxed as V
import Data.Int
fibVec :: Int -> V.Vector Int64
fibVec n = V.constructN (n+1) c
where
c v | V.length v == 0 = 0
c v | V.length v == 1 = 1
c v | V.length v == 2 = 1
c v = let len = V.length v
in v V.! (len-1) + v V.! (len-2)
BUT, the fibVec function only works with unboxed vectors. Regular vectors (and arrays) aren't strict enough, leading back to the same problem you've already found. And unfortunately there isn't an Unboxed instance for Integer, so if you need unbounded integer types (this fibVec has already overflowed in this test) you're stuck with creating a mutable array in IO or ST to enable the necessary strictness.
Referring specifically to your fibArray example, try this and see if it speeds things up a bit:
-- gradually calculate m-th item in steps of k
-- to prevent STACK OVERFLOW , etc
gradualth m k arr
| m <= v = pre `seq` arr!m
where
pre = foldl1 (\a b-> a `seq` arr!b) [u,u+k..m]
(u,v) = bounds arr
For me, for let a=fibArray 50000, gradualth 50000 10 aran at 0.65 run time of just calling a!50000 right away.
I'm trying to check if my arrays are returning nonsense by accessing out of bounds elements, in fortran. And I want to check these values are less than one, and if they are, change them to one.
This is the piece of my code causing issues:
lastNeighLabel=(/clusterLabel(jj-1,kk,ll), clusterLabel(jj,kk-1,ll), clusterLabel(jj,kk,ll-1)/)
LastNeighLabel contains the cluster label (between 1 and n, where n isthe total number of unique seperate clusters found) for the last neighbour in the x,y,z direction respectively.
When jj or kk or ll are 1, they try and access the 0th element in the array, and as FORTRAN counts from 1 in arrays, it tries to destroy the universe. I'm currently in a tangled mess of about 8 if/elseif statements trying to code for every eventuality. But I was hoping there was a way of operating on each element. So basically I'd like to say where((/jj-1,kk-1,ll-1/).lt.1) do clusterLabel(jj-1,kk,ll)=0 etc depending on which element is causing the problem.
But I can't think of a way to do that because where will only manipulate the variables passed to it, not a different array at the same index. Or am I wrong?
Will gladly edit if this doesn't make sense.
It is not obligatory that Fortran accesses arrays starting from one. Any starting value is allowed. If it more convenient to you to have a zero indexed array, declare the array as:
real, dimension (0:N-1, 0:M-1) :: array
Or
real, dimension (0:N, 0:M) :: array
and have the 0 indices be extra to catch special cases.
This might be another solution to your problem, since zero index values would be legal.
Another possible way to approach this, is to create an extended cluster label array (with index bounds starting at 0), which is equal to the cluster label array with a layer of zeroes tacked on the outside. You can then let your loop run safely over all values of jj, kk, and ll. It depends on the size of the array if this is a feasible solution.
integer :: extended_cluster_label(0:size(cluster_label,1), &
0:size(cluster_label,2), &
0:size(cluster_label,3) &
)
extended_cluster_label(0,:,:) = 0
extended_cluster_label(:,0,:) = 0
extended_cluster_label(:,:,0) = 0
extended_cluster_label(1:, 1:, 1:) = cluster_label
Maybe you could use a function?
real function f(A,i,j,k)
real :: A(:,:,:)
integer :: i,j,k
if (i==0.or.j==0.or.k==0) then
f=0
else
f=A(i,j,k)
endif
end function f
and then use f(clusterLabel,jj-1,kk,ll) etc.