I wrote an algorithm for ranking an array.
let rankFun array =
let arrayNew = List.toArray array
let arrayLength = Array.length arrayNew
let rankedArray = Array.create arrayLength 1.0
for i in 0 .. arrayLength - 2 do
for j in (i+1) .. arrayLength - 1 do
if arrayNew.[i] > arrayNew.[j] then
rankedArray.[i] <- rankedArray.[i] + 1.0
elif arrayNew.[i] < arrayNew.[j] then
rankedArray.[j] <- rankedArray.[j] + 1.0
else
rankedArray.[i] <- rankedArray.[i] + 0.0
rankedArray
I wanted to ask you what do you think about performance? I used for loops and I was wondering if you think there's another way better than this one. Before getting to this I was sorting my array, keeping original indexes, ranking and only afterwards resorting each rank to its original position, what was reeeeeally bad in terms of performance. Now I got to this improved version and was looking for some feedback. Any ideas?
Edit: Duplicated elements should have same rank. ;)
Thank you very much in advance. :)
I'm assuming that ranks can be taken from sorting the inputs, since you commented that the question's behavior on duplicates is a bug. It's surprising that the solution with sorting you described ran slower than the code shown in the question. It should be a lot faster.
A simple way to solve this via sorting is to build an ordered set from all values. Sets in F# are always ordered and contain no duplicates, so they can be used to create the ranking.
From the set, create a map from each value to its index in the set (plus one, to keep the ranking that starts with 1). With this, you can look up the rank of each value to fill the output array.
let getRankings array =
let rankTable =
Set.ofArray array
|> Seq.mapi (fun i n -> n, i + 1)
|> Map.ofSeq
array |> Array.map (fun n -> rankTable.[n])
This takes an array, rather than a list, because the input parameter in the question was called array. It also uses integers for the ranks, as this is the normal data type for this purpose.
This is much faster than the original algorithm, since all operations are at most O(n*log(n)), while the nested for-loops in the question are O(n^2). (See also: Wikipedia on Big O notation.) For only 10000 elements, the sorting-based solution already runs over 100 times faster on my computer.
(BTW, the statement else rankedArray.[i] <- rankedArray.[i] + 0.0 appears to do nothing. Unless you're doing some sort of black magic with the optimizer, you can just remove it.)
Related
I have an array of some particular ints, and I want to find all the unique combinations (by addition) of these ints. I'm sure there's a way to do this functionally; I'm trying to avoid the iterative way of shoving for loops inside of for loops. I'm using Rust in this case, but this question is more generally a, "functional programming, how do?" take.
My first thought is that I should just zip every entry in v with every other entry, reduce them by addition to a single element, and filter duplicates. This is O(|v|^2), which feels bad, meaning I'm probably missing something fairly obvious to the functional whiz kids. Also I'm not even sure how to do it, I'd probably use a for loop to construct my new massive array.
My first pass: note that v holds all the numbers I care about.
let mut massive_arr = Vec::new();
for &elem in v.iter(){
for &elem2 in v.iter(){
massive_arr.push((elem,elem2));
}
}
let mut single_massive = Vec::new();
for &tuple in massive_arr.iter(){
single_massive.push(tuple.0 + tuple.1);
}
single_massive.dedup();
let summand: usize = single_massive.iter().sum(); println!("The sum of all that junk is {:?}", summand);```
Help me baptize my depraved iterations in the pure light of functional programming.
edited: I threw up an example before as I was still figuring out the implementation that actually worked, and the question was more of a, how do I do this better question. The thing above now actually works (but is still ugly!).
You can possibly use itertools (I do not have a compiler at hand, but you probably get the idea):
use itertools::Itertools;
iproduct!(v.iter(), v.iter()) // construct all pairs
.map(|tuple| tuple.0+tuple.1) // sum each pair
.unique() // de-duplicate (uses a HashMap internally)
.sum() // sum up
All this is still O(n^2) which is -- as far as I see -- asymptotically optimal, because all pairs of numbers might be needed.
To avoid the obvious duplicates, you can use tuple_combinations:
v.iter()
.tuple_combinations()
.map(|(a, b)| a+b)
.unique()
.sum()
Improving on #phimuemue's answer, you can avoid the obvious duplicates like this:
v.iter()
.enumerate()
.flat_map (|(i, a)| v[i+1..].iter().map (move |b| a+b))
.unique() // May not be needed or what you really want, see note below
.sum()
Playground
Note however that this may not give you the answer you really want if multiple pairs of numbers have the same sum. For example, given vec![1, 2, 3, 4] as input, which do you expect:
(1+2) + (1+3) + (1+4) + (2+3) + (2+4) + (3+4) = 30
or 3 + 4 + 5 + 6 + 7 = 25 because 1+4 == 2+3 == 5 is only counted once?
I am having a large array and I want to to multiply every element of array with a given number N.
I can do this in following way
val arr = Array.fill(100000)(math.random)
val N = 5.0
val newArr = arr.map ( _ * N )
So this will return me new array as i want. An other way could be
def demo (arr :Array [Double] , value : Double ) : Array[Double] ={
var res : Array[Double] = Array()
if ( arr.length == 1 )
res = Array ( arr.head + value )
else
res = demo ( arr.slice(0, arr.length/2) , value ) ++ demo ( arr.slice ( arr.length / 2 , arr.length ) , value )
res
}
I my case I have larger array and I have to perform this operation for Thousands of iterations. I want to ask is there any faster way to get same output? Will tail recursion will increase speed? Or any other technique?
Presumably you mean arr.head * value.
Neither of these are "faster" in big-O terms; they're both O(N), which makes sense because it's right there in the description: you need to multiply "every" number by some constant.
The only difference is that in the second case you spend a bunch of time slicing and concatenating arrays. So the first one is likely going to be faster. Tail recursion isn't going to help you because this isn't a recursive problem. All you need to do is loop once through the array.
If you have a really huge number of numbers in your array, you could parallelize the multiplication across the available processors (if more than one) by using arr.par.map.
Tail recursion will be better than regular recursion if you're writing your own function to recursively loop over the list, since tail recursion doesn't fall into Stack issues like regular recursion does.
arr.map(_ * N) should be fine for you though.
Whatever you do, try not to use var. Mutable variables are a code smell in Scala.
Also, when you're dealing with thousands of values it might be worth looking into different collection types like Vector over Array; different collections are efficient at different things. For more information on the performance of collections, check out the official Scala Collections Performance Characteristics page.
For multiplying the elements of an array with a number N, the time complexity will be O(N) as you will have to evaluate all the elements of the array.
arr.map ( _ * N )
IMHO the above code is the optimal solution evaluating it in O(N). But now if your array is very huge, I would recommend you to convert the list to a stream and then perform your transformation.
For example
arr.toStream.map{ _ * 2}
I have 2 arrays, A and B. I want to form a new array C with same dimension as B where each element will show SUM(A) for A > B
Below is my working code
A = [1:1:1000]
B=[1:1:100]
for n = 1:numel(B)
C(n) = sum(A(A>B(n)));
end
However, when A has millions of rows and B has thousands, and I have to do similar calculations for 20 array-couples,it takes insane amount of time.
Is there any faster way?
For example, histcounts is pretty fast, but it counts, rather than summing.
Thanks
Depending on the size of your arrays (and your memory limitations), the following code might be slightly faster:
C = A*bsxfun(#gt,A',B);
Though it's vectorized, however, it seems to be bottlenecked (perhaps) by the allocation of memory. I'm looking to see if I can get a further speedup. Depending on your input vector size, I've seen up to a factor of 2 speedup for large vectors.
Here's a method that is a bit quicker, but I'm sure there is a better way to solve this problem.
a=sort(A); %// If A and B are already sorted then this isn't necessary!
b=sort(B);
c(numel(B))=0; %// Initialise c
s=cumsum(a,2,'reverse'); %// Get the partial sums of a
for n=1:numel(B)
%// Pull out the sum for elements in a larger than b(n)
c(n)=s(find(a>b(n),1,'first'));
end
According to some very rough tests, this seems to run a bit better than twice as fast as the original method.
You had the right ideas with histcounts, as you are basically "accumulating" certain A elements based on binning. This binning operation could be done with histc. Listed in this post is a solution that starts off with similar steps as listed in #David's answer and then uses histc to bin and sum up selective elements from A to get us the desired output and all of it in a vectorized manner. Here's the implementation -
%// Sort A and B and also get sorted B indices
sA = sort(A);
[sB,sortedB_idx] = sort(B);
[~,bin] = histc(sB,sA); %// Bin sorted B onto sorted A
C_out = zeros(1,numel(B)); %// Setup output array
%// Take care of the case when all elements in B are greater than A
if sA(1) > sB(end)
C_out(:) = sum(A);
end
%// Only do further processing if there is at least one element in B > any element in A
if any(bin)
csA = cumsum(sA,'reverse'); %// Reverse cumsum on sorted A
%// Get sum(A(A>B(n))) for every n, but for sorted versions
valid_mask = cummax(bin) - bin ==0;
valid_mask2 = bin(valid_mask)+1 <= numel(A);
valid_mask(1:numel(valid_mask2)) = valid_mask2;
C_out(valid_mask) = csA(bin(valid_mask)+1);
%// Rearrange C_out to get back in original unsorted version
[~,idx] = sort(sortedB_idx);
C_out = C_out(idx);
end
Also, please remember when comparing the result from this method with the one from the original for-loop version that there would be slight variations in output as this vectorized solution uses cumsum which computes a running summation and as such would have large cumulatively summed numbers being added to individual elements that are comparatively very small, whereas the for-loop version
would sum only selective elements. So, floating-precision issues would come up there.
I have accepted an answer to the question below, but It seemed I misunderstood how Arrays in haskell worked. I thought they were just beefed up lists. Keep that in mind when reading the question below.
I've found that monolithic arrays in haskell are quite inefficient when using them for larger arrays.
I haven't been able to find a non-monolithic implementation of arrays in haskell. What I need is O(1) time look up on a multidimensional array.
Is there an implementation of of arrays that supports this?
EDIT: I seem to have misunderstood the term monolithic. The problem is that it seems like the arrays in haskell treats an array like a list. I might be wrong though.
EDIT2: Short example of inefficient code:
fibArray n = a where
bnds = (0,n)
a = array bnds [ (i, f i) | i <- range bnds ]
f 0 = 0
f 1 = 1
f i = a!(i-1) + a!(i-2)
this is an array of length n+1 where the i'th field holds the i'th fibonacci number. But since arrays in haskell has O(n) time lookup, it takes O(n²) time to compute.
You're confusing linked lists in Haskell with arrays.
Linked lists are the data types that use the following syntax:
[1,2,3,5]
defined as:
data [a] = [] | a : [a]
These are classical recursive data types, supporting O(n) indexing and O(1) prepend.
If you're looking for multidimensional data with O(1) lookup, instead you should use a true array or matrix data structure. Good candidates are:
Repa - fast, parallel, multidimensional arrays -- (Tutorial)
Vector - An efficient implementation of Int-indexed arrays (both mutable and immutable), with a powerful loop optimisation framework . (Tutorial)
HMatrix - Purely functional interface to basic linear algebra and other numerical computations, internally implemented using GSL, BLAS and LAPACK.
Arrays have O(1) indexing. The problem is that each element is calculated lazily. So this is what happens when you run this in ghci:
*Main> :set +s
*Main> let t = 100000
(0.00 secs, 556576 bytes)
*Main> let a = fibArray t
Loading package array-0.4.0.0 ... linking ... done.
(0.01 secs, 1033640 bytes)
*Main> a!t -- result omitted
(1.51 secs, 570473504 bytes)
*Main> a!t -- result omitted
(0.17 secs, 17954296 bytes)
*Main>
Note that lookup is very fast, after it's already been looked up once. The array function creates an array of pointers to thunks that will eventually be calculated to produce a value. The first time you evaluate a value, you pay this cost. Here are a first few expansions of the thunk for evaluating a!t:
a!t -> a!(t-1)+a!(t-2)-> a!(t-2)+a!(t-3)+a!(t-2) -> a!(t-3)+a!(t-4)+a!(t-3)+a!(t-2)
It's not the cost of the calculations per se that's expensive, rather it's the need to create and traverse this very large thunk.
I tried strictifying the values in the list passed to array, but that seemed to result in an endless loop.
One common way around this is to use a mutable array, such as an STArray. The elements can be updated as they're available during the array creation, and the end result is frozen and returned. In the vector package, the create and constructN functions provide easy ways to do this.
-- constructN :: Unbox a => Int -> (Vector a -> a) -> Vector a
import qualified Data.Vector.Unboxed as V
import Data.Int
fibVec :: Int -> V.Vector Int64
fibVec n = V.constructN (n+1) c
where
c v | V.length v == 0 = 0
c v | V.length v == 1 = 1
c v | V.length v == 2 = 1
c v = let len = V.length v
in v V.! (len-1) + v V.! (len-2)
BUT, the fibVec function only works with unboxed vectors. Regular vectors (and arrays) aren't strict enough, leading back to the same problem you've already found. And unfortunately there isn't an Unboxed instance for Integer, so if you need unbounded integer types (this fibVec has already overflowed in this test) you're stuck with creating a mutable array in IO or ST to enable the necessary strictness.
Referring specifically to your fibArray example, try this and see if it speeds things up a bit:
-- gradually calculate m-th item in steps of k
-- to prevent STACK OVERFLOW , etc
gradualth m k arr
| m <= v = pre `seq` arr!m
where
pre = foldl1 (\a b-> a `seq` arr!b) [u,u+k..m]
(u,v) = bounds arr
For me, for let a=fibArray 50000, gradualth 50000 10 aran at 0.65 run time of just calling a!50000 right away.
I have the following (imperative) algorithm that I want to implement in Haskell:
Given a sequence of pairs [(e0,s0), (e1,s1), (e2,s2),...,(en,sn)], where both "e" and "s" parts are natural numbers not necessarily different, at each time step one element of this sequence is randomly selected, let's say (ei,si), and based in the values of (ei,si), a new element is built and added to the sequence.
How can I implement this efficiently in Haskell? The need for random access would make it bad for lists, while the need for appending one element at a time would make it bad for arrays, as far as I know.
Thanks in advance.
I suggest using either Data.Set or Data.Sequence, depending on what you're needing it for. The latter in particular provides you with logarithmic index lookup (as opposed to linear for lists) and O(1) appending on either end.
"while the need for appending one element at a time would make it bad for arrays" Algorithmically, it seems like you want a dynamic array (aka vector, array list, etc.), which has amortized O(1) time to append an element. I don't know of a Haskell implementation of it off-hand, and it is not a very "functional" data structure, but it is definitely possible to implement it in Haskell in some kind of state monad.
If you know approx how much total elements you will need then you can create an array of such size which is "sparse" at first and then as need you can put elements in it.
Something like below can be used to represent this new array:
data MyArray = MyArray (Array Int Int) Int
(where the last Int represent how many elements are used in the array)
If you really need stop-and-start resizing, you could think about using the simple-rope package along with a StringLike instance for something like Vector. In particular, this might accommodate scenarios where you start out with a large array and are interested in relatively small additions.
That said, adding individual elements into the chunks of the rope may still induce a lot of copying. You will need to try out your specific case, but you should be prepared to use a mutable vector as you may not need pure intermediate results.
If you can build your array in one shot and just need the indexing behavior you describe, something like the following may suffice,
import Data.Array.IArray
test :: Array Int (Int,Int)
test = accumArray (flip const) (0,0) (0,20) [(i, f i) | i <- [0..19]]
where f 0 = (1,0)
f i = let (e,s) = test ! (i `div` 2) in (e*2,s+1)
Taking a note from ivanm, I think Sets are the way to go for this.
import Data.Set as Set
import System.Random (RandomGen, getStdGen)
startSet :: Set (Int, Int)
startSet = Set.fromList [(1,2), (3,4)] -- etc. Whatever the initial set is
-- grow the set by randomly producing "n" elements.
growSet :: (RandomGen g) => g -> Set (Int, Int) -> Int -> (Set (Int, Int), g)
growSet g s n | n <= 0 = (s, g)
| otherwise = growSet g'' s' (n-1)
where s' = Set.insert (x,y) s
((x,_), g') = randElem s g
((_,y), g'') = randElem s g'
randElem :: (RandomGen g) => Set a -> g -> (a, g)
randElem = undefined
main = do
g <- getStdGen
let (grownSet,_) = growSet g startSet 2
print $ grownSet -- or whatever you want to do with it
This assumes that randElem is an efficient, definable method for selecting a random element from a Set. (I asked this SO question regarding efficient implementations of such a method). One thing I realized upon writing up this implementation is that it may not suit your needs, since Sets cannot contain duplicate elements, and my algorithm has no way to give extra weight to pairings that appear multiple times in the list.