I'm trying to write the Fisher-Yates shuffle using mutable arrays. So far, I have the following code:
module Main where
import Control.Monad.Random
import Control.Monad.Primitive
import Control.Monad.ST
import qualified Data.Vector as V
import qualified Data.Vector.Mutable as MV
fisherYates :: (MonadRandom m, PrimMonad m) => MV.MVector (PrimState m) a -> m ()
fisherYates v = forM_ [0 .. l - 1] (\i -> do j <- getRandomR (i, l)
MV.swap v i j)
where l = MV.length v - 1
shuffle :: MonadRandom m => V.Vector a -> m (V.Vector a)
shuffle v = _ -- don't know how to write this
main :: IO ()
main = print . evalRand (shuffle . V.generate 10 $ id) $ mkStdGen 42
However, I am totally unsure how to define shuffle, which is meant to be a 'high-level wrapper' around the mutable vector operations. It seems (at least from my understanding), that I first have to 'run' the random 'part' of the stack, save the state, run the ST 'part' to get out an immutable vector, and then rewrap the whole thing. Additionally, I know I have to make use of thaw somewhere, but my attempts are coming up short. Could someone please tell me what I'm missing, and how I can do what I want?
I have two suggestions for you:
Select the right monad nesting.
Separate out your monad implementation from the logic of the algorithm.
You are trying to run the random monad last and use the ST internally, thus you need the ST to be a sort of monad transformer. Decide what your monad stack looks like - which monad is the transformer and which is the inner monad? The easy thing to do is make the ST monad the inner monad and the random monad the transformer (easy just because you have all the needed packages already).
Now make a small set of helper functions. It won't really pay off here -
the payoff is large for complex projects. Here's the monad stack and helpers I came up with:
{-# LANGUAGE RankNTypes #-}
module Main where
import System.Random (StdGen)
import Control.Monad.Random
import Control.Monad.Primitive
import Control.Monad.ST
import qualified Data.Vector as V
import qualified Data.Vector.Mutable as MV
type KozM s a = RandT StdGen (ST s) a
Notice the transformer is RandT and the inner monad of ST s.
rnd :: (Int,Int) -> KozM s Int
rnd = getRandomR
swp :: MV.MVector s a -> Int -> Int -> KozM s ()
swp v i j = lift (MV.swap v i j)
freeze :: MV.MVector s a -> KozM s (V.Vector a)
thaw :: V.Vector a -> KozM s (MV.MVector s a)
freeze = lift . V.freeze
thaw = lift . V.thaw
All the operations you need to mutate the vector. Now we just need to run this monad so we can somehow escape to another useful context. I noticed you previously hard-coded an RNG (42) - I use a random one but whichever...
run :: (forall s. KozM s a) -> IO a -- Can be just `a` if you hard-code
-- an RNG as done in the question
run m = do g <- newStdGen
pure (runST (evalRandT m g))
Finally we can use this monad to implement f-y:
fisherYates :: MV.MVector s a -> KozM s ()
fisherYates v = forM_ [0 .. l - 1] (\i -> do j <- rnd (i, l)
swp v i j)
where l = MV.length v - 1
At this point you might not be feeling like you learned anything, hopefully the run function was helpful but I can see how you might feel this is getting verbose. The important thing to take note of here is how clean the rest of your code can be if you handle the plumbing of the monad above so you don't have lift and module qualifiers polluting the logic of the possibly complex thing you actually set-out to solve.
That said, here's the underwhelming shuffle:
shuffle :: V.Vector a -> KozM s (V.Vector a)
shuffle v = do
vm <- thaw v
fisherYates vm
freeze vm
The type is important. You had previously called evalRand on shuffle which implied it would be some sort of MonadRandom m and simultaneously have to call runST - a conflation of the monad logic and the algorithm logic that just hurts the brain.
The main is uninteresting:
main :: IO ()
main = print =<< (run (shuffle (V.generate 10 id)) :: IO (V.Vector Int))
EDIT: yes you can do this while keeping the methods more general. At some point you need to specify which monad you run or you can't have a main that will execute it, but all the logic can be general using MonadRandom and PrimMonad.
{-# LANGUAGE RankNTypes #-}
module Main where
import System.Random (StdGen)
import Control.Monad.Random
import Control.Monad.Primitive
import Control.Monad.ST
import qualified Data.Vector as V
import qualified Data.Vector.Mutable as MV
type KozM s a = RandT StdGen (ST s) a
rnd :: MonadRandom m => (Int, Int) -> m Int
rnd = getRandomR
swp :: PrimMonad m => MV.MVector (PrimState m) a -> Int -> Int -> m ()
swp v i j = MV.swap v i j
-- freeze :: MV.MVector s a -> KozM s (V.Vector a)
-- thaw :: V.Vector a -> KozM s (MV.MVector s a)
freeze :: PrimMonad m => MV.MVector (PrimState m) a -> m (V.Vector a)
thaw :: PrimMonad m => V.Vector a -> m (MV.MVector (PrimState m) a)
freeze = V.freeze
thaw = V.thaw
-- Some monad libraries, like monadlib, have a generalized `run` class method.
-- This doesn't exist, to the best of my knowledge, for the intersection of ST
-- and mtl.
run :: (forall s. KozM s a) -> IO a -- Can be just `a` if you hard-code
-- an RNG as done in the question
run m = do g <- newStdGen
pure (runST (evalRandT m g))
-- fisherYates :: MV.MVector s a -> KozM s ()
fisherYates :: (MonadRandom m, PrimMonad m) => MV.MVector (PrimState m) a -> m ()
fisherYates v = forM_ [0 .. l - 1] (\i -> do j <- rnd (i, l)
swp v i j)
where l = MV.length v - 1
shuffle :: (MonadRandom m, PrimMonad m) => V.Vector a -> m (V.Vector a)
shuffle v = do
vm <- thaw v
fisherYates vm
freeze vm
main :: IO ()
main = print =<< (run (shuffle (V.generate 10 id)) :: IO (V.Vector Int))
Related
Consider the Haskell functions
test :: ST s [Int]
test = do
arr <- newListArray (0,9) [0..9] :: ST s (STArray s Int Int)
let f i = do writeArray arr i (2*i)
readArray arr i
forM [1,2] f
and
test' :: ST s [Int]
test' = do
arr <- newListArray (0,9) [0..9] :: ST s (STArray s Int Int)
let f = \i -> do writeArray arr i (2*i)
readArray arr i
forM [1,2] f
The first requires FlexibleContexts to compile on ghci 8.10.1, the second compiles with no extra options. Why?
An answer that explains this behaviour in terms of the scope of the type variable s would be especially welcome. As a follow up, what (if any) type signature can be added to the function f to make test compile without FlexibleContexts? Finally, is there a connection with the monomorphism restriction?
You can check which type GHC assigns to f in GHCi:
ghci> import Data.Array
ghci> import Data.Array.MArray
ghci> let arr :: STArray s Int Int; arr = undefined
ghci> :t \i -> do writeArray arr i (2*i); readArray arr i
\i -> do writeArray arr i (2*i); readArray arr i
:: (MArray (STArray s1) Int m, MArray (STArray s2) Int m) =>
Int -> m Int
This is more general than the type you suggest in your comments and the reason that FlexibleContexts are needed.
You can add the type signature you suggested (Int -> ST s Int) to avoid having to use FlexibleContexts:
{-# LANGUAGE ScopedTypeVariables #-}
...
test :: forall s. ST s [Int]
test = do
arr <- newListArray (0,9) [0..9] :: ST s (STArray s Int Int)
let
f :: Int -> ST s Int
f i = do
writeArray arr i (2*i)
readArray arr i
forM [1,2] f
Note that scoped type variables and the forall s. are necessary here because you need to make sure that the s in all the type signatures refer to the same type variable and do not all introduce new type variables.
The reason that the monomorphism restriction treats your first and your second version differently is because it doesn't apply to things that look like functions. In your first version f has an argument, so it looks like a function and therefore will get a general type. In your second version f doesn't have arguments, so it doesn't look like a function which means that the monomorphism restriction forces it to have a more specific type.
I have an Array r Ix2 a such that (Manifest r Ix2 a, Ord a). I'd like to sort this array on its innermost dimension - that is, sort each row internally, but not across rows. According to this, massiv doesn't have any sorting implemented at all. Would I have to roll my own, or can I re-use something that already exists for Vectors (such as vector-algorithms for example)?
Of course, it would be better to roll out your own sorting and submit a PR to massiv library ;) But there is a way to fall back onto vector-algorithms package. I was curious how I'd do it efficiently and here it is, along with automatic parallelization of sorting each row:
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TypeFamilies #-}
module Examples.SortRows where
import Data.Massiv.Array as A
import Data.Massiv.Array.Manifest.Vector as A
import Data.Massiv.Core.Scheduler
import Data.Typeable
import Data.Vector.Algorithms.Merge
import Data.Vector.Generic as VG
import Data.Vector.Generic.Mutable as VGM
import System.IO.Unsafe
sortRows ::
forall r e v.
(Ord e, Typeable v, A.Mutable r Ix2 e, VG.Vector v e, ARepr v ~ r, VRepr r ~ v)
=> Array r Ix2 e
-> Array r Ix2 e
sortRows arr = unsafePerformIO $ do
mv :: VG.Mutable v RealWorld e <- VG.thaw (A.toVector arr :: v e)
let comp = getComp arr
sz#(m :. n) = size arr
case comp of
Seq -> do
loopM_ 0 (< m) (+ 1) $ \i -> sort $ VGM.slice (toLinearIndex sz (i :. 0)) n mv
ParOn wIds ->
withScheduler_ wIds $ \scheduler -> do
loopM_ 0 (< m) (+ 1) $ \i ->
scheduleWork scheduler $ sort $ VGM.slice (toLinearIndex sz (i :. 0)) n mv
v :: v e <- VG.unsafeFreeze mv
return $ A.fromVector comp sz v
I did add this to examples in massiv in this commit together with a simple property test.
I'm doing this Car Game problem on Kattis: https://open.kattis.com/problems/cargame
There's a five-second time limit, but on the last instance, my code requires longer to run. I'm fairly sure I'm doing the right thing (from a big-O standpoint) so now I need to optimize it somehow.
I downloaded the test data from:
http://challenge.csc.kth.se/2013/challenge-2013.tar.bz2
From profiling, it seems like most of the running time is spent in containsSub which is nothing more than an array access together with a tail-recursive call. Furthermore, it's only called about 100M times, so why does it take 6.5 seconds to run (6.5 s on my laptop. I've found Kattis is generally about twice as slow, so probably more like 13 seconds). On the statistics page, some of the C++ solutions run in under a second. Even some python solutions just barely make it under the 5-second bar.
module Main where
import Control.Monad
import Data.Array (Array, (!), (//))
import qualified Data.Array as Array
import Data.ByteString.Char8 (ByteString)
import qualified Data.ByteString.Char8 as BS
import Data.Char
import Data.List
import Data.Maybe
main::IO()
main = do
[n, m] <- readIntsLn
dictWords <- replicateM n BS.getLine
let suffixChains = map (\w -> (w, buildChain w)) dictWords
replicateM_ m $ findChain suffixChains
noWordMsg :: ByteString
noWordMsg = BS.pack "No valid word"
findChain :: [(ByteString, WordChain)] -> IO ()
findChain suffixChains = do
chrs <- liftM (BS.map toLower) BS.getLine
BS.putStrLn
(
case find (containsSub chrs . snd) suffixChains of
Nothing -> noWordMsg
Just (w, _) -> w
)
readAsInt :: BS.ByteString -> Int
readAsInt = fst . fromJust . BS.readInt
readIntsLn :: IO [Int]
readIntsLn = liftM (map readAsInt . BS.words) BS.getLine
data WordChain = None | Rest (Array Char WordChain)
emptyChars :: WordChain
emptyChars = Rest . Array.listArray ('a', 'z') $ repeat None
buildChain :: ByteString -> WordChain
buildChain s =
case BS.uncons s of
Nothing -> emptyChars
Just (hd, tl) ->
let wc#(Rest m) = buildChain tl in
Rest $ m // [(hd, wc)]
containsSub :: ByteString -> WordChain -> Bool
containsSub _ None = False
containsSub s (Rest m) =
case BS.uncons s of
Nothing -> True
Just (hd, tl) -> containsSub tl (m ! hd)
EDIT: TAKE 2:
I tried building a lazy trie to avoid searching things I'd already searched. So for instance, if I've already encountered a triplet beginning with 'a', then in the future I can skip anything which doesn't contain an 'a'. If I've already searched a triplet beginning 'ab', I can skip anything which doesn't contain 'ab'. And if I've already searched the exact triplet 'abc', I can just return the same result from last time. In theory, this should contribute a significant speedup. In practice the running time is identical.
Furthermore, without the seq's, profiling takes forever and gives bogus results (I couldn't guess why).
With the seqs, profiling says the bulk of the time is spent in forLetter (which is where the array accesses have been moved to so again it looks like array access is the slow part)
{-# LANGUAGE TupleSections #-}
module Main where
import Control.Monad
import Data.Array (Array, (!), (//))
import qualified Data.Array as Array
import qualified Data.Array.Base as Base
import Data.ByteString.Char8 (ByteString)
import qualified Data.ByteString.Char8 as BS
import Data.Char
import Data.Functor
import Data.Maybe
main::IO()
main = do
[n, m] <- readIntsLn
dictWords <- replicateM n BS.getLine
let suffixChainsL = map (\w -> (w, buildChain w)) dictWords
let suffixChains = foldr seq suffixChainsL suffixChainsL
suffixChains `seq` doProbs m suffixChains
noWordMsg :: ByteString
noWordMsg = BS.pack "No valid word"
doProbs :: Int -> [(ByteString, WordChain)] -> IO ()
doProbs m chains = replicateM_ m doProb
where
cf = findChain chains
doProb =
do
chrs <- liftM (map toLower) getLine
BS.putStrLn . fromMaybe noWordMsg $ cf chrs
findChain :: [(ByteString, WordChain)] -> String -> Maybe ByteString
findChain [] = const Nothing
findChain suffixChains#(shd : _) = doFind
where
letterMap :: Array Char (String -> Maybe ByteString)
letterMap =
Array.listArray ('a','z')
[findChain (mapMaybe (forLetter hd) suffixChains) | hd <- [0..25]]
endRes = Just $ fst shd
doFind :: String -> Maybe ByteString
doFind [] = endRes
doFind (hd : tl) = (letterMap ! hd) tl
forLetter :: Int -> (ByteString, WordChain) -> Maybe (ByteString, WordChain)
forLetter c (s, WC wc) = (s,) <$> wc `Base.unsafeAt` c
readAsInt :: BS.ByteString -> Int
readAsInt = fst . fromJust . BS.readInt
readIntsLn :: IO [Int]
readIntsLn = liftM (map readAsInt . BS.words) BS.getLine
newtype WordChain = WC (Array Char (Maybe WordChain))
emptyChars :: WordChain
emptyChars = WC . Array.listArray ('a', 'z') $ repeat Nothing
buildChain :: ByteString -> WordChain
buildChain = BS.foldr helper emptyChars
where
helper :: Char -> WordChain -> WordChain
helper hd wc#(WC m) = m `seq` WC (m // [(hd, Just wc)])
The uncons call in containsSub creates a new ByteString. Try speeding it up by keeping track of the offset into the string with an index, e.g.:
containsSub' :: ByteString -> WordChain -> Bool
containsSub' str wc = go 0 wc
where len = BS.length str
go _ None = False
go i (Rest m) | i >= len = True
| otherwise = go (i+1) (m ! BS.index str i)
After much discussion on the #haskell and #ghc IRC channels, I found that the problem was related to this ghc bug: https://ghc.haskell.org/trac/ghc/ticket/1168
The solution was simply to change the definition of doProbs
doProbs m chains = cf `seq` replicateM_ m doProb
...
Or just to compile with -fno-state-hack
ghc's state hack optimization was causing it to unnecessarily recompute cf (and the associated letterMap) on every call.
So it has nothing to do with array accesses.
What do I have to do to make GHC accept this code:
{-# LANGUAGE MultiParamTypeClasses, FlexibleInstances #-}
module STTest where
import Data.Array.ST
import Control.Monad.ST.Strict as S
import Control.Monad.ST.Lazy as L
-- ST monad arrays (unboxed in actual code)
type Arr s a = STArray s Int a
-- representing some algorithm that works on these STArrays
data ArrGen s a = ArrGen (a -> S.ST s (Arr s a)) (Arr s a -> S.ST s ())
-- class for some "generator"
class Generator g a where
gen :: g -> a -> [a]
instance Generator (ArrGen s a) a where
gen (ArrGen create apply) s = L.runST $ do
a <- strictToLazyST $ create s -- DOES NOT WORK
strictToLazyST $ apply a >> getElems a
The error I get is the following:
Couldn't match type `s' with `s1'
`s' is a rigid type variable bound by
the instance declaration at STTest.hs:20:28
`s1' is a rigid type variable bound by
a type expected by the context: L.ST s1 [a] at STTest.hs:21:33
However, this works fine:
data Dummy
create' :: a -> S.ST s (Arr s a)
create' = undefined
apply' :: Arr s a -> S.ST s [a]
apply' = undefined
instance Generator Dummy a where
gen _ s = L.runST $ do
a <- strictToLazyST $ create' s
strictToLazyST $ apply' a >> getElems a
Why does it work with the second and not the first? And what can I do with the data declaration to make it work? Or can I add some sort of "forall" on the instance declaration?
The above is just a minimal test program. I actually loop the apply forever to create an infinite Stream of the output values. (So I can't just merge the two steps together.) And I really want to be able to instantiate once for the ArrGen data type and then make a variety of values of it using these STArray algorithms.
EDIT:
Didn't think to put the forall inside the functions to ArrGen (I put it on the overall type). Though now I have the a problem of getting it to work on STUArray. Like if I use the following:
class (Integral a, Bits a, forall s. MArray (STUArray s) a (S.ST s)) => HasSTU a
type AC a = (HasSTU a) => forall s. a -> S.ST s (STUArray s Int a)
type AU a = (HasSTU a) => forall s. STUArray s Int a -> S.ST s ()
type TX a = (HasSTU a) => a -> a -- or without the context
data ArrayGen a = AG (AC a) (AU a) (TX a)
Then this fails:
instance (HasSTU a) => Generator (ArrayGen a) a [a] where
gens (AG c u p) s = fmap (fmap p) $ L.runST $ do
ar <- strictToLazyST $ (c s)
streamM $ strictToLazyST $ u ar >> getElems ar -- can't use getElems here!
streamM :: (Applicative f) => f a -> f (Stream a))
streamM = Cons <$> a <*> streamM a
It complains:
Could not deduce (MArray (STUArray s) a (S.ST s))
arising from a use of `getElems'
from the context (HasSTU a)
Even though the context (HasSTU a) says (in my mind) that there is an (MArray (STUArray s) a (S.ST s)) context for all s, it doesn't seem to think so. I tried to fix it by changing the (AU a) type:
type AU a = (HasSTU a) => forall s. STUArray s Int a -> S.ST s [a]
And it seems to type check, but I am unable to actually use it. Similarly if I change to:
class (Integral a, Bits a, forall s. MArray (STUArray s) a (S.ST s)) => HasSTU s a
type AC a = (forall s. HasSTU s a) => a -> S.ST s (STUArray s Int a)
...
instance (forall s. HasSTU s a) => Generator (ArrayGen a) a [a] where
...
instance forall s. HasSTU s Word32 -- !!!
But then when I try to run something:
Could not deduce (forall s. HasSTU s Word32)
I hate this s! Why? I have an instance for all s! And I am really lost as to where I should put my foralls and what's really going on.
The problem is that runST requires a forall s. ST s t argument, but your type fixes s, so a use of create and apply in the monadic action makes it unsuitable for runST.
It does not seem to me that your use case forbids giving ArrGen polymorphic (in s) arguments, so
{-# LANGUAGE MultiParamTypeClasses, FlexibleInstances, RankNTypes #-}
module STTest where
import Data.Array.ST
import Control.Monad.ST.Strict as S
import Control.Monad.ST.Lazy as L
-- ST monad arrays (unboxed in actual code)
type Arr s a = STArray s Int a
-- representing some algorithm that works on these STArrays
data ArrGen a = ArrGen (forall s. a -> S.ST s (Arr s a)) (forall s. Arr s a -> S.ST s ())
-- class for some "generator"
class Generator g a where
gen :: g -> a -> [a]
instance Generator (ArrGen a) a where
gen (ArrGen create apply) s = L.runST $ do
a <- strictToLazyST $ create s -- DOES NOT WORK
strictToLazyST $ apply a >> getElems a
making the components polymorphic works (at least in the sense that it compiles, your use case may forbid this approach).
Why does it work with the second and not the first?
Because there, the s was not fixed, the computation is fully polymorphic in s, as required by runST.
let me call the function accumrArray.
accumrArray ::
(e' -> e -> e) An accumulating function
-> e A default element
-> (i, i) The bounds of the array
-> [(i, e')] List of associations
-> a i e The array
accumrArray (:) [] (1,2) [(1,1),(2,2),(2,3)] === array [(1,[1]), (2,[2,3])]
head $ (accumrArray (:) [] (1,1) [(1,x)|x<-[4..]]) ! 1 === 4
How strange... I wrote this function a few days ago for someone else. The function first appeared in LML (I believe), but never made it into the Haskell array library.
Here you go:
{-# LANGUAGE ScopedTypeVariables #-}
import Data.Array
import System.IO.Unsafe
import Data.IORef
import Data.Array.MArray
import Data.Array.Base
import Control.Monad
import Data.Array.IO
accumArrayR :: forall a e i. Ix i => (a -> e -> e) -> e -> (i,i) -> [(i,a)] -> Array i e
accumArrayR f e bounds#(l,u) assocs = unsafePerformIO $ do
ref <- newIORef assocs
arr <- newArray_ bounds
let _ = arr :: IOArray i e
let n = safeRangeSize (l,u)
let elem x = unsafePerformIO $ do
ass <- readIORef ref
let loop [] = writeIORef ref [] >> return e
loop ((y,a):rest) = do
let ix = safeIndex bounds n y
let r = f a (elem x)
unsafeWrite arr ix r
if (ix == x)
then writeIORef ref rest >> return r
else loop rest
loop ass
forM_ [0..n] $ \ix -> unsafeWrite arr ix (elem ix)
unsafeFreeze arr
A challenge for the reader: use accumArrayR to implement linear-time depth-first-search of a graph.
Edit I should mention that the function isn't thread-safe as written. Turning the IORef into an MVar would fix it, but there might be better ways.
Not the most efficient, but...
accumrArray f x b l = accumArray (flip f) x b (reverse l)
I would argue that
accumrArray f x b l = accumArray (flip f) x b (reverse l)
is indeed the best solution (credits to sclv's answer).
Its supposed "inefficiency" comes from fact that foldr applies the function f from right to left.
However, since accumArray is strict, l can never be infinite, otherwise the program would be incorrect. It would never terminate.
Therefore, foldl (flip f) is just as good as a foldr.