Loading and storing a file at compile time with a type provider - ffi

Question

I'd like to load a (binary) file at compile time and store it in a toplevel variable of type Bytes:
module FileProvider
import Data.Bits
import Data.Bytes
import Data.Buffer
%default total
export
loadImage : String -> IO (Provider Bytes)
loadImage fileName = do
Right file <- openFile fileName Read
| Left err => pure (Error $ show err)
Just buf <- newBuffer size
| Nothing => pure (Error "allocation failed")
readBufferFromFile file buf size
Provide . dropPrefix 2 . pack <$> bufferData buf
where
size : Int
size = 256 * 256 + 2
It seems to work correctly at runtime:
module Main
import FileProvider
import Data.Bits
import Data.Bytes
main : IO ()
main = do
Provide mem <- loadImage "ram.mem"
| Error err => putStrLn err
printLn $ length mem
However, if I try to run it at compile time, it fails with a mysterious message mentioning FFI:
module Main
import FileProvider
import Data.Bits
import Data.Bytes
%language TypeProviders
%provide (mem : Bytes) with loadImage "ram.mem"
main : IO ()
main = do
printLn $ length mem
$ idris -p bytes -o SOMain SOMain.idr
Symbol "idris_newBuffer" not found
idris: user error (Could not call foreign function "idris_newBuffer" with args [65538])
What is going on here and how can I load a file's contents at compile time?

Answer 1

idris_newbuffer is a C function used by Data.Buffer. From the docs to type providers:
If we want to call our foreign functions from interpreted code (such as the REPL or a type provider), we need to dynamically link a library containing the symbols we need.
So every function that uses FFI needs to have a dynamic library linked. That'd be Data.Buffer and Data.ByteArray. Lets focus on the first and see what problems arise:
So Data.Buffer needs %dynamic "idris_buffer.so" (and not only the %include C "idris_buffer.h" it currently has). You can copy idris/rts/idris_buffer.(h|c) and remove the function that needs the other rts-stuff. To compile a shared library run:
cc -o idris_buffer.so -fPIC -shared idris_buffer.c
With a modified Data.Buffer to use it, you'd still get an error:
Could not use <<handle {handle: ram.mem}>> as FFI arg.
The FFI call is in Data.Buffer.readBufferFromFile. The File argument causes trouble. That's because Idris sees that openFile is used (another C function) and
transforms it in a Haskell call. On the one hand that's nice, because during compilation we're interpreting Idris code, and that make following functions like readLine C/JS/Node/… agnostic. But in this case it's unfortunate, because the Haskell backend doesn't support the returned file handle for FFI calls. So we can write another fopen : String -> String -> IO Ptr function that does the same thing but has another name so the Ptr will stay Ptr, as those can be used in FFI calls.
With this done, there is is another error thanks to built-ins:
Could not use prim__registerPtr (<<ptr 0x00000000009e2bf0>>) (65546) as FFI arg.
Data.Buffer uses ManagedPtr in its backend. And yep, that's unsupported in FFI calls also. So you'd need to change these to Ptr. Both of these could get supported in the compiler, I guess.
Finally everything should work for a valid %provide (mem : Buffer). But, nope, because:
Can't convert pointers back to TT after execution.
Even though Idris can now read a file in compilation time, it can't make Buffer or anything else with a Pnt accessible to runtime - and that's quite reasonable. Having just a pointer from when the program was compiled is just a random thing at run time. So you either need to transform the data to the result in the provider or use an intermediate format like Provider (List Bits8).
I made a short example to have the List Bits8 accessible in main. Buffer is basically Data.Buffer with _openFile included and Pnt instead of ManagedPtr. I hope this helps you somehow, but maybe some of the compiler people can give more background.