I'm trying to read a gzip file in Fortran using the C functions gzopen, gzread, and gzclose from the zlib library. My subroutine works properly when it contains a print statement, but gives a Z_STREAM_ERROR (-2) without it. What is causing this to happen, and how can I fix it?
module gzmodule
use :: iso_c_binding
implicit none
private
public fastunzip
interface
type(c_ptr) function gzopen(filename,mode) bind(c)
use :: iso_c_binding
character(kind=c_char), dimension(*) :: filename
character(kind=c_char), dimension(*) :: mode
end function gzopen
end interface
interface
integer(c_int) function gzread(gzfile,buffer,length) bind(c)
use :: iso_c_binding
type(c_ptr), value :: gzfile
character(len=1,kind=c_char) :: buffer(*)
integer(c_int) :: length
end function gzread
end interface
interface
integer(c_int) function gzclose(gzfile) bind(c)
use :: iso_c_binding
type(c_ptr), value :: gzfile
end function
end interface
contains
subroutine fastunzip(filename, isize,abuf,ierr)
use :: iso_c_binding
character(len=*,kind=c_char), intent(in) :: filename
integer(c_int), intent(out) :: isize
character(len=1,kind=c_char), intent(inout) :: abuf(:,:,:,:)
integer(4), intent(out) :: ierr
type(c_ptr) :: gzfile
integer(c_int) :: iclose
logical :: c_associated
ierr = 1 !! indicates that an error has occured
isize = 0
gzfile = gzopen(trim(filename)//c_null_char,"rb")
if (.not.c_associated(gzfile)) return
isize = gzread(gzfile,abuf,size(abuf))
print*,isize !! why do I need this for it to work?
if (isize.ne.size(abuf)) return
iclose = gzclose(gzfile)
if (iclose.ne.0) return
ierr = 0 !! success
end subroutine fastunzip
end module gzmodule
program main
use gzmodule
implicit none
character(100) :: filename = './f10_19950120v7.gz'
integer(4) :: isize
integer(4) :: ierr
logical(4) :: exists
integer(4), parameter :: nlon = 1440
integer(4), parameter :: nlat = 720
integer(4), parameter :: nvar = 5
integer(4), parameter :: nasc = 2
character(1) :: abuf(nlon,nlat,nvar,nasc)
inquire(file=filename,exist=exists)
if (.not.exists) stop 'file not found'
call fastunzip(filename, isize,abuf,ierr)
print*,'return value of isize ',isize
if (ierr.ne.0) stop 'error in fastunzip'
print*,'done'
end program main
I'm on CentOS and compiling with:
gfortran -o example_usage.exe example_usage.f90 /lib64/libz.so.1
and the data file is available at this site.
In subroutine fastunzip you declare logical :: c_associated. However, you get this function by use association (of iso_c_binding), so you should remove that line.
My installed gfortran (4.8) marks that as an error, so I guess you have an older version? But once I remove that line your code appears to work even without the print, so perhaps that is worth trying for you.
On a style note, I'd recommend use, intrinsic :: iso_c_binding, perhaps even with only (which would also flag to you that the c_associated is through use association).
Related
I want to open a file in Haskell, but I want the top level function to be called from C (I want to pass the filepath from C).
I'm having trouble getting the filepath CString into a type that I can use readFile on.
Here's my first attempt, adapting the example from the docs:
{-# LANGUAGE ForeignFunctionInterface #-}
import Foreign.C.Types
import Foreign.C (CString, peekCString)
openFileDoStuff :: String -> IO Bool
openFileDoStuff filename = do
lines <- (fmap lines . readFile) filename
print lines
-- do stuff with lines
return True
openFilepathHs :: CString -> IO Bool
openFilepathHs cstr = openFileDoStuff (peekCString cstr)
foreign export ccall openFilepathHs :: CString -> IO Bool
I get a compiler error passing (peekCString cstr) to openFileDoStuff:
• Couldn't match type: IO String
with: [Char]
If I change the signature of my function to openFileDoStuff :: IO String -> IO Bool, I then can't use the filename parameter in the readFile call:
• Couldn't match type: IO String
with: [Char]
If it's not abundantly clear, I am a newbie in Haskell. I know there's no way to convert IO String -> String, but there must be a way to actually use the CString type.
Use >>= to combine IO actions.
openFilepathHs cstr = peekCString cstr >>= openFileDoStuff
Actually, this pattern of passing a piece of data through successive IO transformations is so common it has a standard combinator for abbreviation.
openFilepathHs = peekCString >=> openFileDoStuff
You can also use do syntax to hide calls to >>=, but as a beginner I personally found do syntax very difficult to understand before I understood how to make calls to >>= myself.
openFilepathHs cstr = do
cstrContents <- peekCString cstr
openFileDoStuff cstrContents
I needed to run the IO String and bind it to a variable:
{-# LANGUAGE ForeignFunctionInterface #-}
import Foreign.C.Types
import Foreign.C (CString, peekCString)
openFileDoStuff :: IO String -> IO Bool
openFileDoStuff filename = do
filenameString <- filename
lines <- (fmap lines . readFile) filenameString
print lines
-- do stuff with lines
return True
openFilepathHs :: CString -> IO Bool
openFilepathHs cstr = openFileDoStuff (peekCString cstr)
foreign export ccall openFilepathHs :: CString -> IO Bool
I have problem using the results of a subroutine in the main program. I wrote this code:
Program RK4
implicit none
real k1,k2,k3,k4,h,t,R
integer i,n
real a
read*,n,h
t=0
R=0
Do i=1,n
call Scale_Factor(h,n,t,a)
k1=h*(1/a(t))
k2=h*(1/a(t+h/2.0))
k3=h*(1/a(t+h/2.0))
k4=h*(1/a(t+h))
t=t+h
R=R+(k1+2*k2+2*k3+k4)*(1/6.0)
write(*,*)t,R
End Do
end program
!-----------------------------------------
SUBROUTINE Scale_Factor(h,n,t,a)
implicit none
real t,a,k1,k2,k3,k4,h,g
integer i,n
t=0
a=0.001
Do i=1,n
k1=h*g(a)
k2=h*g(a+k1/2.0)
k3=h*g(a+k2/2.0)
k4=h*g(a+k3)
t=t+h
a=a+(k1+2*k2+2*k3+k4)*(1/6.0)
write(*,*)t,a
END DO
END SUBROUTINE
!-------------------------
FUNCTION g(a)
implicit none
real a,g
g=sqrt((1.0/a)+(1.0/a**2))
END FUNCTION
The subroutine solves a differential equation and produces a for each t. I need to call the result of the subroutine in the main program and use a(t) in the main program. I wanted to define a(t) as an array but since t is real, I could not do it. Can anyone help me?
You have a system
R'(t)=f(a(t))
a'(t)=g(a(t))
that is semi-coupled. To integrate both functions together, use a coupled RK4 method of the principal form
rk4step(R,a,h)
k1a = h*g(a)
k1R = h*f(a)
k2a = h*g(a+k1a/2)
k2R = h*f(a+k1a/2)
k3a = h*g(a+k2a/2)
k3R = h*f(a+k2a/2)
k4a = h*g(a+k3a)
k4R = h*f(a+k3a)
a += (k1a+2*k2a+2*k3a+k4a)/6
R += (k1R+2*k2R+2*k3R+k4R)/6
return R,a
The better approach is to use vector-valued states and functions to avoid repetitions of similar steps.
I want to read a list of strings seperated by newlines from STDIN, until a new line is witnessed and I want an action of the type IO [String]. Here is how I would do it with recursion:
myReadList :: IO String
myReadList = go []
where
go :: [String] -> IO [String]
go l = do {
inp <- getLine;
if (inp == "") then
return l;
else go (inp:l);
}
However, this method of using go obscures readability and is a pattern so common that one would ideally want to abstract this out.
So, this was my attempt:
whileM :: (Monad m) => (a -> Bool) -> [m a] -> m [a]
whileM p [] = return []
whileM p (x:xs) = do
s <- x
if p s
then do
l <- whileM p xs
return (s:l)
else
return []
myReadList :: IO [String]
myReadList = whileM (/= "") (repeat getLine)
I am guessing there is some default implementation of this whileM or something similar already. However I cannot find it.
Could someone point out what is the most natural and elegant way to deal with this problem?
unfoldWhileM is same as your whileM except that it takes an action (not a list) as second argument.
myReadList = unfoldWhileM (/= "") getLine
Yes for abstracting out the explicit recursion as mentioned in the previous answer there is the Control.Monad.Loop library which is useful. For those who are interested here is a nice tutorial on Monad Loops.
However there is another way. Previously, struggling with this job and knowing that Haskell is by default Lazy i first tried;
(sequence . repeat $ getLine) >>= return . takeWhile (/="q")
I expected the above to collect entered lines into an IO [String] type. Nah... It runs indefinitely and IO actişons don't look lazy at all. At this point System IO Lazy might come handy too. It's a 2 function only simple library.
run :: T a -> IO a
interleave :: IO a -> T a
So run takes an Lazy IO action and turns it into an IO action and interleave does the opposite. Accordingly if we rephrase the above function as;
import qualified System.IO.Lazy as LIO
gls = LIO.run (sequence . repeat $ LIO.interleave getLine) >>= return . takeWhile (/="q")
Prelude> gls >>= return . sum . fmap (read :: String -> Int)
1
2
3
4
q
10
A solution using the effectful streams of the streaming package:
import Streaming
import qualified Streaming.Prelude as S
main :: IO ()
main = do
result <- S.toList_ . S.takeWhile (/="") . S.repeatM $ getLine
print result
A solution that shows prompts, keeping them separated from the reading actions:
main :: IO ()
main = do
result <- S.toList_
$ S.zipWith (\_ s -> s)
(S.repeatM $ putStrLn "Write something: ")
(S.takeWhile (/="") . S.repeatM $ getLine)
print result
I have converted around 90+ fortran files into C files using a tool and I need to validate that the conversion is good or not.
Can you give me some ideas on how best to ensure that the functionality has been preserved through the translation?
You need verification tests that exercise those fortran functions. Then you run those tests against the c code.
You can use unit test technology/methodology. In fact I can't see how else you would prove that the conversion is correct.
In lots of unit test methodologies you would write the tests in the same language as the code, but in this case I recommend very very strongly to pick one language and one code base to exercise both sets of functions. Also don't worry about be trying to create pure unit tests rather use the techniques to give you coverage of all the use that the fortran code was supposed to handle.
Use unit tests.
First write your unit tests on the Fortran code and check whether they all run correctly, then rewrite them in C and run those.
The problem in this approach is that you also need to rewrite your unit test, which you normally don't do when refactoring code (except for API changes). This means that you might end up debugging your ported unit testing code as well, beside the actual code.
Therefore, it might be better to write testing code that contains minimal logic and only write the results of the functions to a file. Then you can rewrite this minimal testing code in C, generate the same files and compare the files.
Here is what I did for a "similar" task (comparing fortran 90 to fortran 90 + OpenACC GPU accelerated code):
Analyze what's the output of each Fortran module.
Write these output arrays to .dat files.
Copy the .dat files into a reference folder.
Write the output of the converted modules to files (either CSV or binary). Use the same filename for convenience.
Make a python script that compares the two versions.
I used convenience functions like these in fortran (analogous for 1D, 2D case):
subroutine write3DToFile(path, array, n1, n2, n3)
use pp_vardef
use pp_service, only: find_new_mt
implicit none
!input arguments
real(kind = r_size), intent(in) :: array(n1,n2,n3)
character(len=*), intent(in) :: path
integer(4) :: n1
integer(4) :: n2
integer(4) :: n3
!temporary
integer(4) :: imt
call find_new_mt(imt)
open(imt, file = path, form = 'unformatted', status = 'replace')
write(imt) array
close(imt)
end subroutine write3DToFile
In python I used the following script for reading binary Fortran data and comparing it. Note: Since you want to convert to C you would have to adapt it such that you can read the data produced by C instead of Fortran.
from optparse import OptionParser
import struct
import sys
import math
def unpackNextRecord(file, readEndianFormat, numOfBytesPerValue):
header = file.read(4)
if (len(header) != 4):
#we have reached the end of the file
return None
headerFormat = '%si' %(readEndianFormat)
headerUnpacked = struct.unpack(headerFormat, header)
recordByteLength = headerUnpacked[0]
if (recordByteLength % numOfBytesPerValue != 0):
raise Exception, "Odd record length."
return None
recordLength = recordByteLength / numOfBytesPerValue
data = file.read(recordByteLength)
if (len(data) != recordByteLength):
raise Exception, "Could not read %i bytes as expected. Only %i bytes read." %(recordByteLength, len(data))
return None
trailer = file.read(4)
if (len(trailer) != 4):
raise Exception, "Could not read trailer."
return None
trailerUnpacked = struct.unpack(headerFormat, trailer)
redundantRecordLength = trailerUnpacked[0]
if (recordByteLength != redundantRecordLength):
raise Exception, "Header and trailer do not match."
return None
dataFormat = '%s%i%s' %(readEndianFormat, recordLength, typeSpecifier)
return struct.unpack(dataFormat, data)
def rootMeanSquareDeviation(tup, tupRef):
err = 0.0
i = 0
for val in tup:
err = err + (val - tupRef[i])**2
i = i + 1
return math.sqrt(err)
##################### MAIN ##############################
#get all program arguments
parser = OptionParser()
parser.add_option("-f", "--file", dest="inFile",
help="read from FILE", metavar="FILE", default="in.dat")
parser.add_option("--reference", dest="refFile",
help="reference FILE", metavar="FILE", default="ref.dat")
parser.add_option("-b", "--bytesPerValue", dest="bytes", default="4")
parser.add_option("-r", "--readEndian", dest="readEndian", default="big")
parser.add_option("-v", action="store_true", dest="verbose")
(options, args) = parser.parse_args()
numOfBytesPerValue = int(options.bytes)
if (numOfBytesPerValue != 4 and numOfBytesPerValue != 8):
print "Unsupported number of bytes per value specified."
sys.exit()
typeSpecifier = 'f'
if (numOfBytesPerValue == 8):
typeSpecifier = 'd'
readEndianFormat = '>'
if (options.readEndian == "little"):
readEndianFormat = '<'
inFile = None
refFile = None
try:
#prepare files
inFile = open(str(options.inFile),'r')
refFile = open(str(options.refFile),'r')
i = 0
while True:
passedStr = "pass"
i = i + 1
unpackedRef = None
try:
unpackedRef = unpackNextRecord(refFile, readEndianFormat, numOfBytesPerValue)
except(Exception), e:
print "Error reading record %i from %s: %s" %(i, str(options.refFile), e)
sys.exit()
if (unpackedRef == None):
break;
unpacked = None
try:
unpacked = unpackNextRecord(inFile, readEndianFormat, numOfBytesPerValue)
except(Exception), e:
print "Error reading record %i from %s: %s" %(i, str(options.inFile), e)
sys.exit()
if (unpacked == None):
print "Error in %s: Record expected, could not load record it" %(str(options.inFile))
sys.exit()
if (len(unpacked) != len(unpackedRef)):
print "Error in %s: Record %i does not have same length as reference" %(str(options.inFile), i)
sys.exit()
#analyse unpacked data
err = rootMeanSquareDeviation(unpacked, unpackedRef)
if (abs(err) > 1E-08):
passedStr = "FAIL <-------"
print "%s, record %i: Mean square error: %e; %s" %(options.inFile, i, err, passedStr)
if (options.verbose):
print unpacked
except(Exception), e:
print "Error: %s" %(e)
finally:
#cleanup
if inFile != None:
inFile.close()
if refFile != None:
refFile.close()
I am writing a daemon that reads something from a small file, modifies it, and writes it back to the same file. I need to make sure that each file is closed promptly after reading before I try to write to it. I also need to make sure each file is closed promptly after writing, because I might occasionally read from it again right away.
I have looked into using binary-strict instead of binary, but it seems that only provides a strict Get, not a strict Put. Same issue with System.IO.Strict. And from reading the binary-strict documentation, I'm not sure it really solves my problem of ensuring that files are promptly closed. What's the best way to handle this? DeepSeq?
Here's a highly simplified example that will give you an idea of the structure of my application. This example terminates with
*** Exception: test.dat: openBinaryFile: resource busy (file is locked)
for obvious reasons.
import Data.Binary ( Binary, encode, decode )
import Data.ByteString.Lazy as B ( readFile, writeFile )
import Codec.Compression.GZip ( compress, decompress )
encodeAndCompressFile :: Binary a => FilePath -> a -> IO ()
encodeAndCompressFile f = B.writeFile f . compress . encode
decodeAndDecompressFile :: Binary a => FilePath -> IO a
decodeAndDecompressFile f = return . decode . decompress =<< B.readFile f
main = do
let i = 0 :: Int
encodeAndCompressFile "test.dat" i
doStuff
doStuff = do
i <- decodeAndDecompressFile "test.dat" :: IO Int
print i
encodeAndCompressFile "test.dat" (i+1)
doStuff
All 'puts' or 'writes' to files are strict. The act of writeFile demands all Haskell data be evaluated in order to put it on disk.
So what you need to concentrate on is the lazy reading of the input. In your example above you both lazily read the file, then lazily decode it.
Instead, try reading the file strictly (e.g. with strict bytestrings), and you'll be fine.
Consider using a package such as conduit, pipes, iteratee or enumerator. They provide much of the benefits of lazy IO (simpler code, potentially smaller memory footprint) without the lazy IO. Here's an example using conduit and cereal:
import Data.Conduit
import Data.Conduit.Binary (sinkFile, sourceFile)
import Data.Conduit.Cereal (sinkGet, sourcePut)
import Data.Conduit.Zlib (gzip, ungzip)
import Data.Serialize (Serialize, get, put)
encodeAndCompressFile :: Serialize a => FilePath -> a -> IO ()
encodeAndCompressFile f v =
runResourceT $ sourcePut (put v) $$ gzip =$ sinkFile f
decodeAndDecompressFile :: Serialize a => FilePath -> IO a
decodeAndDecompressFile f = do
val <- runResourceT $ sourceFile f $$ ungzip =$ sinkGet get
case val of
Right v -> return v
Left err -> fail err
main = do
let i = 0 :: Int
encodeAndCompressFile "test.dat" i
doStuff
doStuff = do
i <- decodeAndDecompressFile "test.dat" :: IO Int
print i
encodeAndCompressFile "test.dat" (i+1)
doStuff
An alternative to using conduits et al. would be to just use System.IO, which will allow you to control explicitly when files are closed with respect to the IO execution order.
You can use openBinaryFile followed by normal reading operations (probably the ones from Data.ByteString) and hClose when you're done with it, or withBinaryFile, which closes the file automatically (but beware this sort of problem).
Whatever the method you use, as Don said, you probably want to read as a strict bytestring and then convert the strict to lazy afterwards with fromChunks.