Opening files in a parallel loop crash the program - file

Question

I'm making a simple program in D that loops around the files present inside a given folder and calculate the MD5 of the file. I have got this program working easily with no problem.
I recently learned more about parallelism and thought that my little program would benefit greatly from it.
I changed the loop around the files to a parallel one, but now it is working weird.
The program starts reading files and calculating their MD5, but sometimes, executing close(file) throw an exception with error code 0. Sometimes, there is just an
Object.Error: Access violation
Sometimes the program just freezes, sometimes it simply crashes.
When I remove the file opening (the loop then basically just print the filename to the console) it works well.
The time it takes for the program to crash is maybe linked to the size of workUnitSize. If not set (default 100), it crashes or stop working after processing around 30-40 files. If I set it to 1, it stops after 2 to 10 files.
Here is the smallest reproducible code I've extracted :
import std.md5;
import std.stdio;
import std.file;
import std.conv;
import std.getopt;
import std.string;
import std.process;
import std.parallelism;
import std.exception;
struct Entry
{
string name;
ubyte[16] md5;
}
int ChunkStep = 4096;
void main(string[] args)
{
string folder1 = args[1];
string folder2 = args[2];
Entry[] entries1;
foreach (name; parallel(dirEntries(folder1, SpanMode.breadth), 1)) // not working
{
if(name.isFile())
{
entries1 ~= Entry(name ,mdFile(name));
}
}
writeln(folder1," has ",entries1.length, " entries");
Entry[] entries2;
foreach (string name; dirEntries(folder2, SpanMode.breadth)) //working fine
{
if(name.isFile())
{
entries2 ~= Entry(name ,mdFile(name));
}
}
writeln(folder2," has ", entries2.length, " entries");
}
/// Digests a file and prints the result.
ubyte[16] mdFile(string filename)
{
MD5_CTX context;
ubyte[16] digest;
context.start();
File f = File(filename,"r");
foreach (buffer; f.byChunk(ChunkStep))
context.update(buffer);
context.finish(digest);
try{
f.close();
}
catch(ErrnoException e)
{
writeln(e.errno);
}
writefln("MD5 (%s) = %s", filename, digestToString(digest));
return digest;
}
I'm using Dmd2.064.2 on Windows 7 64b with a Intel Q6600 (quad core).

Answer 1

Do you use 32bit version of dmd?
update:
seems on windows there is some bug with byChunk (rawRead). So you can use std.stream instead.
import std.md5;
import std.stdio;
import std.file;
import std.conv;
import std.getopt;
import std.string;
static import std.stream;
import std.process;
import std.parallelism;
import std.exception;
struct Entry
{
string name;
ubyte[16] md5;
}
enum ChunkStep = 4096;
void main(string[] args)
{
string folder1 = args[1];
string folder2 = args[2];
Entry[] entries1;
foreach (name; parallel(dirEntries(folder1, SpanMode.breadth), 1)) // not working
{
if(name.isFile())
{
entries1 ~= Entry(name ,mdFile(name));
}
}
writeln(folder1," has ",entries1.length, " entries");
}
/// Digests a file and prints the result.
ubyte[16] mdFile(string filename)
{
MD5_CTX context;
ubyte[16] digest;
context.start();
std.stream.File f = new std.stream.File(filename);
byte[ChunkStep] buffer;
size_t len;
while((len = f.readBlock(buffer.ptr, ChunkStep)) > 0) {
context.update(buffer.ptr[0 .. len]);
}
context.finish(digest);
f.close();
writefln("MD5 (%s) = %s", filename, digestToString(digest));
return digest;
}