What is the best way to read multiple files into one? - file

Question

Here's a benchmark comparing two functions that read multiple files into a single one. The one uses read and the other uses read_to_end. My original motivation was to get the buffer's capacity be equal to the len at the end of the process. This did not happen with read_to_end which was quite unsatisfactory.
With read however, this works. The assert_eq!(buf.capacity(), buf.len()); of read_files_into_file2 (which uses read) does not panic.
use criterion::{criterion_group, criterion_main, Criterion};
use std::io::Read;
use std::io::Write;
use std::{
fs,
io::{self, Seek},
};
fn criterion_benchmark(c: &mut Criterion) {
let mut files = get_test_files().unwrap();
let mut file = fs::File::create("output").unwrap();
c.bench_function("1", |b| {
b.iter(|| {
read_files_into_file1(&mut files, &mut file).unwrap();
})
});
c.bench_function("2", |b| {
b.iter(|| {
read_files_into_file2(&mut files, &mut file).unwrap();
});
});
}
criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
/// Goes back to the start so that the files can be read again from the start.
fn reset(files: &mut Vec<fs::File>, file: &mut fs::File) {
file.seek(io::SeekFrom::Start(0)).unwrap();
for file in files {
file.seek(io::SeekFrom::Start(0)).unwrap();
}
}
pub fn read_files_into_file1(files: &mut Vec<fs::File>, file: &mut fs::File) -> io::Result<()> {
reset(files, file);
let total_len = files
.iter()
.map(|file| file.metadata().unwrap().len())
.sum::<u64>() as usize;
let mut buf = Vec::<u8>::with_capacity(total_len);
for file in files {
file.read_to_end(&mut buf)?;
}
file.write_all(&buf)?;
// assert_eq!(buf.capacity(), buf.len());
Ok(())
}
fn read_files_into_file2(files: &mut Vec<fs::File>, file: &mut fs::File) -> io::Result<()> {
reset(files, file);
let total_len = files
.iter()
.map(|file| file.metadata().unwrap().len())
.sum::<u64>() as usize;
let mut vec: Vec<u8> = vec![0; total_len];
let mut buf = &mut vec[..];
for file in files {
match file.read(&mut buf) {
Ok(n) => {
buf = &mut buf[n..];
}
Err(err) if err.kind() == io::ErrorKind::Interrupted => {}
Err(err) => return Err(err),
}
}
file.write_all(&vec)?;
// assert_eq!(vec.capacity(), vec.len());
Ok(())
}
/// Creates 5 files with content "hello world" 500 times.
fn get_test_files() -> io::Result<Vec<fs::File>> {
let mut files = Vec::<fs::File>::new();
for index in 0..5 {
let mut file = fs::OpenOptions::new()
.read(true)
.write(true)
.truncate(true)
.create(true)
.open(&format!("test{}", index))?;
file.write_all("hello world".repeat(500).as_bytes())?;
files.push(file);
}
Ok(files)
}
If you uncomment the assert_eq!s then you will see that only read_files_into_file1 (which uses read_to_end) fails with this panic:
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `55000`,
right: `27500`', benches/bench.rs:53:5
read_files_into_file1 allocates way more memory than needed while read_files_into_file2 allocates the optimal amount.
Despite that, the results say that they perform almost the same (read_files_into_file1 takes 11.439 us and read_files_into_file2 takes 11.098 us):
1 time: [11.417 us 11.439 us 11.463 us]
change: [+3.7987% +3.9997% +4.1984%] (p = 0.00 < 0.05)
Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
2 time: [11.085 us 11.098 us 11.112 us]
change: [+0.1255% +0.5081% +0.9545%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
I expect read_files_into_file2 to be much faster but it was even shown to be slower when I increased the file size. Why is it that read_files_into_file2 does not meet my expectations and what is the best way to read multiple files into one, efficiently?

Answer 1

read_to_end generally isn't a good idea when dealing with large files since it will try to read the whole file into memory which can lead to swapping or out of memory errors.
On linux and assuming single-threaded execution using io::copy should be the fastest method since it contains optimizations for this case.
On other platforms using io::copy and wrapping the writer side in a BufWriter lets you control the buffer size used for copying which will help amortizing syscall costs.
If you can use multiple threads and know that the file lengths don't change then you can use platform-specific positional read/write methods such as read_at to read multiple files in parallel and write the data into the correct places in the destination file. Whether this actually provides a speedup depends on many factors. It's probably most beneficial when concatenating many small files from a network filesystem.
Beyond the standard library there also are crates that expose platform-specific copy routines which may be faster than a naive userspace copy approach.