thread '<main>' has overflowed its stack when creating a large array - arrays

Question

static variable A_INTERSECTS_A from the following code returns the error.
This piece of code should return a big 1356x1356 2D array of bool.
use lazy_static::lazy_static; // 1.2.0
#[derive(Debug, Copy, Clone, Default)]
pub struct A {
pub field_a: [B; 2],
pub ordinal: i32,
}
#[derive(Debug, Copy, Clone, Default)]
pub struct B {
pub ordinal: i32,
}
pub const A_COUNT: i32 = 1356;
lazy_static! {
pub static ref A_VALUES: [A; A_COUNT as usize] = { [A::default(); A_COUNT as usize] };
pub static ref A_INTERSECTS_A: [[bool; A_COUNT as usize]; A_COUNT as usize] = {
let mut result = [[false; A_COUNT as usize]; A_COUNT as usize];
for item_one in A_VALUES.iter() {
for item_two in A_VALUES.iter() {
if item_one.field_a[0].ordinal == item_two.field_a[0].ordinal
|| item_one.field_a[0].ordinal == item_two.field_a[1].ordinal
|| item_one.field_a[1].ordinal == item_two.field_a[0].ordinal
|| item_one.field_a[1].ordinal == item_two.field_a[1].ordinal
{
result[item_one.ordinal as usize][item_two.ordinal as usize] = true;
}
}
}
result
};
}
fn main() {
A_INTERSECTS_A[1][1];
}
I've seen people dealing with this by implementing Drop for structs in a large list, but there aren't any structs in my list and you cant implement it for bool.
If I change A_INTERSECTS_A: [[bool; A_COUNT as usize]; A_COUNT as usize] to A_INTERSECTS_A: Box<Vec<Vec<bool>>> the code works fine, but I really would like to use an array here.

Answer 1

The problem here is almost certainly the huge result array that is being placed on the stack when the initialisation code of A_INTERSECTS_A runs. It is 13562 ≈ 1.8 MB, which is of a similar order of magnitude to the size of the stack. In fact, it is larger than Windows' default size of 1 MB (and I suspect you are on Windows, given you've got that error message).
The solution here is to reduce the stack size by moving it to the heap, by, for instance, using Vec instead (as you indicate works), or using a Box. This will have the added benefit that the initialisation code doesn't have to do a 2MB copy from the stack to A_INTERSECTS_A's memory (it only needs to copy some pointers around).
A direct translation to using a Box:
pub static ref A_INTERSECTS_A: Box<[[bool; A_COUNT as usize]; A_COUNT as usize]> = {
let mut result = Box::new([[false; A_COUNT as usize]; A_COUNT as usize]);
// ...
}
unfortunately doesn't work: Box::new is a normal function call, and hence its argument is placed directly onto the stack.
However, if you're using a nightly compiler and are willing to use unstable features, you can use "placement box", which is literally designed for this purpose: it allocates space on the heap and constructs the value straight into that memory, avoiding intermediate copies, and avoiding the need to have the data on the stack. This simply requires replacing Box::new with box:
let mut result = box [[false; A_COUNT as usize]; A_COUNT as usize];
If you (very sensibly) prefer to stick to stable releases, an alternative until that stabilises is to just replace the outer layer of the arrays with a Vec: this retains all the data locality benefits of the arrays (everything is laid out contiguously in memory), although is slightly weaker in terms of static knowledge (the compiler can't be sure that the length is 1356). Since [_; A_COUNT] doesn't implement Clone, this cannot use thevec!` macro and hence (unfortunately) looks like:
pub static ref A_INTERSECTS_A: Vec<[bool; A_COUNT as usize]> = {
let mut result =
(0..A_COUNT as usize)
.map(|_| [false; A_COUNT as usize])
.collect::<Vec<_>>();
// ...
}
If you absolutely need all the arrays, one could do some unsafe magic to extract this down to the original Box<[[bool; ...]; ...]> from the Vec. It requires two steps (via into_boxed_slice), because a Box<T> needs to have an allocation sized perfectly for T, while a Vec may overallocate in order to achieve its O(1) amortization. This version would look like:
pub static ref A_INTERSECTS_A: Box<[[bool; A_COUNT as usize]; A_COUNT as usize]> = {
let mut result =
(0..A_COUNT as usize)
.map(|_| [false; A_COUNT as usize])
.collect::<Vec<_>>();
// ...
// ensure the allocation is correctly sized
let mut slice: Box<[[bool; A_COUNT as usize]]> = result.into_boxed_slice();
// pointer to the start of the slices in memory
let ptr: *mut [bool; A_COUNT as usize] = slice.as_mut_ptr();
// stop `slice`'s destructor deallocating the memory
mem::forget(slice);
// `ptr` is actually a pointer to exactly A_COUNT of the arrays!
let new_ptr = ptr as *mut [[bool; A_COUNT as usize]; A_COUNT as usize];
unsafe {
// let this `Box` manage that memory
Box::from_raw(new_ptr)
}
}
I've added in some explicit types so that what's going in is a little more clear. This works because Vec<T> exposes into_boxed_slice, and hence we can munge that Box<[T]> (i.e. dynamic length) into a Box<[T; len]> given we know the exact length at compile time.