Autofixture: Issue when creating value types - autofixture

Question

Debugging with xUnit.net the test-methods Test1 and Test2 of the following code and putting a breakpoint and the end of CreateValueAndReferenceType() you see that the variable valueType is the same in both runs, whereas the variable referenceType is altered. The former is for me surprising and an issue as well (I added the row with the string-type only for completeness).
public class MyFixture : Fixture
{
public void CreateValueAndReferenceType()
{
var valueType = this.Create<int>();
var referenceTye = this.Create<string>();
}
}
public class TestClass1
{
[Fact]
public void Test1()
{
var myFixture = new MyFixture();
myFixture.CreateValueAndReferenceType();
}
}
public class TestClass2
{
[Fact]
public void Test2()
{
var myFixture = new MyFixture();
myFixture.CreateValueAndReferenceType();
}
}

Answer 1

What you're seeing is, I think, a basic issue related to pseudo-random number generation in .NET (IIRC, other platforms have similar issues). In essence, System.Random is deterministic, but initialised with a random seed that, among other things, depend on the computer's current time. If you create instances of Random in a tight loop, the code executes faster than the precision of the system clock. Something like this:
for (int i = 0; i < 10; i++)
Console.Write(new Random().Next(0, 9));
will often produce output like this:
5555555555
Most values in AutoFixture are generated by various Random instances - the exception is the string type, of which values are generated by Guid.NewGuid().ToString().
I think that the reason you're seeing this is because of xUnit.net's parallel execution.
In order to pinpoint the problem, I rephrased the issue so that it doesn't rely on debugging or inheritance:
public static class Reporter
{
public static void CreateValueAndReferenceType(
IFixture fixture,
ITestOutputHelper #out)
{
var valueType = fixture.Create<int>();
var referenceTye = fixture.Create<string>();
#out.WriteLine("valueType: {0}", valueType);
#out.WriteLine("referenceType: {0}", referenceTye);
}
}
public class TestClass1
{
private readonly ITestOutputHelper #out;
public TestClass1(ITestOutputHelper #out)
{
this.#out = #out;
}
[Fact]
public void Test1()
{
Reporter.CreateValueAndReferenceType(new Fixture(), this.#out);
}
}
public class TestClass2
{
private readonly ITestOutputHelper #out;
public TestClass2(ITestOutputHelper #out)
{
this.#out = #out;
}
[Fact]
public void Test2()
{
Reporter.CreateValueAndReferenceType(new Fixture(), this.#out);
}
}
When you run this with the xUnit.net console runner, you can see the issue nicely reproduced:
$ packages/xunit.runner.console.2.1.0/tools/xunit.console 37925109/bin/Debug/Ploeh.StackOverflow.Q37925109.dll -diagnostics
-parallel all
xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
Discovering: Ploeh.StackOverflow.Q37925109 (app domain = on [shadow copy], method display = ClassAndMethod)
Discovered: Ploeh.StackOverflow.Q37925109 (running 2 test cases)
Starting: Ploeh.StackOverflow.Q37925109 (parallel test collections = on, max threads = 4)
Ploeh.StackOverflow.Q37925109.TestClass2.Test2 [PASS]
Output:
valueType: 246
referenceType: cc39f570-046a-4a0a-8adf-ab7deadd0e26
Ploeh.StackOverflow.Q37925109.TestClass1.Test1 [PASS]
Output:
valueType: 246
referenceType: 87455351-03f7-4640-99fb-05af910da267
Finished: Ploeh.StackOverflow.Q37925109
=== TEST EXECUTION SUMMARY ===
Ploeh.StackOverflow.Q37925109 Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0,429s
In the above example, you'll notice that I've explicitly invoked the runner with -parallel all, but I didn't have to do that, since it's the default.
If, on the other hand, you turn off parallelisation with -parallel none, you'll see that the values are different:
$ packages/xunit.runner.console.2.1.0/tools/xunit.console 37925109/bin/Debug/Ploeh.StackOverflow.Q37925109.dll -diagnostics
-parallel none
xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
Discovering: Ploeh.StackOverflow.Q37925109 (app domain = on [shadow copy], method display = ClassAndMethod)
Discovered: Ploeh.StackOverflow.Q37925109 (running 2 test cases)
Starting: Ploeh.StackOverflow.Q37925109 (parallel test collections = off, max threads = 4)
Ploeh.StackOverflow.Q37925109.TestClass2.Test2 [PASS]
Output:
valueType: 203
referenceType: 1bc75a33-5542-4d9f-b42d-57ed85dc418d
Ploeh.StackOverflow.Q37925109.TestClass1.Test1 [PASS]
Output:
valueType: 117
referenceType: 6a508699-dc35-4bcd-8a7b-15eba64b24b4
Finished: Ploeh.StackOverflow.Q37925109
=== TEST EXECUTION SUMMARY ===
Ploeh.StackOverflow.Q37925109 Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0,348s
What I think happens is that because of the parallelism, both Test1 and Test2 are executed in parallel, and essentially within the same tick.
One workaround is to place both tests in the same test class:
public class TestClass1
{
private readonly ITestOutputHelper #out;
public TestClass1(ITestOutputHelper #out)
{
this.#out = #out;
}
[Fact]
public void Test1()
{
Reporter.CreateValueAndReferenceType(new Fixture(), this.#out);
}
[Fact]
public void Test2()
{
Reporter.CreateValueAndReferenceType(new Fixture(), this.#out);
}
}
This produces two different integer values, because (IIRC) xUnit.net only run different test classes in parallel:
$ packages/xunit.runner.console.2.1.0/tools/xunit.console 37925109/bin/Debug/Ploeh.StackOverflow.Q37925109.dll -diagnostics
-parallel all
xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
Discovering: Ploeh.StackOverflow.Q37925109 (app domain = on [shadow copy], method display = ClassAndMethod)
Discovered: Ploeh.StackOverflow.Q37925109 (running 2 test cases)
Starting: Ploeh.StackOverflow.Q37925109 (parallel test collections = on, max threads = 4)
Ploeh.StackOverflow.Q37925109.TestClass1.Test2 [PASS]
Output:
valueType: 113
referenceType: e8c30ad8-f2c8-4767-9e9f-69b55c50e659
Ploeh.StackOverflow.Q37925109.TestClass1.Test1 [PASS]
Output:
valueType: 232
referenceType: 3eb60bf3-4d43-4a91-aef2-42f7e23e35b3
Finished: Ploeh.StackOverflow.Q37925109
=== TEST EXECUTION SUMMARY ===
Ploeh.StackOverflow.Q37925109 Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0,360s
This theory is also corroborated by the fact that if you repeat the experiment sufficiently many times, you'll see the numbers being different once in a while. Here are the integer results from 25 test runs:
33 33
92 92
211 211
13 13
9 9
160 160
55 55
155 155
137 137
161 161
242 242
183 183
237 237
151 151
104 104
254 254
123 123
244 244
144 144
223 9
196 196
126 126
199 199
221 221
132 132
Notice that all except one test run has equal numbers.