Difference between two buffers in C/C++ - c

Question

This development is being done on Windows in usermode.
I have two (potentially quite large) buffers, and I would like to know the number of bytes different between the two of them.
I wrote this myself just checking byte by byte, but this resulted in a quite slow implementation. As I'm comparing on the order of hundreds of megabytes, this is undesirable. I'm aware that I could optimize this though many different means, but this seems like a common problem that's probably got optimized solutions already out there, and there's no way I'm going to optimize this as effectively as if it was written by optimization experts.
Perhaps my Googling is inadequate, but I'm unable to find any other C or C++ functions that can count the number of different bytes between two buffers. Is there such a built in function to the C standard library, WinAPI, or C++ standard library that I just don't know of? Or do I need to manually optimize this?

Answer 1

I ended up writing this (perhaps somewhat poorly) optimized code to do the job for me. I was hoping it would vectorize this under the hood, but that doesn't appear to be happening unfortunately, and I didn't feel like digging around the SIMD intrinsics to do it manually. As a result, my bit fiddling tricks may end up making it slower, but it's still fast enough that it's no more than about 4% of my code's runtime (and almost all of that was memcmp). Whether or not it could be better, it's good enough for me.
I'll note that this is designed to be fast for my use case, where I'm expecting only rare differences.
inline size_t ComputeDifferenceSmall(
_In_reads_bytes_(size) char* buf1,
_In_reads_bytes_(size) char* buf2,
size_t size) {
/* size should be <= 0x1000 bytes */
/* In my case, I expect frequent differences if any at all are present. */
size_t res = 0;
for (size_t i = 0; i < (size & ~0xF); i += 0x10) {
uint64_t diff1 = *reinterpret_cast<uint64_t*>(buf1) ^
*reinterpret_cast<uint64_t*>(buf2);
if (!diff1) continue;
/* Bit fiddle to make each byte 1 if they're different and 0 if the same */
diff1 = ((diff1 & 0xF0F0F0F0F0F0F0F0ULL) >> 4) | (diff1 & 0x0F0F0F0F0F0F0F0FULL);
diff1 = ((diff1 & 0x0C0C0C0C0C0C0C0CULL) >> 2) | (diff1 & 0x0303030303030303ULL);
diff1 = ((diff1 & 0x0202020202020202ULL) >> 1) | (diff1 & 0x0101010101010101ULL);
/* Sum the bytes */
diff1 = (diff1 >> 32) + (diff1 & 0xFFFFFFFFULL);
diff1 = (diff1 >> 16) + (diff1 & 0xFFFFULL);
diff1 = (diff1 >> 8) + (diff1 & 0xFFULL);
diff1 = (diff1 >> 4) + (diff1 & 0xFULL);
res += diff1;
}
for (size_t i = (size & ~0xF); i < size; i++) {
res += (buf1[i] != buf2[i]);
}
return res;
}
size_t ComputeDifference(
_In_reads_bytes_(size) char* buf1,
_In_reads_bytes_(size) char* buf2,
size_t size) {
size_t res = 0;
/* I expect most pages to be identical, and both buffers should be page aligned if
* larger than a page. memcmp has more optimizations than I'll ever come up with,
* so I can just use that to determine if I need to check for differences
* in the page. */
for (size_t pn = 0; pn < (size & ~0xFFF); pn += 0x1000) {
if (memcmp(&buf1[pn], &buf2[pn], 0x1000)) {
res += ComputeDifferenceSmall(&buf1[pn], &buf2[pn], 0x1000);
}
}
return res + ComputeDifferenceSmall(
&buf1[size & ~0xFFF], &buf2[size & ~0xFFF], size & 0xFFF);
}