Broadcasting each element of a SIMD register in a loop - c

Question

I need to fill a SIMD register with one element of another SIMD register. i.e. "broadcast" or "splat" a single element to every position.
My current code for doing it is (it's simplified, my real functions are declared inline):
__m128
f4_broadcast_1(__m128 a, int i) {
return _mm_set1_ps(a[i]);
}
This seem to generate efficient code on clang and gcc, but msvc forbids index accesses. Therefore, I instead write:
__m128
f4_broadcast_2(__m128 a, int i) {
union { __m128 reg; float f[4]; } r = { .reg = a };
return _mm_set1_ps(r.f[i]);
}
It generates the same code on clang and gcc but bad code on msvc. Godbolt link: https://godbolt.org/z/IlOqZl
Is there a better way to do it? I know there are similar questions on SO already, but my use case involves both extracting a float32 from a register and putting it back into another one, which is a slightly different problem. It would be cool if you could do this without having to touch the main memory at all.
Is the index variable or constant? Apparently it matters a lot to SIMD performance whether it is. In my case, the index is a loop variable:
for (int i = 0; i < M; i++) {
... broadcast element i of some reg
}
where M is either 4, 8 or 16. Maybe I should manually unroll the loops to make it a constant? It's a lot of code in the for-loop so the amount of code would grow considerably.
I also wonder how to do the same thing but for the __m256 and __m512 registers found on modern cpu:s.

Answer 1

Some of the shuffles in Get an arbitrary float from a simd register at runtime? can be adapted to broadcast an element instead of just get 1 copy if it to the low element. It discusses tradeoffs of shuffle vs. store/reload strategies in more detail.
x86 doesn't have a 32-bit-element variable-control shuffle until AVX vpermilps and AVX2 lane-crossing vpermps / vpermd. e.g.
// for runtime-variable i. Otherwise use something more efficient.
_mm_permutevar_ps(v, _mm_set1_epi32(i));
Or broadcast the low element with vbroadcastss (the vector-source version requires AVX2)
Broadcast loads are very efficient with AVX1: _mm_broadcast_ss(float*) (or _mm256/512 of the same) or simply 128/256/512 _mm_set1_ps(float) of a float that happened to come from memory, and let your compiler use a broadcast load if compiling with AVX1 enabled.
With a compile-time-constant control, you can broadcast any single element with SSE1
_mm_shuffle_ps(same,same, _MM_SHUFFLE(i,i,i,i));
Or for integer, with SSE2 pshufd: _mm_shuffle_epi32(v, _MM_SHUFFLE(i,i,i,i)).
Depending on your compiler, it may have to be a macro for i to be a compile-time constant with optimization disabled. The shuffle-control constant has to compile into an immediate byte (with 4x 2-bit fields) embedded in the machine code, not loaded as data or from a register.
Iterating over elements in a loop.
I'm using AVX2 in this section; this easily adapts to AVX512. Without AVX2 the store/reload strategy is your only good option for 256-bit vectors, or vpermilps for 128-bit vectors.
Possibly incrementing counters (by 4) for SSSE3 pshufb (with casting between __m128i and __m128) `could be a good idea without AVX where you don't have an efficient broadcast load.
the index is a loop variable
Compilers will often fully unroll loops for you, turning the loop variable into a compile-time constant for each iteration. But only with optimization enabled. In C++ you could maybe use template recursion to iterate with a constexpr.
MSVC doesn't optimize intrinsics, so if you write _mm_permutevar_ps(v, _mm_set1_epi32(i)); you're actually going to get that in each iteration, not 4x vshufps. But gcc and especially clang do optimize shuffles, so they should do well with optimization enabled.
It's a lot of code in the for-loop
If it's going to need a lot of registers / spend a lot of time, a store/reload might be a good choice especially with AVX available for broadcast reloads. Shuffle throughput is more limited (1/clock) than load throughput (2/clock) on current Intel CPUs.
Compiling your code with AVX512 will even allow broadcast memory-source operands, not a separate load instruction, so the compiler can even fold a broadcast-load into a source operand if it's only needed once.
/********* Store/reload strategy ****************/
#include <stdalign.h>
void foo(__m256 v) {
alignas(32) float tmp[8];
_mm256_store_ps(tmp, v);
// with only AVX1, maybe don't peel first iteration, or broadcast manually in 2 steps
__m256 bcast = _mm256_broadcastss_ps(_mm256_castps256_ps128(v)); // AVX2 vbroadcastss ymm, xmm
... do stuff with bcast ...
for (int i=1; i<8 ; i++) {
bcast = _mm256_broadcast_ss(tmp[i]);
... do stuff with bcast ...
}
}
I peeled the first iteration manually to just broadcast the low element with an ALU operation (lower latency) so it can get started right away. Later iterations then reload with a broadcast load.
Another option would be to use a SIMD increment for a vector shuffle-control (aka mask), if you have AVX2.
// Also AVX2
void foo(__m256 v) {
__m256i shufmask = _mm256_setzero_si256();
for (int i=1; i<8 ; i++) {
__m256 bcast = _mm256_permutevar8x32_ps(v, shufmask); // AVX2 vpermps
// prep for next iteration by incrementing the element selectors
shufmask = _mm256_add_epi32(shufmask, _mm256_set1_epi32(1));
... do stuff with bcast ...
}
}
This does one redundant vpaddd on shufmask (in the last iteration), but that's probably fine and better than peeling the first or last iteration. And obviously better than starting with -1 and doing an add before the shuffle in the first iteration.
Lane-crossing shuffles have 3-cycle latency on Intel so putting it right after the shuffle is probably good scheduling unless there's other per-iteration work that doesn't depend on bcast; out-of-order exec makes this a minor issue anyway. In the first iteration, vpermps with a mask that was just xor-zeroed is basically just as good as vbroadcastss on Intel, for out-of-order exec to get started quickly.
But on AMD CPUs (at least before Zen2), lane-crossing vpermps is pretty slow; lane-crossing shuffles with granularity <128-bit are extra expensive because it has to decode into 128-bit uops. So this strategy isn't wonderful on AMD. If store/reload performs equally for your surrounding code on Intel, then it might be a better choice to make your code AMD-friendly as well.
vpermps also has a new intrinsic introduced with AVX512 intrinsics: _mm256_permutexvar_ps(__m256i idx, __m256 a) which has the operands in the order that matches asm. Use whichever one you like, if your compiler supports the new one.

Answer 2

Broadcasting can be achieved by using the AVX2 instruction VBROADCASTSS, but moving the value to the input position (first position) depends on your instruction set:
VBROADCASTSS (128 bit version VEX and legacy)
This instruction broadcasts the source value on position [0] of the source XMM register to all four FLOATS of the destination XMM register. Its intrinsic is __m128 _mm_broadcastss_ps(__m128 a);.
If the position of your value is constant, you can use the instruction PSHUFD to move the value from its current position to the first position. Its intrinsic is __m128i _mm_shuffle_epi32(__m128i a, int n). To move the value that should be broadcasted to the first position of the input XMM vector, use the following values for int n:
1. : 0h
2. : 1h
3. : 2h
4. : 3h
This moves the value from the 0..3 position to the first position.
So use, for example, use the following to move the fourth position of the input vector to the first one:
__m128 newInput = _mm_shuffle_epi32(__m128i input, 3)
Then apply the following intrinsic:
__m128 result = _mm_broadcastss_ps(__m128 newInput);
Now the value from the fourth position of your input XMM vector should be on all positions of your result vector.