setgid does not work on /tmp/ - c

Question

I have this C program:
#include <sys/stat.h>
#include <stdlib.h>
int main(void) {
if (chmod("/tmp/foo", 0755 | S_ISGID) < 0) {
exit(1);
}
exit(0);
}
When I run it like this:
rm -f /tmp/foo &&
touch /tmp/foo &&
./a.out &&
ls -al /tmp/foo &&
a.out runs with exit code 0 but the output is:
-rwxr-xr-x 1 philipp wheel 0 Mar 16 06:58 /tmp/foo
Why is the groupid flag not set here? The permissions should be -rwxr-sr-x.
The following things would fix the issue (but I still wonder why I see this effect):
running the program as root
running it in a different directory
running it on Linux
setting the set-user-id (S_ISUID)
I can swear it worked in an earlier version of OSX
What I tried but didn't work:
chmod g+s /tmp/foo also doesn't work
disabling csrutil did not change anything
altering the permissions on /tmp/ to something different, e.g. 0777 or 4777
So the question remains: what does make /tmp/ different from the other directories if it's not the permissions? The only difference I could see is:
ls -al /
showed tmp as this:
lrwxr-xr-x# 1 root wheel 11 Dec 11 19:28 tmp -> private/tmp
The # sign at the end shows that there are some non-unix properties set on the directory. Querying those with ls -l# /tmp shows:
lrwxr-xr-x# 1 root wheel 11 Dec 11 19:28 /tmp -> private/tmp
com.apple.FinderInfo 32
com.apple.rootless 0
Update: According to comment feedbacks and a downvote I figured the question is confusing, so I totally revised the question and the title. During revision I found out that I wrongly compared the effects of my program against chmod u+s which was wrong, I need to compare against chmod g+s, I also corrected this in my question.

Answer 1

The chmod() system call sets the permissions on a file to only the value you provide. This means that setting permissions to S_IRUSR | S_ISGID clears all other permissions, including user write and execute.
What you probably want is:
chmod("/tmp/foo", 0755 | S_ISGID);
(0755 being the octal mode for user read+write+execute and group/other read+execute -- it's a lot less typing than the equivalent constants.)