Intermittent permissions issues publishing to pub/sub - google-cloud-pubsub

Question

We are getting intermittent permissions issues publishing from a GCP Kubernetes container to a GCP pub/sub. In one process/pid, we'll see ~50 messages go through, a few seconds of The request is missing a valid API key followed by ~50 messages going through. The full error message looks like:
{
"error": {
"code": 403,
"message": "The request is missing a valid API key.",
"status": "PERMISSION_DENIED"
}
}
Our setup:
We setup a Workload Identity User for our Kubernetes pod.
We linked that Workload Identity User to a service account.
We gave that service account publishing rights on the pub/sub topic.
We are using PHP with the vendor package google/cloud-pubsub version 1.30.2
I'd say 99% of our messages go through. But a few times a day, we get outages where the messages do not go through for a few seconds with the permissions error, and then publishing starts working again a few seconds later.
I saw another post suggesting perhaps GOOGLE_APPLICATION_CREDENTIALS could be getting in the way. We verified we are not setting GOOGLE_APPLICATION_CREDENTIALS as we are relying on the service account to give the docker containers publish access to the pub/sub topic.
An early thought experiment was suspicious of a race condition with docker images coming online/offline. But given we see a few seconds of permissions issues in a long running process makes us think it's likely something else. Does anyone have any ideas or things we should look at?
We appreciate the help!
UPDATE 2021-09-28:
Here are our logs from yesterday on when and how many permissions errors we saw against when and how many publishes went through. This is showing how many events happened per hour. This came from one docker instance:
Permissions issues:
fgrep 'The request is missing a valid API key' /log/file/was/here.log | perl -ne '/(Sep 27 [0-9]{2}):.*/; print $1 . "\n"' | uniq -c
38 Sep 27 06
90 Sep 27 07
176 Sep 27 17
186 Sep 27 18
54 Sep 27 19
No permissions issues:
fgrep 'published index requeue' /log/file/was/here.log | perl -ne '/(Sep 27 [0-9]{2}):.*/; print $1 . "\n"' | uniq -c
571 Sep 27 00
900 Sep 27 01
1117 Sep 27 02
562 Sep 27 03
3396 Sep 27 04
1767 Sep 27 05
4568 Sep 27 06
3857 Sep 27 07
2160 Sep 27 08
74 Sep 27 09
218 Sep 27 10
55 Sep 27 11
1992 Sep 27 12
4376 Sep 27 13
482 Sep 27 14
839 Sep 27 15
3533 Sep 27 16
8903 Sep 27 17
10067 Sep 27 18
9098 Sep 27 19
1006 Sep 27 20
2932 Sep 27 21
919 Sep 27 22
1104 Sep 27 23
The sum of these comes to 544 failed and 64496 success which is a failure rate of about .84% when averaged over the whole day. If looking at an hour we saw failures such as 1800Z, we see 186 failed and 10067 success or a failure rate of 1.8% for the 1800Z hour
UPDATE 2021-09-28 part 2:
Just dove into one of those hours we saw failures and grouped it down to the minute.
Permissions issues in the 1800Z hour:
fgrep 'The request is missing a valid API key' /log/file/was/here.log | perl -ne '/(Sep 27 18:[0-9]{2}):.*/; print $1 . "\n"' | uniq -c
4 Sep 27 18:01
2 Sep 27 18:02
2 Sep 27 18:03
2 Sep 27 18:04
2 Sep 27 18:05
16 Sep 27 18:10
6 Sep 27 18:12
6 Sep 27 18:16
2 Sep 27 18:20
4 Sep 27 18:21
2 Sep 27 18:30
2 Sep 27 18:32
2 Sep 27 18:33
2 Sep 27 18:34
14 Sep 27 18:35
8 Sep 27 18:36
8 Sep 27 18:37
2 Sep 27 18:38
2 Sep 27 18:40
8 Sep 27 18:41
8 Sep 27 18:42
8 Sep 27 18:43
10 Sep 27 18:44
10 Sep 27 18:45
38 Sep 27 18:46
4 Sep 27 18:47
8 Sep 27 18:51
4 Sep 27 18:58
No permissions issues in the 1800Z hour:
fgrep 'published index requeue' /log/file/was/here.log | perl -ne '/(Sep 27 18:[0-9]{2}):.*/; print $1 . "\n"' | uniq -c
249 Sep 27 18:00
187 Sep 27 18:01
156 Sep 27 18:02
266 Sep 27 18:03
302 Sep 27 18:04
149 Sep 27 18:05
29 Sep 27 18:06
572 Sep 27 18:07
499 Sep 27 18:08
395 Sep 27 18:09
189 Sep 27 18:10
257 Sep 27 18:11
194 Sep 27 18:12
185 Sep 27 18:13
94 Sep 27 18:14
49 Sep 27 18:15
71 Sep 27 18:16
77 Sep 27 18:17
36 Sep 27 18:18
34 Sep 27 18:19
73 Sep 27 18:20
114 Sep 27 18:21
82 Sep 27 18:22
87 Sep 27 18:23
20 Sep 27 18:24
39 Sep 27 18:25
56 Sep 27 18:26
31 Sep 27 18:27
79 Sep 27 18:28
82 Sep 27 18:29
270 Sep 27 18:30
32 Sep 27 18:31
174 Sep 27 18:32
165 Sep 27 18:33
91 Sep 27 18:34
213 Sep 27 18:35
243 Sep 27 18:36
208 Sep 27 18:37
92 Sep 27 18:38
76 Sep 27 18:39
211 Sep 27 18:40
269 Sep 27 18:41
164 Sep 27 18:42
322 Sep 27 18:43
228 Sep 27 18:44
294 Sep 27 18:45
242 Sep 27 18:46
318 Sep 27 18:47
145 Sep 27 18:48
42 Sep 27 18:49
115 Sep 27 18:50
206 Sep 27 18:51
214 Sep 27 18:52
213 Sep 27 18:53
169 Sep 27 18:54
169 Sep 27 18:55
208 Sep 27 18:56
190 Sep 27 18:57
88 Sep 27 18:58
43 Sep 27 18:59