publishing messages to GCP pubSub topic fails - google-app-engine

Question

I have a Spring Boot application which tries to publishes messages to a Cloud Pub/Sub topic. When I run the application in local machine, it successfully publishes the messages into the topic. But after deploying it to App Engine it fails with the below exception
com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:69) ~[gax-1.16.0.jar:1.16.0]
at com.google.cloud.pubsub.v1.Publisher$4.onFailure(Publisher.java:373) ~[google-cloud-pubsub-0.33.0-beta.jar:0.33.0-beta]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) [guava-20.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) [guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) [guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) [guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) [guava-20.0.jar:na]
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:492) [grpc-stub-1.9.0.jar:1.9.0]
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:467) [grpc-stub-1.9.0.jar:1.9.0]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:392) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.9.0.jar:1.9.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.9.0.jar:1.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112-google-v7]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_112-google-v7]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_112-google-v7]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:295) [na:1.8.0_112-google-v7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112-google-v7]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112-google-v7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112-google-v7]
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata
at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-core-1.9.0.jar:1.9.0]
... 19 common frames omitted
Caused by: java.io.IOException: Could not get the access token.
at com.google.auth.oauth2.AppEngineCredentials.refreshAccessToken(AppEngineCredentials.java:136) ~[google-auth-library-oauth2-http-0.7.1.jar:na]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149) ~[google-auth-library-oauth2-http-0.7.1.jar:na]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135) ~[google-auth-library-oauth2-http-0.7.1.jar:na]
at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:103) ~[google-auth-library-credentials-0.9.0.jar:na]
at com.google.auth.Credentials$1.run(Credentials.java:92) ~[google-auth-library-credentials-0.9.0.jar:na]
... 7 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112-google-v7]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112-google-v7]
at com.google.auth.oauth2.AppEngineCredentials.refreshAccessToken(AppEngineCredentials.java:131) ~[google-auth-library-oauth2-http-0.7.1.jar:na]
... 11 common frames omitted
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call memcache.Get in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:844) ~[runtime-shared.jar:na]
at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:183) ~[runtime-shared.jar:na]
at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:180) ~[runtime-shared.jar:na]
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93) ~[appengine-api.jar:na]
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28) ~[appengine-api.jar:na]
at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java:51) ~[appengine-api.jar:na]
at com.google.appengine.api.appidentity.AppIdentityServiceImpl.getAccessToken(AppIdentityServiceImpl.java:302) ~[appengine-api.jar:na]
... 15 common frames omitted
Code is copied from this Spring guide.

Answer 1

The error message you shared clearly states UNAVAILABLE: Credentials failed to obtain metadata, so it looks like the error is related to credentials in App Engine. Apparently, and according to your description of the issue, your application is working properly when being run in the local development server, but once you deploy it to your App Engine Standard application, it fails. It all indicates that you may have a faulty credentials configuration.
As explained in the Authentication section in the guide you shared, and also in App Engine's documentation, App Engine Standard uses the Application Default Credentials for authentication. To do so, it checks for credentials in the environment variable GOOGLE_APPLICATION_CREDENTIALS, and if they are missing, it will use the default service account from your App Engine application.
Knowing that, your possible issue is that you are pointing to a credentials file that is not available in the remote (App Engine) environment. Therefore, my recommendation is that, if you think you can work with the default App Engine service account, you just leave the GOOGLE_APPLICATION_CREDENTIALS environment variable and the spring.cloud.gcp.credentials.location property empty so that App Engine takes the default credentials from its own service account. Bear in mind that it should have permissions to interact with Pub/Sub in order for it to work too, so you may have to grant roles to the service account if needed.