I have a Google Appengine app (Java) and the datastore was recently auto-migrated into "Cloud Firestore in Datastore mode" (completed on 6/6/2022) that has been running for years.
It has now started throwing these exceptions in new deployments without any relevant code change to trigger them:
com.google.appengine.api.datastore.DatastoreFailureException: Missing or invalid authentication.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:69)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:127)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:68)
at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
at com.google.appengine.api.datastore.BaseQueryResultsSource.getIndexList(BaseQueryResultsSource.java:154)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:187)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:166)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:146)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:64)
at com.googlecode.objectify.impl.KeysOnlyIterator.hasNext(KeysOnlyIterator.java:29)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:48)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:20)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:52)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1105)
at com.googlecode.objectify.impl.ChunkingIterator.hasNext(ChunkingIterator.java:51)
at com.google.common.collect.Iterators.addAll(Iterators.java:366)
at com.google.common.collect.Lists.newArrayList(Lists.java:163)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:34)
at com.sun.proxy.$Proxy68.isEmpty(Unknown Source)
My code. After loading a list of entities, the exception is thrown on the isEmpty() call on the returned list:
List<SystemSetting> applicationSettings = ofy().load().type(SystemSetting.class).filter("name", name).list();
if (applicationSettings.isEmpty()) { // Exception thrown here
It's worth also noting that on previous versions, datastore reads and writes are still happening without any issues. It is only new deployments that exhibit this issue. However, redeploying previous versions from previous commits using the same CI service to deploy them are failing.
Is this caused by the datastore migration?
Slight update: these errors are only happening on newly deployed versions... In fact, I deployed a new version (from CI, bitbucket pipeline) that worked, but when I redeployed it with no changes it no longer works. Same app engine version id, code and users etc.
Things we've tried without success:
Re-deploying last known version
Clearing memcache
Deploying from local dev
Stopping instances
Trying alternative service account key
Expanding service account permissions
Clearing BitBucket pipeline caches (the CI builder)
Have recreated the issue with datastore api directly (no objectify)
Have redeployed with different user account - no success
Have redeployed with new service account - no success
Update 1:
If I run the following code on a local java application (pointing at live datastore using same service account) it works locally, but fails if I deploy to app engine with my project-id.
DatastoreOptions datastoreOptions = DatastoreOptions.getDefaultInstance();
Datastore datastore = datastoreOptions.getService();
// A known entity in my live datastore
Key key = datastore.newKeyFactory()
.setKind("MyEntityType")
.newKey(123123123l);
Entity retrieved = datastore.get(key);
When it fails, I can get slightly more information:
java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
full stack:
com.google.cloud.datastore.DatastoreException: I/O error
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:434)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:431)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:430)
...
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:171)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:117)
at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:93)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
... 74 more
Caused by: java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:206)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
... 76 more
Update 2:
I've since paid for Google support and they have now confirmed it "is linked to a known issue related to the GAE standard Service Agent and access token which is affecting some users"
Related
We are currently migrating our App Engine from Java 8 to Java 11 using the bundled services (https://cloud.google.com/blog/products/serverless/support-for-app-engine-services-in-second-generation-runtimes).
We made it as far as that the application seems to be running as expected - BE, FE and Datastore all seem to work in our staging GCP project.
The following errors are showing up in our logs every 10 seconds, however:
Failed to query GCE metadata service
and then
java.io.IOException: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:70)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:609)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:488)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.readResponse(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.queryMetadataAttribute(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.getProjectId(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GcpHubClient.registerDebuggee(Unknown Source)
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:844)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:117)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:54)
... 6 more
Edit:
The Cloud Debugger says "The debugger could not find a debug target for the application". The versions still relying on Java 8 work though, so what extra step is missing for the migration to set up the debugger correctly? As we're using the App Engine standard environment, that should be enabled by default (https://cloud.google.com/debugger/docs/setup/java#gae-standard).
Edit 2:
Looks like this could be a Google Cloud Debugger bug:
https://github.com/GoogleCloudPlatform/cloud-debug-java/issues/18
As per the App Engine documentation regarding issuing HTTP requests, if you use URL Fetch, it will cause requests to cloud client libraries (and that includes the Google Cloud Debugger Client for Java) to fail.
As wer are using the bundled services, we are still using an appengine-web.xml file.
We had the following line in there:
<url-stream-handler>urlfetch</url-stream-handler>
Removing that line fixed the issue for us.
I'm using cloud app deploy but it comes:
Server responded with code [400]:
Bad Request Unexpected HTTP status 400.
Failed Project Preparation (app_id='b~eloquent-env-168317'). Out of retries. Last error: Temporary error occurred while verifying project: TEMPORARY_ERROR: Operation does not satisfy the following requirements: billing-enabled {Billing must be enabled for activation of service '' in project 'eloquent-env-168317' to proceed., https://console.developers.google.com/project/eloquent-env-168317/settings}
com.google.api.management.server.common.exceptions.ServiceManagementNonRetriableStorageException: Operation does not satisfy the following requirements: billing-enabled {Billing must be enabled for activation of service '' in project 'eloquent-env-168317' to proceed., https://console.developers.google.com/project/eloquent-env-168317/settings}
Beginning deployment of service [default]...
Building and pushing image for service [default]
Some files were skipped. Pass --verbosity=info to see which ones.
You may also view the gcloud log file, found at
[/Users/majnun/.config/gcloud/logs/2017.05.22/01.24.54.321942.log].
ERROR: (gcloud.app.deploy) You do not have permission to access project [eloquent-env-168317] (or it may not exist): The caller does not have permission
what's wrong, how can I fix it, thanks a lot
On 2017-May-21 your project was suspended. You should have received an email explaining the reason for the suspension, and the steps you need to follow to un-suspend your project. You can find out more information in the Project Suspension Guidelines.
You can see in the error you provided "Billing must be enabled for activation of service..". Therefore, submitting the Account Verification form should be all you need to do.
I'm trying to connect to a Cloud SQL database using GORM in golang.
db, _ = gorm.Open("mysql", "user:pass#cloudsql(connection:name:example)/")
if err != nil {
log.Println(err)
//panic(err)
}
When I attempt to serve the app
goapp serve appengine/
I get a runtime error
ERROR 2017-02-19 20:48:05,436 http_runtime.py:396] bad runtime process port ['\r\n']
Which I found was related to the database migration
db.AutoMigrate(&models.Event{})
If I remove the AutoMigrate, the runtime process port error goes away. However whenever I access a route (ie /events) that does a database query, the connection gets dropped, a 404 page is thrown, and an error message is logged sql: database is closed
When I run the app locally by building the package go build && ./appname and using a local MySQL server, it works fine.
Can someone please tell me how to connect to a Cloud SQL database using Go's GORM framework and App Engine?
This is due to the call to log.New in this file: https://github.com/jinzhu/gorm/blob/master/logger.go#L15
This anwser explain why dev_appserver.py gets it: https://stackoverflow.com/a/24112953/4266494
To disable this, you can either disable all GORM logging:
db.LogMode(false)
Or use an adapter the logger output: https://github.com/benguild/GAEBridge/blob/master/log/debugLevel.go
db.SetLogger(NewDebugLogger(nil)) // on application scope
db.SetLogger(NewDebugLogger(appengine.NewContext(req))) // on request scope
I'm setting a new logger with the real context
This is the only workaround I found to avoid crashes while keeping some logs, it could be awesome if one of you had a real one.
As it usually happens, the customer changed their G Suite email password which we were using for sending out emails (on Java#AppEngine). The token has been revoked as expected, and we managed to get everything working on DEV and UAT environments, however PROD for some reason still getting the infamous "invalid grant". I wonder if there's some caching that holds to original Credential in the datastore? Stackdriver doesn't let evaluate the method call so can't see what's inside. We suspect it has something to do with cache but restarting GAE instance (by redeploying) hasn't resolved the issue. Any suggestions are welcome!
This is a bit from the logs:
Caused by: com.google.api.client.auth.oauth2.TokenResponseException: 400
{
"error" : "invalid_grant"
}
at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
at com.google.api.client.auth.oauth2.Credential.executeRefreshToken(Credential.java:570)
at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:217)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:859)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
Not sure how it all worked this time, but this sequence seem to work:
pause the queues to avoid "old" token being cached again and again when it tries to resend the email
delete the stored credential from datastore
flush the memcache
re-configure your app to authorize it to have access to your gmail (essentially get the new credential to your datastore)
resume the queue and enjoy
I am running into an issue similar to this one. I have a GAE app and a GCE app that seem to work fine once in the cloud but I am having trouble getting my local environment working so that both of them access the same datastore.
I have setup the local datastore as described in the link above except my code looks like this (I had to build it this way in order to get it working in the cloud):
print("Connecting to datastore - " + datasetId);
// Get credentials form GCE if not local.
Credential computeEngineCredential = DatastoreHelper.getComputeEngineCredential();
if(computeEngineCredential != null)
{
print("Compute Engine Credetianls are not null! Access token: " + computeEngineCredential.getAccessToken());
}
DatastoreOptions options = DatastoreHelper.getOptionsfromEnv().credential(DatastoreHelper.getComputeEngineCredential()).dataset(datasetId).build();
print("Connection Host: " + options.getHost());
print("Connection Dataset: " + options.getDataset());
datastore = DatastoreFactory.get().create(options);
When I run the GCE app locally and try to connect to the running GAE datastore I get the following (I have replaced the actual data set id with "myDatasetId" in the output below):
Connecting to datastore - "myDatasetId"
Connection Host: http://localhost:8888
Connection Dataset: "myDatasetId"
com.google.api.services.datastore.client.DatastoreFactory makeClient
WARNING: Not using any credentials
There was a problem running query: Error:
runQuery Error 404 Error
404
com.google.api.services.datastore.client.DatastoreException: Error 404 Error
404
at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:114)
at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:80)
at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109)
My GAE console prints this out (I can access the admin console in the 8888 port just fine):
com.google.appengine.tools.development.AbstractModule startup
INFO: The admin console is running at http://localhost:8888/_ah/admin
com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for:
/datastore/v1beta2/datasets/"myDatasetId"/runQuery
I have verified the dataset ID in the GCE app and the GAE app match. I have been able to successfully run each app localy on their own and they both are able to properly connect to the local Datastore (this is while using the gcd.cmd tool for the GCE app). Based on the answer in the link above, it sounds like this is possible, am I doing something wrong?
Update
Not sure if this is related but I am getting the following error when starting up the GCD Tool:
SEVERE: Unable to load the App Engine dev agent. Security restrictions
will not be completely emulated. java.lang.RuntimeException:
Unexpected exception during cast.
at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:385)
at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:252)
at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:263)
at com.google.appengine.tools.development.agent.AppEngineDevAgent.premain(AppEngineDevAgent.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown
Source)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown
Source) Caused by: java.lang.IllegalAccessException: Class
com.google.apphosting.utils.clearcast.ClearCast$CasterImpl can not ac
cess a member of class
com.google.appengine.tools.development.agent.$Proxy0 with modifiers
"public"
at sun.reflect.Reflection.ensureMemberAccess(Unknown Source)
at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(Unknown
Source)
at java.lang.reflect.AccessibleObject.checkAccess(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:383)
... 9 more
If this initialized properly, could I somehow connect my GAE App to the GCD Tool datastore? So confused.
There's no officially supported way to share Datastore data between the Java Development Server (dev_appserver.sh) and the local Cloud Datastore tool (gcd.sh).
However, if your app is written in Java, you may be able to use the workaround described here and point dev_appserver.sh to the data file generated by gcd.sh by specifying the -Ddatastore.backing_store=<project dir>/WEB-INF/appengine-generated/local_db.bin option.
Thanks for replying this was very useful. I was able to point my local GAE to the local_db.bin that the gcd tool uses through eclipse by providing the "-Ddatastore.backing_store" as a VM argument as you suggested.
However, they still seem to have a different views of the datastore. The admin viewer for the GAE app running on the default 8888 port only shows the data added by the GAE app, on the other hand, the gcd one running on the gcd tool port (8080) shows the data added by the GCE app.
I assumed this was just a visibility issue on the admin site so I tried to access the GAE data through my GCE app but the query is unsuccessful, it doesn't seem to find the Entity kind and thus returns no results. I was able to verify that once deployed GCE is able to access the data the GAE app wrote to the datastore with the same query, due to this I am assuming it is not a namespace issue but more of an issue on where the data is held. Even though they are both pointing to the same local_db.bin file it seems the data is still split somewhere.
Am I supposed to run the dev_appserver.cmd directly form the command line maybe? If so, How do I do this for an EAR project (currently on eclipse).