As it usually happens, the customer changed their G Suite email password which we were using for sending out emails (on Java#AppEngine). The token has been revoked as expected, and we managed to get everything working on DEV and UAT environments, however PROD for some reason still getting the infamous "invalid grant". I wonder if there's some caching that holds to original Credential in the datastore? Stackdriver doesn't let evaluate the method call so can't see what's inside. We suspect it has something to do with cache but restarting GAE instance (by redeploying) hasn't resolved the issue. Any suggestions are welcome!
This is a bit from the logs:
Caused by: com.google.api.client.auth.oauth2.TokenResponseException: 400
{
"error" : "invalid_grant"
}
at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
at com.google.api.client.auth.oauth2.Credential.executeRefreshToken(Credential.java:570)
at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:217)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:859)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
Not sure how it all worked this time, but this sequence seem to work:
pause the queues to avoid "old" token being cached again and again when it tries to resend the email
delete the stored credential from datastore
flush the memcache
re-configure your app to authorize it to have access to your gmail (essentially get the new credential to your datastore)
resume the queue and enjoy
Related
I have a Google Appengine app (Java) and the datastore was recently auto-migrated into "Cloud Firestore in Datastore mode" (completed on 6/6/2022) that has been running for years.
It has now started throwing these exceptions in new deployments without any relevant code change to trigger them:
com.google.appengine.api.datastore.DatastoreFailureException: Missing or invalid authentication.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:69)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:127)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:68)
at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
at com.google.appengine.api.datastore.BaseQueryResultsSource.getIndexList(BaseQueryResultsSource.java:154)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:187)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:166)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:146)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:64)
at com.googlecode.objectify.impl.KeysOnlyIterator.hasNext(KeysOnlyIterator.java:29)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:48)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:20)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:52)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1105)
at com.googlecode.objectify.impl.ChunkingIterator.hasNext(ChunkingIterator.java:51)
at com.google.common.collect.Iterators.addAll(Iterators.java:366)
at com.google.common.collect.Lists.newArrayList(Lists.java:163)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:34)
at com.sun.proxy.$Proxy68.isEmpty(Unknown Source)
My code. After loading a list of entities, the exception is thrown on the isEmpty() call on the returned list:
List<SystemSetting> applicationSettings = ofy().load().type(SystemSetting.class).filter("name", name).list();
if (applicationSettings.isEmpty()) { // Exception thrown here
It's worth also noting that on previous versions, datastore reads and writes are still happening without any issues. It is only new deployments that exhibit this issue. However, redeploying previous versions from previous commits using the same CI service to deploy them are failing.
Is this caused by the datastore migration?
Slight update: these errors are only happening on newly deployed versions... In fact, I deployed a new version (from CI, bitbucket pipeline) that worked, but when I redeployed it with no changes it no longer works. Same app engine version id, code and users etc.
Things we've tried without success:
Re-deploying last known version
Clearing memcache
Deploying from local dev
Stopping instances
Trying alternative service account key
Expanding service account permissions
Clearing BitBucket pipeline caches (the CI builder)
Have recreated the issue with datastore api directly (no objectify)
Have redeployed with different user account - no success
Have redeployed with new service account - no success
Update 1:
If I run the following code on a local java application (pointing at live datastore using same service account) it works locally, but fails if I deploy to app engine with my project-id.
DatastoreOptions datastoreOptions = DatastoreOptions.getDefaultInstance();
Datastore datastore = datastoreOptions.getService();
// A known entity in my live datastore
Key key = datastore.newKeyFactory()
.setKind("MyEntityType")
.newKey(123123123l);
Entity retrieved = datastore.get(key);
When it fails, I can get slightly more information:
java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
full stack:
com.google.cloud.datastore.DatastoreException: I/O error
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:434)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:431)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:430)
...
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:171)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:117)
at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:93)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
... 74 more
Caused by: java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:206)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
... 76 more
Update 2:
I've since paid for Google support and they have now confirmed it "is linked to a known issue related to the GAE standard Service Agent and access token which is affecting some users"
I'm trying to write a VSCode extension where users could log into Google AppEngine with a google account, and I need to get their SACSID cookie to make appengine requests.
So I'm opening a browser window at
https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttp://localhost:3000/
(generated by google.appengine.api.users.create_login_url)
The user logs in and is redirected to my local webserver at
localhost:3000/_ah/conflogin/?state={state}
Now I try to forward the request to my AppEngine app (since it knows how to decode the state parameter), so I do a request to
https://my-app.appspot.com/_ah/conflogin/?state={state}
basically just replacing localhost with the actual app.
but it doesn't work, presumably because the domain is different. I assume this is on purpose, for security.
Is there any way I can make this work ?
Not ideal, but the only solution I've found is to have an endpoint on my GAE instance that does the redirection. Then I can set that as the continue url, when I'm starting the authentication process
https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://my-app.appspot.com/redirect?to=http://localhost:3000
I think you should center the attention on the protocols you are using, since it’s known that the cookie name is based on the http protocol (HTTP : ACSID, HTTPS:SACSID), and that’s the security perspective till this point for me.
Having the error you are facing now would be helpful to understand the problem better. Also, how are you performing the call to the API and the code you are using would be helpful too.
I am getting error on test and development server for uploading file with azure blob storage.IT is uploading locally without any problem.We are using Nuget for File handling. On debugging we are getting error on container.CreateIfNotExist()
Could anybody help me solving the error.
Thanks in advance!
Based on your description, I assumed that you are using the azure storage client library WindowsAzure.Storage for uploading files to your blob storage.
On debugging we are getting error on container.CreateIfNotExist()
If you construct the CloudStorageAccount with the AccountName and AccountKey, please make sure your AccountKey is correct, and you could login into azure portal and check with it. If you construct the CloudStorageAccount via the account-level SAS token, please make sure the SAS token is valid and it contains the related permissions. Moreover, you could re-generate your account key or new SAS token to narrow this issue. Also, you could leverage fiddler to capture the network traces when executing the operations to narrow this issue.
Additionally, you need to check your server time. As Authentication for the Azure Storage Services states as follows:
The storage services ensure that a request is no older than 15 minutes by the time it reaches the service. This guards against certain security attacks, including replay attacks. When this check fails, the server returns response code 403 (Forbidden).
Also, you could Enabling Storage Logging and Accessing Log Data to retrieve the detailed error message.
Problem was Microsoft.ApplicationInsights 2.4.0.
Solved by downgrading to 2.3.0.
It is really strange but below links really helped me out to solve the issue
Azure Storage Emulator 403 Forbidden
Azure CloudBlobContainer.CreateIfNotExists() throws Forbidden (403) on Local Development
Thanks!
I am using the Gmail API to put messages into a Google Apps email account. I use
the OAuth 2.0 authentication protocol with a service account. This is more or
less working fine. One of our customers has asked us to put messages
directly into a Google Vault. I don't see a Vault API, but I did find this
information related to the "insert" method (which is what we use to add
messages to a normal account):
parameter "deleted" (boolean): Mark the email as permanently deleted
(not TRASH) and only visible in Google Apps Vault to a Vault administrator.
Only used for Google Apps for Work accounts.
When I do this, some messages are accepted, but frequently I get http error
500 in response to the POST. The error text says "Backend Error". I thought
the pattern was that the first time the message was posted, it would work,
but the second time would generate the error. Therefore I was thinking it
was a duplicate check issue. However I now see some examples of messages
that fail immediately. The POST url looks like this:
https://www.googleapis.com/upload/gmail/v1/users/user#domain.com/messages?uploadType=multipart&internalDateSource=dateHeader&deleted=true&access_token=ABC...
As I mentioned, the same message to the same url (without deleted=true) will
always work. Any ideas what is causing the error?
Was just fighting this issue myself. Apparently the error has something to do if the message is compatible with the Google vault retention policies:
If I turn on a default policy of "Retain everything" then I've been able to get the messages to import correctly. HTH!
I'm using the import api method and the backendError seems to be related to filters/policies. For example we asked Google to reject messages with xls and macros and we get the error on mail with that kind of attachment
I'm using app engine for windows and when I try to deploy the app to the cloud, I'm asked for a username and password. When I enter the username and password, I get the following error:
Password for xxxxx#xxxxx.xxx: Invalid username or password.
2014-11-06 18:48:41,628 ERROR appcfg.py:2417 An error occurred processing file '': HTTP Error 401: > > > Unauthorized. Aborting.
Error 401: --- begin server output ---
Must authenticate first.
--- end server output ---
2014-11-06 18:48:41 (Process exited with code 1)
Then, I got an email from Google stating, that an attempt was made to login my account and it was blocked. If it wasn't me, take the following steps. But, if it was me:
If this was you
You can switch to an app made by Google such as Gmail to access your account (recommended) or > change your settings at https://www.google.com/settings/security/lesssecureapps so that your >account is no longer protected by modern security standards.
Any ideas? I'd hate to lower my security standards. Is this some sort of bug?
Use OAuth2 rather than a username and password to authenticate when uploading your app.
Looks like you are hitting the same issue as this guy.
As I told him there:
Go here under Account permissions -> Access for less secure apps -> Settings make sure its set to enabled.
Otherwise you will have to use the console.
You might be accessing google servers via some proxy. And google being a LOCATION geek, over-intelligently blocks you. Try authenticating it once, and it should not be a problem from then on, provided proxy or IP addresses does not change frequently. NOTE: i might be completely wrong as well.