Google App Engine deployment failure: "The request is invalid for an unspecified reason." How to fix? - google-app-engine

Our deployments for the default module of a particular app are failing, seemingly at random, at least 50% of the time, which is constantly disrupting our workflow.
With verbose logging turned on via appcfg.py update app.yaml --verbose, this error appears on the terminal:
03:43 PM Uploaded 4 files and blobs.
03:43 PM Compilation starting.
2015-09-23 15:43:51,886 INFO appcfg.py:1735 Send: /api/appversion/precompile, params={'version': 'myversion', 'app_id': 'myappid', 'module': 'default'}
03:43 PM Compilation completed.
03:43 PM Starting deployment.
2015-09-23 15:43:54,215 INFO appcfg.py:1735 Send: /api/appversion/deploy, params={'version': 'myversion', 'app_id': 'myappid', 'module': 'default'}
2015-09-23 15:43:56,341 INFO appcfg.py:2601 HTTP Error (HTTP Error 400: Bad Request Unexpected HTTP status 400)
03:43 PM Rolling back the update.
2015-09-23 15:43:56,341 INFO appcfg.py:1735 Send: /api/appversion/rollback, params={'version': 'myversion', 'app_id': 'myappid', 'module': 'default'}
Error 400: --- begin server output ---
Client Error (400)
The request is invalid for an unspecified reason.
--- end server output ---
The failure is extremely consistent, in that most of the time we try deploying the module after not having deployed for a few hours, the deployment attempt will fail with the above output.
Then, without changing any app code, retrying the deployment usually succeeds (but at times, the second attempt also fails, requiring subsequent deployment attempts).
This problem started happening earlier this year. Once the problem started occurring, it has not ceased. Before it occurred, we had no no issues with deployments.
The version of the module being deployed has no effect on the rate of deployment success. We are using the Python runtime for this module.
I have already emailed a Solutions Architect from Google about this, but apart from upgrading to a paid support plan to ensure someone looks into this, he suggested I post on here with the hope that the App Engine support team gets back to me.
App Engine support team - can you find out what is going on (and ideally provide a fix)? If you need more information (such as my app ID), please let me know.

Related

Amplify Init - InvalidSignatureException: Signature not yet current

Encountered and solved this problem. Posting for posterity since my situation differed from the usual.
Important: I am running a derivative of Ubuntu in a VirtualBox VM on a freshly installed Windows host.
The Problem
When creating a new react amplify project using create-react-app, amplify init fails with an InvalidSignatureException after selecting a profile:
$ amplify init
...
For more information on AWS Profiles, see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html
? Please choose the profile you want to use default
InvalidSignatureException: Signature not yet current: 20220528T081112Z is still later than 20220528T051608Z (20220528T051108Z + 5 min.)
at Object.extractError (/snapshot/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/snapshot/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/snapshot/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/snapshot/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
...
at IncomingMessage.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
code: 'InvalidSignatureException',
time: 2022-05-28T08:11:12.872Z,
requestId: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX',
statusCode: 403,
retryable: false,
retryDelay: 35.74742577991159
}
I also experienced errors with amplify push and amplify pull complaining about things such as "Time skew" and "Timeouts", and AWS Console threw "List apps call failed: Network Error".
Similar Posts
Here are a sample of posts discussing similar problems.
'amplify init' keeps failing
https://github.com/aws-amplify/amplify-js/issues/2014
https://github.com/concourse/s3-resource/issues/34
https://github.com/aws-amplify/amplify-hosting/issues/2417
Failed ideas
Sample of failed ideas:
I tried creating a second project to see if problem persisted. It did.
I tried syncing my time via ntp, ntpdate, ntpd. Same error.
I tried setting time via GUI. Failed to set. Same error.
Story leading to solution
At this point, I noticed my time occasionally jumps on sync with ntpd. Actual time was 01:40; on sync, it jump to 04:40, then back to 01:40. This would occur intermittently on sync. Timezones were set correctly. Trial and error exhausts my ideas, so I return to my host. Host time is 01:40, but discord messages are timestamping at 10:40. Wait.
Windows host timezone was set to UTC-8:00 Pacific Time. VirtualBox Linux guest was set to UTC-5:00 Eastern Time. Oddly, both were displaying 01:40.
Steps to resolve
Set host timezone to UTC-5:00 Eastern Time
Sync Clock in "Time and Date" settings in Windows
Disable/uninstall ntp from Linux guest
Issue was resolved. amplify init succeeded.

How to fix deployment error QUOTA_FOR_INSTANCES?

I can't deploy my app, and I don't know how to fix it. I don't have any scaling configurations in my app.yaml, so I assume I'm under automatic scaling stuff. I'm using the flex env.
I only have two instances of the app right now and only three versions of it. So I'm not over any version limit.
I deployed once yesterday and it worked fine. when I changed some code and went to deploy again this happened. (I didn't save my changes to a different branch, so I don't know if it's just my code or some config I need to change in the console).
I'm not seeing anything related to my code in the build logs. The build seems like it succeeds 'cause it's green in the logs and says complete. Then it fails to update and gives me this quota error I can't find anything about in any docs.
Does this have anything to do with me getting two new domain names after the first deployment? (why would that matter?)
Here's the error:
DONE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [13] Flex operation projects/blah/regions/us-east1/operations/2c43cd11-2628-430b-9593-2020202020 error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2020-05-15T12:29:00.026Z67978.jc.2: Deployment Manager operation blah/operation-1589545740347-5a5aef53528c6-475fc7be-b67854 errors: [code: "RESOURCE_ERROR"
location: "/deployments/abc-default-12300515t082612/resources/abc-default-12300515t082612"
message: "{\"ResourceType\":\"compute.beta.regionAutoscaler\",\"ResourceErrorCode\":\"403\",\"ResourceErrorMessage\":{\"code\":403,\"errors\":[{\"domain\":\"usageLimits\",\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'abc-default-12300515t082612\'. Limit: 8.0\",\"reason\":\"limitExceeded\"}],\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'abc-default-20200515t0820000\'. Limit: 8.0\",\"statusMessage\":\"Forbidden\",\"requestPath\":\"https://compute.googleapis.com/compute/beta/projects/blah/regions/us-east1/autoscalers\",\"httpMethod\":\"POST\"}}"
dumb dumb dumb dumb dumb ;)

Download logs from Google App Engine with server errors

I try to download the logs from my Google App Engines with the command:
appcfg.py request_logs [directory] mylogs.txt
However, I encountered the server error as below:
C:\Program Files (x86)\Google\google_appengine>appcfg.py request_logs E:\PhD\cod
e\python\webDevelop\similarAPI\similarAPI/ mylogs.txt
05:22 PM Application: similarapi
05:22 PM Host: appengine.google.com
05:22 PM Downloading request logs for app similarapi version 1.
Error 500: --- begin server output ---
Server Error (500)
A server error has occurred.
--- end server output ---
I have looked at the official document and related Stack Overflow questions, but no clues are found. Actually I can get the logs in Dec, 2016 but not now. And I tried it for one week without the success, so it should not be the error of Google's infrastructure.
Does anyone know how to solve that problem? Thanks in advance.
Just a workaround, not a direct answer.
The GAE SDK might be getting a bit less attention compared to the cloud SDK (which is the recommended one lately). I, for one, can't even get as far as you got since appcfg.py request_logs can't properly detect my multi-service app from its directory and was not happy with any syntax attempt I made to specify it via args.
But I have the cloud SDK installed as well and I was able to get the logs for my app using gcloud app logs read (after setting up the gcloud context for my app)
FWIW, this isn't yet enough to make me switch, I'm still primarily a GAE SDK user. YMMV.
Finally, it has be proved that it is a bug of Google's infrastructure from 22 Mar to 10 Apr, 2017. Three related bug reports can be seen in the Google Issue Tracker 36637246, 37192011, 36969442.
Now the command appcfg.py request_logs [directory] mylogs.txt works well.

Can't deploy my App Engine since SDK 1.9.46

Since I upgraded my Gradle build to deploy my App Engine application with SDK 1.9.46, I get deployment conflict messages, like the one below.
Beginning interaction for module admin...
0% Created staging directory at: '/var/folders/m8/6z4h4k2x11s3whxrqmd79lym0000gn/T/appcfg2292422843793738444.tmp'
5% Scanning for jsp files.
8% Generated git repository information file.
20% Scanning files on local disk.
25% Scanned 250 files.
28% Scanned 500 files.
31% Scanned 750 files.
33% Scanned 1000 files.
34% Initiating update.
Nov 13, 2016 11:11:41 AM com.google.appengine.tools.admin.AbstractServerConnection send1
WARNING: Error posting to URL: https://appengine.google.com/api/appversion/create?module=admin&app_id=the-outdoor-game&version=dev&
409 Conflict
There is an operation pending for this application: AppLock held for "Updating engine_version_id='dev' within engine_id='default'." acquired by Peter.Fortuin at 2016-11-13 09:15:05.482616 GMT. Please wait and try again or use 'appcfg rollback' to attempt to cleanly abort the current operation.
This is try #0
A rollback if not solving the problem. The only thing that seems to work is downgrading the App Engine SDK to 1.9.44. But that gives me other problems that I don't want to solve.
Anybody has any idea how to solve this? Can it be that this is a bug in the 1.9.46 SDK? Anybody else seeing the same thing?
I have resolved this problem by changing 'service' to 'module' in appengine-web.xml of my module.
See https://github.com/GoogleCloudPlatform/gradle-appengine-plugin/issues/268 for more information.

Google app engine launcher: Invalid username or password

I've download google app engine (python) for windows and install it. Then I went to console.developers.google.com/project and create new project. After that, I launched google app engine launcher, create new application and named it as project I have recently create. I click on deploy and constantly getting this error:
03:35 PM Application: udacity-hello-pozdrav; version: 1
03:35 PM Host: appengine.google.com
03:35 PM
Starting update of app: udacity-hello-pozdrav, version: 1
03:35 PM Getting current resource limits.
Password for t******.****#gmail.com: Invalid username or password.
2014-09-06 15:35:56,690 ERROR appcfg.py:2411 An error occurred processing file '': HTTP Error 401: Unauthorized. Aborting.
Error 401: --- begin server output ---
Must authenticate first.
--- end server output ---
2014-09-06 15:35:56 (Process exited with code 1)
You can close this window now.
I've read about appcfg.py but when I enter it in cmd, It says that it's not recognized.
Would be much better if you will use appcfg.py with --oauth2 flag.
Like appcfg.py --oauth2 update /path/to/your/app.
For the first time you will see the browser window where you'll need to allow access to your account. Then you can deploy without entering any credentials and without confirmations. This will work even with 2-step authentication turned on.
The "Invalid username or password" error is due to Google security protection. To overcome this error message and be able to deploy from the Launcher, go to your Google Account Settings, then enable "Access for Less Secure Apps" under Security Check Up.

Resources