I can't deploy my app, and I don't know how to fix it. I don't have any scaling configurations in my app.yaml, so I assume I'm under automatic scaling stuff. I'm using the flex env.
I only have two instances of the app right now and only three versions of it. So I'm not over any version limit.
I deployed once yesterday and it worked fine. when I changed some code and went to deploy again this happened. (I didn't save my changes to a different branch, so I don't know if it's just my code or some config I need to change in the console).
I'm not seeing anything related to my code in the build logs. The build seems like it succeeds 'cause it's green in the logs and says complete. Then it fails to update and gives me this quota error I can't find anything about in any docs.
Does this have anything to do with me getting two new domain names after the first deployment? (why would that matter?)
Here's the error:
DONE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [13] Flex operation projects/blah/regions/us-east1/operations/2c43cd11-2628-430b-9593-2020202020 error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2020-05-15T12:29:00.026Z67978.jc.2: Deployment Manager operation blah/operation-1589545740347-5a5aef53528c6-475fc7be-b67854 errors: [code: "RESOURCE_ERROR"
location: "/deployments/abc-default-12300515t082612/resources/abc-default-12300515t082612"
message: "{\"ResourceType\":\"compute.beta.regionAutoscaler\",\"ResourceErrorCode\":\"403\",\"ResourceErrorMessage\":{\"code\":403,\"errors\":[{\"domain\":\"usageLimits\",\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'abc-default-12300515t082612\'. Limit: 8.0\",\"reason\":\"limitExceeded\"}],\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'abc-default-20200515t0820000\'. Limit: 8.0\",\"statusMessage\":\"Forbidden\",\"requestPath\":\"https://compute.googleapis.com/compute/beta/projects/blah/regions/us-east1/autoscalers\",\"httpMethod\":\"POST\"}}"
dumb dumb dumb dumb dumb ;)
Related
Encountered and solved this problem. Posting for posterity since my situation differed from the usual.
Important: I am running a derivative of Ubuntu in a VirtualBox VM on a freshly installed Windows host.
The Problem
When creating a new react amplify project using create-react-app, amplify init fails with an InvalidSignatureException after selecting a profile:
$ amplify init
...
For more information on AWS Profiles, see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html
? Please choose the profile you want to use default
InvalidSignatureException: Signature not yet current: 20220528T081112Z is still later than 20220528T051608Z (20220528T051108Z + 5 min.)
at Object.extractError (/snapshot/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/snapshot/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/snapshot/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/snapshot/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
...
at IncomingMessage.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
code: 'InvalidSignatureException',
time: 2022-05-28T08:11:12.872Z,
requestId: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX',
statusCode: 403,
retryable: false,
retryDelay: 35.74742577991159
}
I also experienced errors with amplify push and amplify pull complaining about things such as "Time skew" and "Timeouts", and AWS Console threw "List apps call failed: Network Error".
Similar Posts
Here are a sample of posts discussing similar problems.
'amplify init' keeps failing
https://github.com/aws-amplify/amplify-js/issues/2014
https://github.com/concourse/s3-resource/issues/34
https://github.com/aws-amplify/amplify-hosting/issues/2417
Failed ideas
Sample of failed ideas:
I tried creating a second project to see if problem persisted. It did.
I tried syncing my time via ntp, ntpdate, ntpd. Same error.
I tried setting time via GUI. Failed to set. Same error.
Story leading to solution
At this point, I noticed my time occasionally jumps on sync with ntpd. Actual time was 01:40; on sync, it jump to 04:40, then back to 01:40. This would occur intermittently on sync. Timezones were set correctly. Trial and error exhausts my ideas, so I return to my host. Host time is 01:40, but discord messages are timestamping at 10:40. Wait.
Windows host timezone was set to UTC-8:00 Pacific Time. VirtualBox Linux guest was set to UTC-5:00 Eastern Time. Oddly, both were displaying 01:40.
Steps to resolve
Set host timezone to UTC-5:00 Eastern Time
Sync Clock in "Time and Date" settings in Windows
Disable/uninstall ntp from Linux guest
Issue was resolved. amplify init succeeded.
We implemented Dependency-check-maven and this was working fine up till now. But suddenly getting this error now,
[WARNING] An error occurred while analyzing 'oldui/includes/ckeditor/lang/no.js' (Sonatype OSS Index Analyzer).
Not sure where this is originating from?
I think the reason is due to https://ossindex.sonatype.org/ certificate is expired on 25th May. When you go to that website, you can see that you get the SSL error.
when i configure the flink-conf.yaml to collect metrics with the graphite plugin,
the most time only incomplete metrics are being sent. On the Taskmanager output multiple errors occur like:
2018-08-15 00:58:59,016 WARN org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while registering metric.
java.lang.IllegalArgumentException: A metric named mycomputer.taskmanager.8ceab4c3dfbf9fc5fa2af0447f1373a1.State machine job.Source: Custom Source.0.numRecordsOut already exists
at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91)
at org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131)
at org.apache.flink.runtime.metrics.MetricRegistryImpl.register(MetricRegistryImpl.java:329)
at org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:379)
at org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:312)
at org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:302)
at org.apache.flink.runtime.metrics.groups.OperatorIOMetricGroup.<init>(OperatorIOMetricGroup.java:41)
at org.apache.flink.runtime.metrics.groups.OperatorMetricGroup.<init>(OperatorMetricGroup.java:48)
at org.apache.flink.runtime.metrics.groups.TaskMetricGroup.addOperator(TaskMetricGroup.java:146)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.setup(AbstractStreamOperator.java:174)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.setup(AbstractUdfStreamOperator.java:82)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:143)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)
I've tried this on a completely freshly prepared flink-1.6.0 release with following config and the precompiled "State machine job" in the examples folder:
metrics.reporters: grph
metrics.reporter.grph.class: org.apache.flink.metrics.graphite.GraphiteReporter
metrics.reporter.grph.host: localhost
metrics.reporter.grph.port: 2003
metrics.reporter.grph.interval: 1 SECONDS
metrics.reporter.grph.protocol: TCP
I use the official graphite docker image (https://hub.docker.com/r/graphiteapp/docker-graphite-statsd/) that is running on the default configuration.
Has anybody an idea, how i can fix this issue?
Thank's and best regards
update
to exclude that a specific local setting is responsible for this behaviour, I repeated the process on a clean EC2 instance. There's exactly the same error here.
How to reproduce:
start EC2 t2.xlarge
installed java
download flink at https://www.apache.org/dyn/closer.lua/flink/flink-1.6.0/flink-1.6.0-bin-scala_2.11.tgz
added the flink-metrics-graphite-1.6.0.jar to lib
configured the flink-yaml.conf as mentioned in my previous post
./bin/start-cluster.sh
./bin/flink run examples/streaming/StateMachineExample.jar
I have not set up graphite in this case, because the error obviously already
occurs before.
After the job has been started you can view the error in the flink dashboard under Task Manager -> Logs
Our deployments for the default module of a particular app are failing, seemingly at random, at least 50% of the time, which is constantly disrupting our workflow.
With verbose logging turned on via appcfg.py update app.yaml --verbose, this error appears on the terminal:
03:43 PM Uploaded 4 files and blobs.
03:43 PM Compilation starting.
2015-09-23 15:43:51,886 INFO appcfg.py:1735 Send: /api/appversion/precompile, params={'version': 'myversion', 'app_id': 'myappid', 'module': 'default'}
03:43 PM Compilation completed.
03:43 PM Starting deployment.
2015-09-23 15:43:54,215 INFO appcfg.py:1735 Send: /api/appversion/deploy, params={'version': 'myversion', 'app_id': 'myappid', 'module': 'default'}
2015-09-23 15:43:56,341 INFO appcfg.py:2601 HTTP Error (HTTP Error 400: Bad Request Unexpected HTTP status 400)
03:43 PM Rolling back the update.
2015-09-23 15:43:56,341 INFO appcfg.py:1735 Send: /api/appversion/rollback, params={'version': 'myversion', 'app_id': 'myappid', 'module': 'default'}
Error 400: --- begin server output ---
Client Error (400)
The request is invalid for an unspecified reason.
--- end server output ---
The failure is extremely consistent, in that most of the time we try deploying the module after not having deployed for a few hours, the deployment attempt will fail with the above output.
Then, without changing any app code, retrying the deployment usually succeeds (but at times, the second attempt also fails, requiring subsequent deployment attempts).
This problem started happening earlier this year. Once the problem started occurring, it has not ceased. Before it occurred, we had no no issues with deployments.
The version of the module being deployed has no effect on the rate of deployment success. We are using the Python runtime for this module.
I have already emailed a Solutions Architect from Google about this, but apart from upgrading to a paid support plan to ensure someone looks into this, he suggested I post on here with the hope that the App Engine support team gets back to me.
App Engine support team - can you find out what is going on (and ideally provide a fix)? If you need more information (such as my app ID), please let me know.
I use java backend on google's appengine with eclipse sdk.
When I tried to update my backend with eclipse, i got error that says
com.google.appengine.tools.admin.HttpIoException: Error posting to URL: >https://appengine.google.com/api/appversion/create?app_id=infra-filament-610&version=2&
409 Conflict
Another transaction by user heodong91 is already in progress for app: s~infra-filament-610, version: 2. >That user can undo the transaction with "appcfg rollback".
So I ran the rollback on terminal with command
appcfg.sh rollback /Users/Donghyeok/PlanItBackend/src/main/webapp
but its giving me this error
Unable to rollback app: Error posting to URL: httpsds://appengine.google.com/api/appversion/rollback?>app_id=infra-filament-610&version=2&
404 Not Found
This application does not exist (app_id=u'infra-filament-610').
So then I looked up possible solutions and ran this on the command line.
appcfg.sh --no_cookies rollback /Users/Donghyeok/PlanItBackend/src/main/webapp
It still gave me the same error
This is try #3
Unable to rollback app: Error posting to URL: https://appengine.google.com/api/appversion/rollback?app_id=infra-filament-610&version=2&
404 Not Found
This application does not exist (app_id=u'infra-filament-610').
I tried again with --oauth2 option and still give me the same error.
I also tried removing .appcfg_oauth2_tokens_java file but same error.
I have tried many things in many combinations but I wasn't able to solve this issue. So the main problem is the first one. But to solve this I had to rollback, but its not letting me rollback.
Could someone please help?
I'm like desperate...
This can be fixed by changing the app name in app.yaml to the name of the project (visible in Google Developer Console, often suffixed with a number). It may also work by simply adding at the end of your rollback command line the following:
--application=<projectname>
Regards