error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2020-05-22T15:14:57.416Z3210.jc.5: Deployment Manager operation thesis-lock/operation-1590160497681-5a63e1799a578-3c148be2-663d8bc4 errors: [code: "RESOURCE_ERROR"
location: "/deployments/aef-flex-20200522t171231/resources/aef-flex-20200522t171231"
message: \"ResourceType\":\"compute.beta.regionAutoscaler\",\"ResourceErrorCode\":\"403\",\"ResourceErrorMessage\":{\"code\":403,\"errors\":[{\"domain\":\"usageLimits\",\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'aef-flex-20200522t171231\'. Limit: 8.0\",\"reason\":\"limitExceeded\"}],\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'aef-flex-20200522t171231\'. Limit: 8.0\",\"statusMessage\":\"Forbidden\",\"requestPath\":\"https://compute.googleapis.com/compute/beta/projects/.../regions/asia-south1/autoscalers\",\"httpMethod\":\"POST\"}}"
I'm getting the aboveerror when trying to deploy my app to Google App Engine using gcloud app deploy.
I've been able to deploy previously without any problems or errors in exactly the same way. I haven't changed my app.yaml. I have checked my quota in the Console yet I can find no quota that have been exceeded. The documentation does not provide any insights.
Any ideas as to what I can do?
I have found some similar questions on SO, but none of them seem to point to this issue specifically and none of the proposed solutions to those questions seem to work.
Google is already aware of this, and the engineering team is working on it. There is an Issue Tracker thread that you can "star" to keep track of updates and follow-ups regarding this issue.
Current workaround: Switch from App Engine Flexible to Standard environment.
Related
I have an appengine (Java) app with the ability to sign in via Google. For this I use
UserServiceFactory.getUserService().createLoginURL(...).
This has been working fine so far, and still works well locally (using mvn appengine:run) but on production it consistently throws a generic 500 while on /_ah/conflogin?state=~AJKiYcHHHXI45-... (the 5th URL of the login process, while already being logged in with Google) and I can find nothing relevant in the Logs Explorer at https://console.cloud.google.com/logs/...
I've since even updated to the latest https://mvnrepository.com/artifact/com.google.appengine/appengine-api-1.0-sdk/2.0.10 but the problem remains.
Where should I look at to solve this issue?
To find the root cause of this issue, you can find the logs for this error will be helpful.
In this documentation there is a sample query that you could try to check for logs error with status 500.
resource.type="gae_app" AND
log_id("appengine.googleapis.com/request_log") AND
httpRequest.status>=500
Alternatively you could also try running ‘gcloud app logs read’ as mentioned in this document to see if you get any logs.
The issue tracker mentioned by Rez in comment is marked as Fixed and after checking your issue I think it also relates to the same issue tracker. As it closed I suggest to get your issue resolved I recommend to raise new issue tracker by referencing the fixed one or else you may raise support ticket with google
I ran into this error while trying to deploy my Node.js app on Google's App Engine:
(gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2022-01-07T08:03:21.449Z334.zf.2: The region us-west3 does not have enough resources available to fulfill the request. Please try again later.
It seems like the error is with the region "not having enough resources," but how is this possible? Is it really the case that us-west3 has run out of "capacity"? Or does this error indicate something wrong with my app? I've deployed it before on Google App Engine but I am open to the possibility that something in my code could be causing errors. I don't know what this could be, but the problem has to be on my end, right?
As suggested by John Hanley, you will have to wait and try again, but you cannot change the region of App Engine, you will have to create another project in another region if you want to.
We are deploying java application to use VisionAPI in Google app engine and we are getting capacity error, we were asked to try with different zones still we are getting the same error,
GCLOUD: ERROR: (gcloud.app.deploy) Error Response: [8] Flex operation projects/text-convert-304513/regions/us-east1/operations/6d4717fc-a5e9-419c-85cc-72394ed9e68a error [RESOURCE_EXHAUSTED]: An internal error occurred while processing task /app-engine-flex/insert_flex_deployment/flex_create_resources>2021-02-11T13:54:32.406Z50061.ue.1: The requested amount of instances has exceeded GCE's default quota. Please see https://cloud.google.com/compute/quotas for more information on GCE resources.
As per the GCP doc the parameter 'max_num_instances', The maximum number of instances in your project should be 8 by default where I can see you may want to use more than that limit. I would like to suggest you to increase the quota limit for your project will solve the issue.
I am having a problem on some of my AppEngine projects, since a few days I started to I see a lot of errors (which I noticed they might happen when an health check arrives) in my vm.syslog logs from Stackdriver Logging.
In the specific these are:
write_gcm: Server response (CollectdTimeseriesRequest) contains errors:#012{#012 "payloadErrors": [#012 {#012 "index": 71,#012 "error": {#012 "code": 3,#012 "message": "Expected 4 labels. Found 0. Mismatched labels for payload [values {\n data_source_name: \"value\"\n data_source_type: GAUGE\n value {\n double_value: 694411264\n }\n}\nstart_time {\n seconds: 1513266364\n nanos: 618061284\n}\nend_time {\n seconds: 1513266364\n nanos: 618061284\n}\nplugin: \"processes\"\nplugin_instance: \"all\"\ntype: \"ps_rss\"\n] on resource [type: \"gce_instance\"\nlabels {\n key: \"instance_id\"\n value: \"xxx\"\n}\nlabels {\n key: \"zone\"\n value: \"europe-west2-a\"\n}\n] for project xxx"#012 }#012 }#012 ]#012}
write_gcm: Unsuccessful HTTP request 400: {#012 "error": {#012 "code": 400,#012 "message": "Field timeSeries[11].metric.labels[1] had an invalid value of \"health_check_type\": Unrecognized metric label.",#012 "status": "INVALID_ARGUMENT"#012 }#012}
write_gcm: Error talking to the endpoint.
write_gcm: wg_transmit_unique_segment failed.
write_gcm: wg_transmit_unique_segments failed. Flushing.
At the same time, I noticed that my Memory Usage in the AppEngine dashboard for the very same projects is increasing with the passing of time at the point where it reaches the max amount available and the instance restarts, throwing a 502 error when visiting the web site that the app is serving.
All this is not happening on a couple of projects that have not been updated since at least 2 weeks (neither the errors above or the memory increase) but it does happen on a newly created instance when deployed with the same codebase of one of the healthy projects. In addition, I don't happen to see any increase in the memory when running my project locally.
Can someone gently tell me if they experienced something similar or if they think that the errors and the memory increase are related? I have haven't changed my yaml file for deployment recently and I haven't specified any custom configuration for the health checks (which run on legacy mode at the default rate).
Thank you for your help,
Nicola
Simliar question here App Engine Deferred: Tracking Down Memory Leaks
Going through same thing in compute engine on a single VM. I've tried increasing memory but the problem persists. Seems to be tied to a stackdriver method call. Not sure what to do, causes machines to stop after about 24hrs for me. In my case, I'm getting information every 3 seconds from a set of API's, but the error comes up every minute in the serial port 1 (console), which makes me suspect that it is a some kind of failure outside of my code. More from Google here: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.collectdTimeSeries/create .
I'm not sure about all of the errors, but for the "write_gcm: Server response (CollectdTimeseriesRequest)" I had the same issue and contacted Google Cloud Support. They told me that the Stackdriver service has been updated recently to accept more detailed information on ps_rss metrics, but it has caused metrics from older agents to not be sent at all.
You should be able to fix this issue by upgrading your Stackdriver agent to the latest version. On Compute Engine (that I was running) you have control over this, I'm not sure how you'd do it on AppEngine, maybe trigger a new deploy?
I've deployed an app using googles cloud compute engine service.
I get an error when I try to register a new user:
We're sorry, but something went wrong. If you are the application owner check the logs for more information.
I'm in the logs section of the google cloud console and it has all sorts of random logs I can look at but I cant find where the errors that the app coding itself caused caused are put, e.g. I'm sure its a mail configuration problem that is causing the error message for new registered users but I cant fix the problem until I find out what the actual error is.
Where exactly are actual app errors put, because they aren't displayed on the page the like in production on my local computer. (I'm using a rails app if that makes a difference to anything)
All errors are logged in the logging section, try applying filters or sorting the logs by date. Sometimes they take a little while to appear.