How does app engine flexible environment works? It creates one BIG instance, by increase memory and CPU capacity, or creates several small instances?
How I can implement one BIG instance, which will increase memory and CPU only, without creating new instances?
Look at the resource settings to increase your host size.
https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml#resource-settings
Related
I am a cloud noob.
I have a node application with GAE. I am using basic scaling to serve up the requests. I have specified instance class B4_1G which has memory limit of 2048mb.(https://cloud.google.com/appengine/docs/standard#second-gen-runtimes)
The application is supposed to perform DOM scraping using Cheerio on some extremely large HTML files. This works well until the HTML that I need to scrape is beyond huge. Then I start getting memory error in logs :
Exceeded hard memory limit of 2048 MB with 2052 MB after servicing 1 requests total. Consider setting a larger instance class in app.yaml.
Is there any way I can override the memory limit to say 4096mb or even more?
Setting the resources additionally in app.yaml did not seem to help.
Any help or pointers appreciated. Thank you.
The link you provided shows the support instance sizes.
If you need more than 2 GB of memory you will need to switch to App Engine Flexible or a Compute Engine instance.
In Google Cloud Platform Monitoring, is there any way to monitor memory usage by App Engine instance? I see that there is the Memory usage metric for the GAE Application resource type, but I don't see a Memory usage metric for the GAE Instance resource type.
My particular use case is that I'd like to see (and make alerts based on) memory usage per instance.
Currently there's no built-in metric to monitor per instance memory. The only way to monitor the Memory Usage within a GAE Instance is by creating a custom metric, in this link you can find more information about custom metrics.
Just remember that the amount of instances will increase according to the load, so a per instance alert shouldn't reflect the actual load of your application, that's why the memory usage is normally measured by group on the built in metric rather than per instance.
I have a few requests that needs to use extensive amount of memory i.e. 40 MB more than other requests.
At the default of 10 max concurrent requests using a F1 auto-scaling instance, it can potentially use 400+ MB, which is way more than the 130MB-ish system memory it has available. There is no memory utilization setting in the yaml file, so I wonder what can be done to prevent such situations.
Google App Engine don't have any memory utilization beside Python Garbage Collection.
My Advise is,
Try to release memory as soon as response
Try to optimize memory usage on that part, may be you need to use other service to help solving memory usage problem, eg. file serving via Google Storage, etc...
Scale up instance to F2 which more suitable for production, but you still need to optimize your memory usage for higher usage.
Is there a memory limit to the Task Queue on Google App Engine. I'm specifically concerned with the Go runtime, but it would be nice to get answers on all runtimes if someone can provide them.
The tasks are executed by the same app instance(s) as the regular requests, only tasks are allowed to run longer. So the same memory limits apply (also subject to the task queue specific limits and quota, which might also eat into the instance memory).
You might chose to direct tasks to a dedicated module to which you assign an instance class with more memory (more powerful as well), if memory consumption is a concern.
But since the max instance class memory size is currently 1G I suspect your instance will most likely hit the 'soft private memory limit' and be killed before loading a full 1G file into memory :)
A "task" is essentially represented by a URL that's stored away for later delivery to an instance of your app. The representation of a task is independent of language, unless you use a stringified, language-specific serialization of something as a value.
If by memory limit you mean "how much (where much = count*size) task queue stuff can I have pending?," the answer is spelled out in the Task Queue section of the quotas document.
If you're asking how big a single task can be, that'll depend on the memory size of your instances, since you'll need enough memory to construct a task before enquing it.
For task processing the app instances need enough memory to accept and process a task, or enough memory to accept and process many concurrently, if your app is configured to accept multiple simultaneous requests. How much memory that takes beyond accepting the URL is basically up to how the app is coded.
What exactly is a Google App Engine Instance as per their plans to change the billing model as they exit preview?
Is an instance a VM with a set of memory and a fixed amount of dedicated CPU power, memory etc or something else?
Is an instance a VM with a set of memory and a fixed amount of dedicated CPU power, memory etc or something else?
Yes, exactly that it is. A VM with a set of memory and a fixed amount of dedicated CPU power.
I can't remember how many instances a google server has but they said that in a I/O session. They also said what kind of servers google is using for App Engine ( quad core with some big amount of ram). Here it's the recording if I'm not wrong https://www.youtube.com/watch?v=7B7FyU9wW8Y
http://code.google.com/appengine/docs/adminconsole/instances.html#Introduction_to_Instances for a general description.
http://code.google.com/appengine/docs/python/config/backends.html for specifics on backends, where you have more control over the resource that you're either reserving or using on demand.