I have a webserver running in App Engine and the client is a mobile app. I am seeing that a lot of requests on the mobile are starting to fail once we scaled up to a lot of users. I am not seeing any failures in our logs however. I noticed in our quotas that our ip address in use for Compute Engine API is at its max of 8 (even though we're not running any services on Compute Engine).
I am not sure if this is the root cause but it wasn't like this before, I was wondering if there is any advice on how to address this problem or if there are better way to structure our server to meet our use case.
EDIT:
Our current configuration is a flex environment on App engine, with minimum 2 instances. We also have a MySQL instance. Those pretty much so far everything we've used.
runtime: php
env: flex
api_version: 1
handlers:
- url: /.*
script: public/index.php
runtime_config:
document_root: public
beta_settings:
# for Cloud SQL, set this value to the Cloud SQL connection name,
# e.g. "project:region:cloudsql-instance"
cloud_sql_instances: "<project>:<region>:<sql-instance>"
You didn't mention it in your question but I believe you are using App Engine Flexible environment. Under the hood, App Engine flex apps run on (hidden from you) Compute Engine instances in your project. So it actually goes against Compute Engine quotas as well, including the "ip address in use" for your App Engine Region.
The "ip address in use" impacts your App Engine flex app in that it'll limit the number of instances your app will be able to scale up to, since each instance uses its own IP. For example, as per the app.yaml file you provided, your scaling setting defaults to automatic scaling with minimum 2 instances and maximum 20 instances. The "ip address in use" quota will prevent your app to upscale above 8 instances when the number of users using your app increases.
One other thing to note is that you may have previous versions of your service that are still running. If they have the same scaling setting, this means they'll have minimum 2 instances running each, which will count towards the "ip address in use" quota also.
Since you can't deploy your App Engine instances in a network in another region that the one you set for your App Engine app, the only solution here is to request a quota increase. In your Developer Console, got to IAM & admin > Quotas, select this particular quota and click on the "Edit Quotas" button at the top and follow the instructions.
Related
According to DataDog JVM metrics collection is enabled by default for Java tracer v0.29.0+
https://docs.datadoghq.com/tracing/metrics/runtime_metrics/java/
My agent is running and trace metrics are coming in fine, but I am not seeing the data coming in on the JVM metrics tab in the APM section.
I confirmed with DD helpdesk that everything is configured correctly for a containerized environment. I was expecting the JVM metrics to automatically like this doc describes:
https://docs.datadoghq.com/tracing/metrics/runtime_metrics/java/
app.yaml
DD_AGENT_HOST: "our_gcp_host"
DD_TRACE_AGENT_PORT: "80"
DD_ENV: "dev"
DD_SERVICE: "our_service_tag"
dd-app.yaml
service: dd-agent
runtime: custom
env: flex
env_variables:
DD_APM_ENABLED: "true"
DD_APM_NON_LOCAL_TRAFFIC: "true"
DD_APM_RECEIVER_PORT: 8080 // custom port configuration
DD_DOGSTATSD_NON_LOCAL_TRAFFIC: 'true'
DD_DOGSTATSD_PORT: 8125
network:
forwarded_ports:
- 8125/udp
I posted this so that I can answer this question. It was a few days of investigation, but we figured it out.
The solution is to deploy the agent to a compute engine instance. According to my colleague that figured it out the reason for this is:
Despite the fact app engine and docs say you can port forward, it looks like it doesn't actually allow the port to be accessible via the dns, just the ips which change as instances go up/down. We made a compute engine instance of the dd-agent and set our api to it's ip.
GCP isn't honest about port forwarding in App Engine. You can port forward but the app engine dns can't be used so you would have to use the instance ips. It also looks like udp load balancers may not work with app engine which makes the entire idea behind the port forwarding kinda pointless.
Try it out! We saw our metrics show up immediately.
I'm using Google App Engine flexible environment on PHP 7.2 application and we are using auto_scaling.
We occasionally get traffic spikes, but App engine will always keep 4 instances running.
What settings to add to app.yaml to force App Engine to always keep only one idle instance.
Thank you.
Since you are using App Engine Flex with auto scaling, you can't use the flag "max_idle_instances" in the app.yaml since this would give you an error; however, I would advice you to set a "max_num_instances" in order to prevent you from starting too many instances during one of your spikes.
I would also advice you to check out this video and this other one, as they explain very well how to set up your app.yaml to optimize your scaling.
Hope you find this helpful!
I have a python app engine that handles api results and it's stateful. However it seems that after a few hours of inactivity (no requests), the server shuts off, resetting all states, and when a new request is made, it's listening again.
But the states are reset. I want the server to actively remain unchanged 24/7 and not reset/restart as I want to maintain states.
I have configured as per documentation but it's still restarting, I am not sure what's wrong
Here is my app.yaml:
runtime: python37
entrypoint: python main.py
manual_scaling:
instances: 1
In App Engine the general recomendation is to create stateless applications as mentioned on the documentation
Your app should be "stateless" so that nothing is stored on the instance.
As an alternative for the application not to get restarted you can deploy it on Compute Engine, As that service is a Virtual Machine you can have total control of the states.
We configured our App Engine Standard python 3 service to connect to Cloud Memorystore via the Serverless VPC service (per the documentation, and other stack overflow threads). (I've included the app.yaml config below). This all worked well, unless an instance went idle for a little while. Over time we saw a high volume of:
Long unexplained hangs when making calls to Memorystore, even though they eventually worked
redis.exceptions.ConnectionError: Error 110 connecting to 10.0.0.12:6379. Connection timed out.
redis.exceptions.TimeoutError: Timeout reading from socket
These happened to the point where I had to move back to App Engine Flexible, where the service runs great without any of the above problems.
My conclusion is that Serverless VPC does not handle the fact that the redis client tries hard to leave the connection to redis open all the time. I tried a few variations of timeout settings, but nothing that helped. Has anyone successfully deployed App Engine Standard, Memorystore, and Serverless VPC?
env_variables:
REDISHOST: <IP>
REDISPORT: 6379
network:
name: "projects/<PROJECT-ID>/global/networks/default"
vpc_access_connector:
name: "projects/<PROJECT-ID>/locations/us-central1/connectors/<VPC-NAME>
Code used to connect to Memorystore (using redis-py):
REDIS_CLIENT = redis.StrictRedis(
host=REDIS_HOST,
port=REDIS_PORT,
retry_on_timeout=True,
health_check_interval=30
)
(I tried various timeout settings but couldn't find anything that helped)
I created a Memorystore instance and a Serverless VPC Access connector as stated in the docs (https://cloud.google.com/vpc/docs/configure-serverless-vpc-access), then deployed this sample (https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard_python37/redis) from Google Cloud Platform Python doc samples repo to App Engine Standard after making some modifications:
This is my app.yaml:
runtime: python37
# Update with Redis instance details
env_variables:
REDIS_HOST: <memorystore-ip-here>
REDIS_PORT: 6379
# Update with Serverless VPC Access connector details
vpc_access_connector:
name: 'projects/<project-id>/locations/<region>/connectors/<connector-name>'
# [END memorystore_app_yaml_standard]
I edited the code on main.py and used the snippet that you use to connect to the memorystore instance. It ended up like this:
redis_client = redis.StrictRedis(
host=redis_host, port=redis_port,
password=redis_password,
retry_on_timeout=True,
health_check_interval=30
)
I edited the requirements.txt. I changed “redis==3.3.8” for “redis>=3.3.0”
Things to note:
Make sure to use “gcloud beta app deploy” instead of “gcloud app deploy” since it is needed in order for the Serverless VPC Access connector to work.
Make sure that the authorized network you set to the memorystore instance is the same that you select for the Serverless VPC Access connector
This works as expected for me, could you please check if this works for you?
You may try to use min idle instance option, so you will have at least one idle instance to wait to serve your traffic. Bear in mind that this may change your billing cost. Also here you can find a billing calculator.
If the min idle instances are set to 0 there are no available instances to serve your traffic when the requests are starting and may this be the reason of having exceptions.
I have one python instance running in Google app engine (billing enabled). Now if my users increase and number of requests increase, then -
Google app engine automatically scales and handles all the requests or should I have to do anything like creating more frontend instances (to accommodate the increase in user traffic)?
App Engine will automatically scale the instances to accomodate for increased traffic.
You can change how instances are scaled up with the settings in the App Engine dashboard. Go to Application Settings > Performance and you can change a couple of settings to alter how instances are spun up. The settings you want to look at are:
Frontend Instance Class: How beefy your instances are.
Idle Instances: The minimum number of instances that will be waiting for traffic.
Pending Latency: How much latency will trigger spinning up a new instance.
App engine automatically scales base on your traffic, the only limit is your daily budget.
You should read more about warmup request and set application configuration but if it heavy traffic if not do care just pay more. You should think also about some sharding if it required.