I have a java app deployed to GCP App Engine with a jar. I tested with a load testing framework to generate steady state 100 QPS of traffic. I have also set the maximum instance count to 1. I am following the automatic scaling.
When I go to "instances" page, it says "This instance is being restarted" all the time. But all requests are succeeding.
Why is the instance being restarted while serving requests?
How do I know the cause for this restart?
Related
I have a golang gorilla websocket server hosted on Google App Engine (GAE) that randomly disconnects connected clients after about an hour of being connected (without any activity).
I checked GAE logs and compared timestamps of close events (code 1006) and noticed that the time differences between opening and closing connections are all nearly the same (~60.1 minutes).
Is there a proper way to fix this issue (example: update associated nginx config) for Google App Engine?
I'm fairly certain that this an issue with GAE as I'm unable to reproduce this issue locally/in-development.
I couldn't find any other good ways of hosting an autoscaled/managed server that supports the websocket protocol.
My alternative hosting choice may be kubernetes, but the costs are much higher than GAE AFAIK and K8s is slightly more difficult to manage.
I found a few related stackoverflow links with no proper solutions (mostly migrations to other services or hacks)
example: GAE App Engine Websocket clients are disconnected after 1 hour
It's well documented on GCP docs :
WebSockets are always available to your application without any additional setup. Once a WebSockets connection is established, it will time out after one hour.
I wouldn't suggest creating a Websocket server on a serverless platform because these products tend to scale down over time when there's no requests on it so you cannot guarantee that it will continue holding the connection (also along with the hard limit mentioned).
You can host your Websocket server on Compute Engine. In fact, GAE Flex uses Compute Engine autoscaler. Here's a link if you're interested to see how it works.
I have a Python GAE flex app, and I received an email from Google stating:
We are writing to let you know that the v0.1 and v1beta1 endpoints of
the Compute Engine Metadata Server will be shut down on January 15,
2020. From January 15, 2020, requests to the v0.1 and v1beta1 endpoints will no longer be supported, and may return HTTP 404 NOT
FOUND responses.
The email also stated that my app used these old metadata endpoints in the last 90 days.
I had never heard of the Compute Engine Metadata Server before so it is not something that I have intentionally used. I suppose it is possible that one of my dependencies (Flask etc.) but it seems unlikely.
Any idea where my GAE Flex app might be making a call to the Compute Engine Metadata Server? Is this a false alarm from Google?
No, it is not false alarm. As it writes in the public documentation:
v1beta1 server and v0.1 metadata server endpoints are deprecated and scheduled for shutdown. Ensure that you update all requests to use v1. For more information, see Transitioning to the v1 metadata server endpoint.
The metadata server is the place on where the GCE instances are storing their metadata. Your App Engine Flex app is running on compute instances. You can use the Metadata server to query information about the instance like : ip, instance id, service account info etc..
App Engine Flex uses the metadata server in the back scene, so, even though you did not use the Metadata Server explicitly, app engine does it. Although App Engine flex should do all the updates by itself and it is less probable for you to face any issues,it is recommended to migrate from v0.1 to v1 metadata server endpoint just to be sure. Here
there is a guide in the official documentation on how to achieve that.
I just deployed the helloworld app on Google App Engine Flexible Environment.
Randomly, let's say, one each 3-4 requests, I get this:
The only change I've made from the original helloworld code provided in google cloud docs was that I changed the automatic scaling to manual and set it to 1 instance.
Anyways, I deployed the same docker image in Container Engine, and it works perfect.
There is nothing on the app logs, the request does not reach the app.
Something is not working with GAE Flexible Env. load balancer, any ideas?
I destroyed all the instances and deleted the service. After 24hs tried to create it again and now it is working good.
We run 2 apps on Google AppEngine, one in Python, and one in Java.
The Python app faces the internet and serves browser users or API calls from our smartphone app.
The Python app often communicates with the Java app, which offers workflow engine services.
The Java app is never accessed directly from the internet. This is similar to a service oriented architecture.
Our Java app is configured with 1 resident instance.
However, we often see that when the Python app sends a request to the Java app (which is doing nothing at that moment), that a new Java instance is spawned, instead of having the resident instance serve the request. This way, latencies up to 10 seconds were discovered, whereas the request handling itself takes less than 1 second.
Could it be possible that Google considers a request from one appengine app to another one, as a sort of backend workload, for which throughput is more important than latency? Are there different policies for requests coming from the internet versus requests coming from Google internally (defered methods, cron, ... maybe inter-appengine requests also resorts in this category, which is maybe not always correct).
Google recommended me to run the Java app as a backend of the Python app, and in fact have only 1 single Appengine app, instead of 2.
Some information in this thread:
https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/8O7K3cFzBbY
Any advice on how to force appengine resident instances handle incoming requests, rather than spawn and warm up new instances, which might be good for throughput, but bad for latency?
Have tried to add <threadsafe>true</threadsafe> element to appengine-web.xml on your java app?
I'm wondering if anyone knows any script in Python or Java to setup Google App Engine as Load Balancer server?
I choose GAE, because of 100% uptime and free, so it would be the best to setup as Load balancer.
So what I want to do is set up a GAE app (a load balancer application) in "www.domain.com" - using Google Apps, it will share the load to all my servers (with round-robin, or any technique), but when all servers are down, show maintenance page which is host in GAE.
Is this possible?
You could do it, but you'd
incur extra latency by requiring an extra hop from google app engine to your servers
have all your servers public to the web, since otherwise google app engine won't be able to redirect to them Edit: apparently this is incorrect. See Nick Johnson's comment below!
have to implement something in google app engine to keep track of what servers you have up and how and when to route to them
Seems like it might not be worth it, but it'd be cool if you did it and posted some numbers about how it works!