Our Google Appengine flex app has configured HeapDumpOnOutOfMemoryError.. we did have a few crashes due to OOM. But where can we find the heap dump?
Right now, in App Engine, there isn't a way to track heap dump logs directly using Stackdriver - the tool used for logging in App Engine. The developers at Google are aweare of it and you can track the Feature Request for this functionality here.
In alternative to that, you can try one thing. As indicated in this other similar case here, you can enable the debug mode in your instance and then, connect to it via SSH. Once you connect via SSH, you need to run the following commands.
sudo docker ps - to list the containers of your instance and find the one related to your project ID;
sudo docker logs [CONTAINER-NAME] - with the name got from the last command, get the logs here;
Or start a shell to check it further with sudo docker exec -it [CONTAINER-NAME] /bin/bash
All these steps are further explained and with more information in the official documentation: Debugging an instance
Let me know if the information helped you!
Related
I have read the document which says that gcloud app versions migrate operation can migrate traffics into another version, but it couldn't. The error description seems to say something informative, but no help to me. On GCP control panel, version migration works fine. What's happening on this?
$ gcloud app versions migrate 20190122-120543-ebbfbc9 -s q
Migrating all traffic from version [q/20190129-164614-09e8288] to
[q/20190122-120543-ebbfbc9]
Do you want to continue (Y/n)?
ERROR: (gcloud.app.versions.migrate) Issues migrating all traffic of service(s): [q]
INVALID_ARGUMENT: Invalid request.
- '#type': type.googleapis.com/google.rpc.BadRequest
fieldViolations:
- description: Warmup requests must be enabled for all versions that will gain additional
traffic as a result of the traffic migration.
field: service.split.allocations[20190122-120543-ebbfbc9]
When you use gcloud app versions migrate it will try to do a gradual migration. Gradual traffic migration is only supported for the Standard environment and you have to enable warmup requests on the target version for this to work.
It’s working on GCP Console because there you’ll be prompted to do an immediate migration if the above conditions are not met.
If you want to use gcloud to migrate traffic to a GAE Standard version set to warmup disabled, or Flex environment, you can use the following command which will actually split the traffic immediately (not gradually) 100% to the target version:
gcloud app services set-traffic [MY_SERVICE] --splits [MY_VERSION]=1
You can find more information here
Hope it helps! :)
Apologies for the seemingly obvious question, but I figure the answer might help others. I can't for the life of me find documentation on the filepath within the Google App Engine VM (Cloud Shell) where I can find the static files being served from. I need to pull the latest upstream changes from a private github repo.
Note that I navigated elsewhere in the VM and even restarting the session didn't put me in a default project root path within the VM as I expected it to.
There are several issues to address here:
The Cloud Shell is a virtual shell
Google Cloud Shell is an interactive shell environment for Google
Cloud Platform.
The environment where you're working is a container running in a VM in a Google-owned project inside GCP.
You can verify this by checking the metadata server (only available for GCP VMs):
curl -H 'Metadata-Flavor:Google' "http://metadata.google.internal/computeMetadata/v1/?recursive=true&alt=text"
In the metadata provided you'll see how this container is created and configured.
The Cloud Shell is tied to the user, so you'll always access the same environment if you access it with the same credentials, no matter the project. However, if you access with a different user, you'll get a different environment.
You can't access GAE standard instances
GAE is a fully managed environment, and you won't be able to access it. In this way, you won't be able to find the root of the running app engine project.
However, by the way GAE deploys your code, it uses a staging bucket to gather the code before compiling. You can find your staging bucket through the App Engine Admin API. This is usually staging.<PROJECT_ID>.appspot.com, although you can change this configuration. You can get your files from there.
You can access GAE flex apps
However, the deployment in flex gets your files, build a Docker container with them, and then deploys this container inside a VM.
As per the docs, you can connect directly to your container by running:
gcloud app instances ssh [INSTANCE-NAME] --service [SERVICE] --version [VERSION]
docker exec -it gaeapp /bin/bash
Regarding your issue
According what you say in the comments of the question, your issue could come from a myriad of places. From changing the shell you're connecting to, to resetting your shell environment (deleting all the files), to a thousand different possible problems.
The best way to think about it is regard the Cloud Shell as a temporal environment to run commands, but not as a virtual machine.
Knowing that, you could mount a persistent filesystem (GCS through GCSFuse, Cloud Filestore, ...) to persist your work, or simply use Git to have your work always synced on a repo.
GAE Flex has some nice CI integrations, so that's a plus for going the Git route.
(Background: I am new to Google App Engine, familiar with other cloud providers' services)
I am looking for access/view similar to shell access to production node.
With a Python/Django based Google App Engine App, I would like to view the code in production.
One view I could find is the StackDriver 'Debug' view.
However, apparently the code shown in the Debug view doesn't reflect the updated production code (based on what is showing on the production site, for example, the text on the home page different).
Does Google App Engine allow me to ssh into the VM where the application/code is running?
If not, how can check the code that's running in production?
Thanks.
According to the SSH debugging row in the Comparing environments table SSH access is supported for flex environment apps but not for standard environment apps.
From Connecting to the instance:
If a VM instance is in debug mode, you can connect to its host by
using SSH in the console or with gcloud.
To connect to an instance in the console:
Visit the Cloud Platform Console instances page for your project:
Go to the instances page
Click SSH in the far right of the row containing the instance you want to access:
This puts the instance into debug mode, and opens an SSH session for the instance in a terminal window.
You can also select different options to start an SSH session from the drop-down list.
At this point you are in the instance host, which has several containers running in it. See Understanding common
containers next for more information about these.
In the terminal window, list the containers running in the instance:
sudo docker ps
The output of the sudo docker ps command lists each container by row; locate the row that contains your project ID: this is the
container running your code. Note the NAME of this container.
Optionally, list logging information for your application by invoking:
sudo docker logs [CONTAINER-NAME]
Start a shell in the container that is running your code:
container_exec [CONTAINER-NAME] /bin/bash
When finished debugging, enter exit to exit the container, then exit again to exit the SSH session.
Disable debugging for your instance to allow it to resume normal operation.
If you are using the standard environment, the answer is no, you can't really inspect or see the code directly. You've mentioned looking at it via Stackdriver Debugger, which is one way to see a representation of it.
It sounds like if you have a reason to be looking at the code, then someone in your organization should grant you the appropriate level of access to your source code management system. I'd imagine if you're deployment practices are mature, then they'd likely branch the code to map to your deployed versions and you could inspect in detail locally.
To run PHPUnit tests on Google Cloud's App Engine, I understand I would need to run a command on the command line like for example phpunit --bootstrap src/autoload.php tests/MoneyTest.
I tried opening the Google Cloud Shell in the Cloud Console, but I cannot seem to locate the code that I uploaded to be able to execute that command. When I use find -name "index.php" it shows me directories like ./etc/docker and ./run/dockerand a bunch of other locations, but all return a Permission denied message.
On the other hand, I tried creating a special webpage test.php that would just run the tests that I wanted to do, but I get a timeout error since the tests take way to long to run and it gets cut off after some seconds.
How could I use PHPUnit tests on the AppEngine, and not only locally on my machine?
App Engine is not the best tool to solve this problem because App Engine is designed to handle short-lived HTTP requests, not running long processes. Yes, there are ways run jobs of up to 10 minutes, but at some point you'll hit that timeout too.
Google Cloud Platform offers better solutions for long running processes. The most straight-forward is Google Compute Engine where you can run a dedicated Linux instance.
I'm just going through the node.js tutorials with a free trial account, and i'm stuck on the second one where you add a db. I add the mongodb deployment, shows up as a VM instances, fine. And my first deploy worked, but now that i'm trying to edit stuff, my deploy's keep failing.
The error i get is that I've exceeded my CPU quota. Watching the list of VM Instances under Compute Engine, i see it keeps spawning up instances, even though the app isn't being used. Guessing it just spins up 8 instances by default?
But then i guess the build system needs its own VM's, but the CPU capacity is used up, so none available to do subsequent builds?! I feel like i'm missing something...
Also, i see i can explicitly start VM's myself, so what process is creating them form me? And can i turn it off? or set a cap on number of instances it spawns?
Can i tell my project to only use 4
Also, the deploy takes forever, is that normal? Following the tutorials, so far I've only seen this command to deploy:
gcloud preview app deploy app.yaml --set-default
Is there another command that does an incremental deploy or something?
By using gcloud preview app deploy you're actually using Managed VMs which is an App Engine runtime which in turn runs Docker containers on Google Compute Engine, which it creates on its own. In other words, you're not using Google Compute Engine directly.
To get rid of extra VMs, you need to delete old app versions: navigate to Compute > App Engine > Versions and delete the versions you don't want.
See also this answer for more details and suggestions.