Apologies for the seemingly obvious question, but I figure the answer might help others. I can't for the life of me find documentation on the filepath within the Google App Engine VM (Cloud Shell) where I can find the static files being served from. I need to pull the latest upstream changes from a private github repo.
Note that I navigated elsewhere in the VM and even restarting the session didn't put me in a default project root path within the VM as I expected it to.
There are several issues to address here:
The Cloud Shell is a virtual shell
Google Cloud Shell is an interactive shell environment for Google
Cloud Platform.
The environment where you're working is a container running in a VM in a Google-owned project inside GCP.
You can verify this by checking the metadata server (only available for GCP VMs):
curl -H 'Metadata-Flavor:Google' "http://metadata.google.internal/computeMetadata/v1/?recursive=true&alt=text"
In the metadata provided you'll see how this container is created and configured.
The Cloud Shell is tied to the user, so you'll always access the same environment if you access it with the same credentials, no matter the project. However, if you access with a different user, you'll get a different environment.
You can't access GAE standard instances
GAE is a fully managed environment, and you won't be able to access it. In this way, you won't be able to find the root of the running app engine project.
However, by the way GAE deploys your code, it uses a staging bucket to gather the code before compiling. You can find your staging bucket through the App Engine Admin API. This is usually staging.<PROJECT_ID>.appspot.com, although you can change this configuration. You can get your files from there.
You can access GAE flex apps
However, the deployment in flex gets your files, build a Docker container with them, and then deploys this container inside a VM.
As per the docs, you can connect directly to your container by running:
gcloud app instances ssh [INSTANCE-NAME] --service [SERVICE] --version [VERSION]
docker exec -it gaeapp /bin/bash
Regarding your issue
According what you say in the comments of the question, your issue could come from a myriad of places. From changing the shell you're connecting to, to resetting your shell environment (deleting all the files), to a thousand different possible problems.
The best way to think about it is regard the Cloud Shell as a temporal environment to run commands, but not as a virtual machine.
Knowing that, you could mount a persistent filesystem (GCS through GCSFuse, Cloud Filestore, ...) to persist your work, or simply use Git to have your work always synced on a repo.
GAE Flex has some nice CI integrations, so that's a plus for going the Git route.
Related
I'm running a local instance of a PHP app-engine project, I have got some buckets setup in GCP specifically for the local dev version, however instead of the data that I write to the bucket appearing online, they are being saved locally into the dev_appserver Datastore. I can see the files in the local admin interface (localhost:8000) under Datastore.
This is an issue as the application I'm developing has a companion app which needs to also access the bucket files.
The
--support_datastore_emulator=[true|false]
flag is documented under
dev_appserver.py -h
But it doesn't seem to have any effect when using =false.
So my question is: How do I stop the dev_appserver from using the local Datastore and make it use the 'real' buckets on the web?
Try setting the --default_gcs_bucket_name flag documented here to establish the default GCS bucket to use:
dev_appserver.py app.yaml --default_gcs_bucket_name gs://BUCKET-NAME
So it seems from a few SO questions I've seen that this is a problem among other users. Recently one of our head dev's left and I inherited a lot of his projects. One of which, is a website that what seems like lives on an app engine from google cloud platforms. From the App Engine documentation, to download source code you use the appcfg.py download_app command. Which I did, however the only results I get back from that call is:
Fetching file list...
Fetching files...
And then it just ends. No error message or any kind of message at all, and of course, it did not download the source code into the output dir I specified.
Scratching my head and looking at various SO posts, someone mentioned something about going into the google cloud vm directly and doing the same command, and to my surprise finding the same exact behavior that I did in my local terminal.
This made me realize it must be something else at play. I took a look at my versions tab in the App Engine dashboard on GCP. I see my instance running, it correctly says Serving and if I click the link it brings me to the website which loads fine. However, under Size it says 0 B which made me think perhaps this is why the download_app isn't downloading anything, because the version is 0 B?
What I'm trying to figure out is why it says 0 B for the version, when clearly the site runs fine and how I can get the source code for this. Here's a screenshot for reference
And screenshot of my terminal (local). Obviously I omitted the -A and -V flags, but they are correctly set and if I purposely make them incorrect I do indeed get an error message.
EDIT
Just so everyone is aware, I also made sure my user had the correct permissions. Owner, App Engine Owner... and some others. I don't think that's the problem.
When you deploy an App Engine Flexible application, the source code is uploaded to Cloud Storage on your project in a bucket named staging.<project-id>.appspot.com. You can navigate in this bucket and download the source code for a specific version as a .tar file.
Alternatively, you can find the exact Cloud Storage URL for your source code by going to Dev Console > Container Registry > Build History and select the build for your version. You'll find the link to your source code under Build Information.
One thing to note however is that the staging... bucket is created by default with a Lifecycle rule that deletes files older than 15 days automatically. You can delete this rule if you want so that all versions' source code is kept indefinitely.
In your case I believe that may not have helped since files may have been deleted already but it's worth knowing you can get the source code from there (source code isn't pushed to Source Repository by default, your developer had to configure it manually).
Posting this since none of the listed methods on the web didn't take me to the code (by June 2021)
Note: appcfg.py is deprecated by Google
You could try accessing your source code through;
Google Cloud Platform > Debugger > choosing the version of the
Application from combo at top.
This will list the files of that version on the left pane. There is no way to download code automatically but you can copy-paste the code.
Advice: Push your code to a Git repository to avoid this hassle next time.
Hope you will find this helpful.
In the developer console you can select the respective project and check:
on the Services page - which services, AKA modules - as they used to be (and still are) called in various places, you app has deployed
on the Versions page - which versions for each of the services are deployed
This information is what appcfg.py download_app expects. See also:
the various appcfg.py options using its --help flag
How do I download a specific service's source code off of AppEngine?
You can also access the deployed source code live (if everything else fails it could still be a last resort method to get the code, but tedious), see my answer to Google Cloud DataStore automatic indexing
Update:
I just now noticed in your screenshot that it's a flexible environment app. The appcfg.py docs are in the standard environment section, I suspect it's not applicable to the flexible environment, for which what's deployed is actually a docker image built during the deployment operation. From Deploying your application:
Deploy your app to App Engine using the gcloud app deploy
command. This command automatically builds a container image by using
the Container Builder service and then deploys that image to the
App Engine flexible environment. The container will include any local
modifications that you've made to the runtime image.
It might be possible to access the code on the actual GCE instance running the app, by connecting to the running instance and starting a shell in your app container, see Connecting to the instance
(Background: I am new to Google App Engine, familiar with other cloud providers' services)
I am looking for access/view similar to shell access to production node.
With a Python/Django based Google App Engine App, I would like to view the code in production.
One view I could find is the StackDriver 'Debug' view.
However, apparently the code shown in the Debug view doesn't reflect the updated production code (based on what is showing on the production site, for example, the text on the home page different).
Does Google App Engine allow me to ssh into the VM where the application/code is running?
If not, how can check the code that's running in production?
Thanks.
According to the SSH debugging row in the Comparing environments table SSH access is supported for flex environment apps but not for standard environment apps.
From Connecting to the instance:
If a VM instance is in debug mode, you can connect to its host by
using SSH in the console or with gcloud.
To connect to an instance in the console:
Visit the Cloud Platform Console instances page for your project:
Go to the instances page
Click SSH in the far right of the row containing the instance you want to access:
This puts the instance into debug mode, and opens an SSH session for the instance in a terminal window.
You can also select different options to start an SSH session from the drop-down list.
At this point you are in the instance host, which has several containers running in it. See Understanding common
containers next for more information about these.
In the terminal window, list the containers running in the instance:
sudo docker ps
The output of the sudo docker ps command lists each container by row; locate the row that contains your project ID: this is the
container running your code. Note the NAME of this container.
Optionally, list logging information for your application by invoking:
sudo docker logs [CONTAINER-NAME]
Start a shell in the container that is running your code:
container_exec [CONTAINER-NAME] /bin/bash
When finished debugging, enter exit to exit the container, then exit again to exit the SSH session.
Disable debugging for your instance to allow it to resume normal operation.
If you are using the standard environment, the answer is no, you can't really inspect or see the code directly. You've mentioned looking at it via Stackdriver Debugger, which is one way to see a representation of it.
It sounds like if you have a reason to be looking at the code, then someone in your organization should grant you the appropriate level of access to your source code management system. I'd imagine if you're deployment practices are mature, then they'd likely branch the code to map to your deployed versions and you could inspect in detail locally.
Is there a way to update selected files when using the App Engine Flexible env?
I'm facing an issue whenever I do a small change in the app.yaml file: to test it I would need to deploy the whole application which takes ~5mins.
Is there a way to update only the config file? OR is there a way to test these files locally.
Thanks!
The safe/blanket answer would be no as the flex env docker image would need to be updated regardless of how tiny the changes are, see How can I speed up Rails Docker deployments on Google Cloud Platform?
However, there might be something to try (YMMV).
From App Engine Flexible Environment:
You always have root access to Compute Engine VM instances. SSH access to VM instances in the flexible environment is disabled by
default. If you choose, you can enable root access to your app's VM
instances.
So you might be able to login as root on your GAE instance VM and try to manually modify a particular app artifact. Of course, you'd need to locate the artifact first.
Some artifacts might not even be present in the VM image itself (those used exclusively by the GAE infra, queue definitions, for example). But it should be possible to update these artifacts without updating the docker image, since they aren't part of the flex env service itself.
Other artifacts might be read-only and it might not be possible to change them to read-write.
Even if possible, such manual changes would be volatile, they would not survive an instance reload (which would be using the unmodified docker image), which might be required for some changes to take effect.
Lots of "might"s, lots of risks (manual fiddling with the app code could negatively impact its functionality), up to you to determine if a try is really worthy.
Update: it seems this is actually documented and supported, see Accessing Google App Engine Python App code in production
I am writing files to Google app engine managed vm (flexible environment). I have deployed the code to cloud. The code opened the file and write it without show errors like permission errors. However, my another code trying to open the file failed with "No such file or directory".
I have printed the location directory, so I used ssh command, also cannot find the file inside managed vm. And I cannot find any documentation about writing files to local storage of managed VM.
So how to write it to managed vm? What is the default storage location? Why I cannot find the file?
The reason you can't find the file inside the VM is because the VM is running a Docker container, and your app is actually inside that container. If you really wanted to, you could docker exec -it <container-name> -- /bin/bash to poke around inside the container and then you should see your file (use docker ps to find the container ID).
On App Engine flexible (new name for Managed VMs), your code could be scaled across many container so there's no promise that a file you write to one will be accessible later. You can get away with dealing with some writing to some /tmp files but not much more. You will be much better off writing any data to something like Cloud Datastore, Cloud SQL, Memcache, or Cloud Storage.