I'm using Google App Engine and I've followed the instruction to setup Google Debugger as show here:
https://cloud.google.com/debugger/docs/setup/nodejs
My project is written in Node.js and I'm starting the service with this line:
require('#google-cloud/debug-agent').start({serviceContext: {enableCanary: true}});
I've also generated the source-context file.
When I go to the debugger console (https://console.cloud.google.com/debug) I see the green tick about the debug agent running but nothing is displayed on the left side, under "Deployed files".
Any ideas?
What you're seeing is intended behavior. Deployments to App Engine Standard will show the source code as the source files are included on deployment and not using a Docker container while deployments to App Engine Flex are using a Docker container, which currently need not to include the source files, hence why you need a source context information which is generated from a Git Repository. For more information about differences of standard and flex, please see Choose an App Engine environment.
For App Engine Flex app source code to appear on Cloud Debugger, it must be located on a remote code hosting provider. Make sure that you follow the Selecting source code automatically properly. Your main source code (js, yaml, and source-context.json) should be in the root directory of the repository as shown below.
App-directory/
main.py
app.yaml
source-context.json
I've successfully deployed sample application using Github showing the source codes on the left side as shown below.
Some important notes:
When using "Cloud Source Repositories" or "GitHub" or any of the other remote code hosting options, one must take some care:
Changes to the source file must be committed to version control before running gcloud debug source gen-repo-info-file.
gcloud debug source gen-repo-info must be run before deploying the App.
Commits must have been pushed to Cloud Source Repository/GitHub/etc before running Cloud Debugger.
Access must have been granted to the repo to the Cloud Debugger application.
Always redeploy application when there are changes to the repository or source code.
Related
This question already has answers here:
How do I download the source code of a google app engine project?
(12 answers)
Closed 1 year ago.
What are the alternatives to download source code from app engine instead of using appcfg.py?
Yes, you are correct, downloading source code using appcfg.py is deprecated. The alternative way is by downloading in Cloud Storage bucket named staging.project-id.appspot.com but the name of the files are encrypted. For you to easy track the files use the following gcloud command:
To view all versions and services in App Engine:
gcloud app versions list
Select the version ID and service name from the output then execute it in the gcloud command below:
gcloud app versions describe VERSIONID -s SERVICENAME
From the output, look for:
deployment:
files:
filesample.txt:
sha1Sum: encryptedname
sourceUrl: https://storage.googleapis.com/staging.project-id.appspot.com/encryptedname
To download click the output of sourceUrl.
There's also a Feature Request for downloading source code for app engine, you can star the public issue tracker feature requests to ensure that you will receive the updates about it. But please note that there is no guarantee that this feature will be implemented.
I did compile a few options to fetch App Engine's code in this repo, but to sum it up:
If you use App Engine Standard:
Try to look in Stackdriver Debugger
The staging bucket contains the files used on deployment time, you can get them from there (although it's not a good idea if there are many files deployed).
In case of App Engine Flex:
SSH into any of the VMs and then download the files directly from there
Fetch the image from GCR that was created when deploying so you can download the files from there
Apologies for the seemingly obvious question, but I figure the answer might help others. I can't for the life of me find documentation on the filepath within the Google App Engine VM (Cloud Shell) where I can find the static files being served from. I need to pull the latest upstream changes from a private github repo.
Note that I navigated elsewhere in the VM and even restarting the session didn't put me in a default project root path within the VM as I expected it to.
There are several issues to address here:
The Cloud Shell is a virtual shell
Google Cloud Shell is an interactive shell environment for Google
Cloud Platform.
The environment where you're working is a container running in a VM in a Google-owned project inside GCP.
You can verify this by checking the metadata server (only available for GCP VMs):
curl -H 'Metadata-Flavor:Google' "http://metadata.google.internal/computeMetadata/v1/?recursive=true&alt=text"
In the metadata provided you'll see how this container is created and configured.
The Cloud Shell is tied to the user, so you'll always access the same environment if you access it with the same credentials, no matter the project. However, if you access with a different user, you'll get a different environment.
You can't access GAE standard instances
GAE is a fully managed environment, and you won't be able to access it. In this way, you won't be able to find the root of the running app engine project.
However, by the way GAE deploys your code, it uses a staging bucket to gather the code before compiling. You can find your staging bucket through the App Engine Admin API. This is usually staging.<PROJECT_ID>.appspot.com, although you can change this configuration. You can get your files from there.
You can access GAE flex apps
However, the deployment in flex gets your files, build a Docker container with them, and then deploys this container inside a VM.
As per the docs, you can connect directly to your container by running:
gcloud app instances ssh [INSTANCE-NAME] --service [SERVICE] --version [VERSION]
docker exec -it gaeapp /bin/bash
Regarding your issue
According what you say in the comments of the question, your issue could come from a myriad of places. From changing the shell you're connecting to, to resetting your shell environment (deleting all the files), to a thousand different possible problems.
The best way to think about it is regard the Cloud Shell as a temporal environment to run commands, but not as a virtual machine.
Knowing that, you could mount a persistent filesystem (GCS through GCSFuse, Cloud Filestore, ...) to persist your work, or simply use Git to have your work always synced on a repo.
GAE Flex has some nice CI integrations, so that's a plus for going the Git route.
So it seems from a few SO questions I've seen that this is a problem among other users. Recently one of our head dev's left and I inherited a lot of his projects. One of which, is a website that what seems like lives on an app engine from google cloud platforms. From the App Engine documentation, to download source code you use the appcfg.py download_app command. Which I did, however the only results I get back from that call is:
Fetching file list...
Fetching files...
And then it just ends. No error message or any kind of message at all, and of course, it did not download the source code into the output dir I specified.
Scratching my head and looking at various SO posts, someone mentioned something about going into the google cloud vm directly and doing the same command, and to my surprise finding the same exact behavior that I did in my local terminal.
This made me realize it must be something else at play. I took a look at my versions tab in the App Engine dashboard on GCP. I see my instance running, it correctly says Serving and if I click the link it brings me to the website which loads fine. However, under Size it says 0 B which made me think perhaps this is why the download_app isn't downloading anything, because the version is 0 B?
What I'm trying to figure out is why it says 0 B for the version, when clearly the site runs fine and how I can get the source code for this. Here's a screenshot for reference
And screenshot of my terminal (local). Obviously I omitted the -A and -V flags, but they are correctly set and if I purposely make them incorrect I do indeed get an error message.
EDIT
Just so everyone is aware, I also made sure my user had the correct permissions. Owner, App Engine Owner... and some others. I don't think that's the problem.
When you deploy an App Engine Flexible application, the source code is uploaded to Cloud Storage on your project in a bucket named staging.<project-id>.appspot.com. You can navigate in this bucket and download the source code for a specific version as a .tar file.
Alternatively, you can find the exact Cloud Storage URL for your source code by going to Dev Console > Container Registry > Build History and select the build for your version. You'll find the link to your source code under Build Information.
One thing to note however is that the staging... bucket is created by default with a Lifecycle rule that deletes files older than 15 days automatically. You can delete this rule if you want so that all versions' source code is kept indefinitely.
In your case I believe that may not have helped since files may have been deleted already but it's worth knowing you can get the source code from there (source code isn't pushed to Source Repository by default, your developer had to configure it manually).
Posting this since none of the listed methods on the web didn't take me to the code (by June 2021)
Note: appcfg.py is deprecated by Google
You could try accessing your source code through;
Google Cloud Platform > Debugger > choosing the version of the
Application from combo at top.
This will list the files of that version on the left pane. There is no way to download code automatically but you can copy-paste the code.
Advice: Push your code to a Git repository to avoid this hassle next time.
Hope you will find this helpful.
In the developer console you can select the respective project and check:
on the Services page - which services, AKA modules - as they used to be (and still are) called in various places, you app has deployed
on the Versions page - which versions for each of the services are deployed
This information is what appcfg.py download_app expects. See also:
the various appcfg.py options using its --help flag
How do I download a specific service's source code off of AppEngine?
You can also access the deployed source code live (if everything else fails it could still be a last resort method to get the code, but tedious), see my answer to Google Cloud DataStore automatic indexing
Update:
I just now noticed in your screenshot that it's a flexible environment app. The appcfg.py docs are in the standard environment section, I suspect it's not applicable to the flexible environment, for which what's deployed is actually a docker image built during the deployment operation. From Deploying your application:
Deploy your app to App Engine using the gcloud app deploy
command. This command automatically builds a container image by using
the Container Builder service and then deploys that image to the
App Engine flexible environment. The container will include any local
modifications that you've made to the runtime image.
It might be possible to access the code on the actual GCE instance running the app, by connecting to the running instance and starting a shell in your app container, see Connecting to the instance
I have a website that is currently running under GAE... unfortunately, I, nor anyone on the team, does not have access the local environment that it was created from.... Is it possible to create a local environment or at least get a copy of the application files and database from an existing GAE installation?
What you need is the application source code, not the "local environment".
Ideally this source code would be on a version control system (ie GIT,SVN), Google cloud platform provides free GIT repositories for your projects so you might try looking there first. There's also a tool for both Java and python that allow you to download the source of a deployed version, provided you are authenticated as either the dev who uploaded it or a project owner. EDIT: as stated by Dan Cornilescu this feature can be disabled.
As for the database info there's plenty of tools available to "export" your GAE datastore info, just consider for your project that it might be easier to do the queries manually than actually implementing this tools.
Thanks for help... But unfortunately, this code is not in GIT. Furthermore,
being new to Google hosting, I wasn't clear on my setup... My web instance is actually running within Compute Engine not Application Engine. Be that as it may, with some additional search, I was first able to find out how to browse my filesystem by accessing the VM Instances menu option under the Compute Engine section of the Google Cloud Platform interface. On the VM Instances page, it will show your instance and an option to the left side of the instance to connect with a drop down box that will allow you to open a browser window that shows the instance's file system. In addition to this, I found this link https://www.youtube.com/watch?v=9ssfE6ODpak that shows how to configure Filezila FTP client to access your server instance - very helpful. From there, I was able to download all of my site files from the var/www directory. Now, onto extracting my data... Thanks again!
I am trying to follow the instructions for running a simple new Google App Engine web application locally (without Google Web Toolkit, just the Web App) named "tunes".
I am following these instructions. Step one is to make a run configuration. I made one using all the defaults; I checked that under the Server tab the "Run built-in Server" box is checked. However, when I Run the app, I get the Console; right below the console tab it says
<terminated> tunes [Web Application] C:\Program Files (x86)\Java\jre7\bin\javaw.exe (Feb 26, 2014, 5:35:44 PM)
then below the line is what looks like a classic Unix "Usage" error message in the console, in red type, whose first line is
Usage: <dev-appserver> [options] <app directory>
followed by a list of options, and then nothing else happens that I can see.
I tried pointing a browser at http://localhost:8888/tunes as suggested by the documentation, but Firefox could not find a server active at that port.
Under the Arguments tab in the Run Configuration is the following:
"-codeServerPort 9997 --port=8888 org.tunes.gaeproject.Tunes C:\Users\cdf\java\eclipse4.3.1workspace\tunes\war"
I can successfully deploy the web application to the Google App Engine site and run it there.
What do I have misconfigured?
Google AppEngine application is not meant to run on local server, and neither you could create its docker image etc to deploy it anywhere you wish.
Instead I suggest you to port your application to Google Compute Engine (GCE) first within your deployment setup, which might require minor code refactorings, and Kubernetes kinda auto-scaling deployment will functional equivalent to what you have now. But with approach you may port the application easily to local server setup, or a docker image to run from any virtualization environment