I have a Java 11 GCP App Engine project and I'm trying to use different app.yaml files depending on the environment (e.g., app-dev.yaml, app-prod.yaml, etc). The yaml files in the /appengine directory like src/main/appengine/app-dev.yaml and so on.
There is an SO post about this already, but the answer doesn't work because it clobbers the descriptor which in Java should be the pom.xml (see my Approach 2 for more information).
Approach #1
UPDATE: Solved! In order to use this approach you must be on gcloud 298.0.0+
First, I tried using the --appyaml=APPYAML argument found in https://cloud.google.com/sdk/gcloud/reference/app/deploy#--appyaml:
gcloud app deploy [DEPLOYABLES …] [--appyaml=APPYAML] [--bucket=BUCKET] ...
I ran the following and received an error that the appyaml argument isn't recognized.
$ gcloud --project=my-project app deploy --appyaml=app-dev.yaml
ERROR: (gcloud.app.deploy) unrecognized arguments: --appyaml=app-dev.yaml
The fully qualified path to app-dev.yaml doesn't work either.
Approach #2
Next I found a slighly different syntax in https://cloud.google.com/appengine/docs/flexible/java/configuring-your-app-with-app-yaml that looks like this:
gcloud app deploy service-name-app.yaml
I tried the same locally but pointed to my custom app-dev.yaml like so, but it breaks:
$ gcloud --project=my-project app deploy src/main/appengine/app-dev.yaml
...
descriptor: [/Users/SomeDev/IdeaProjects/my-project/app-server/src/main/appengine/app-dev.yaml]
source: [/Users/SomeDev/IdeaProjects/my-project/app-server/src/main/appengine]
target project: [my-project]
target service: [default]
target version: [20200831abcdefg]
target url: [https://my-project.uc.r.appspot.com]
This breaks because it thinks the app-dev.yaml is the descriptor file instead of a pom.xml, so it errors out with the following:
Error message: did not find any jar files with a Main-Class manifest entry
To compare, I ran a normal deployment without a custom yaml file and you can see the pom.xml is the value of the descriptor.
$ gcloud --project=my-project app deploy
...
descriptor: [/Users/SomeDev/IdeaProjects/my-project/app-server/pom.xml]
source: [/Users/SomeDev/IdeaProjects/my-project/app-server]
target project: [my-project]
target service: [default]
target version: [20200831abcdefg]
target url: [https://my-project.uc.r.appspot.com]
Is there a recommended way to make this work, or is this the wrong approach entirely?
Looking at your "Approach #1" you have to upgrade your gcloud to version >= 298.0.0 here --appyaml parameter have been added - quite recently in Jun'20.
Looking at your "Approach #2". If you are running gcloud app deploy (without parameters) it search for descriptor app.yaml in current directory and if not found - than for pom.xml. If you want to use pom.xml from different localization you have to remove it from current directory. I didn't test it to the very end, just tested the descriptor value in summary.
Anyway I don't think using above is best way to do it. When you use pom.xml as descriptor it means that you are using feature called "deploy your Maven project as source code". Which is not main way to deploy app engine with maven.
According to my understanding if maven was used for build, its possible to use the jar in entrypoint of app.yaml file (reference) or maven goal appengine:deploy (reference + article that should be interesting).
dev_appserver.py starts a local deployement of my appengine service. I want to run my tests on behave on this local service. I want to start the server within my tests first. How to run the dev_appsrrver.py app.yaml command in my behave feature file in the start ?
I have tried subprocess.run("python","dev_appserver.py") but it says couldnt find the file dev_appserver.py. I'm trying on windows.
When you're attempting to launch executables using subprocess methods typically you're not getting by default the same environment (execution path and current working directory) you're getting yourself in a shell/terminal. Which means you may need to reference files (both executables and regular files) using full paths in the list of arguments you pass to those methods.
Since the subprocess.run() execution complaints about the dev_appserver.py location it means it's finding python OK (you may still want to check that's it's the 2.7 version) and you need to provide the full path for dev_appserver.py, which depends on your OS and the SDK you use. On Linux, for example (sorry, I'm not a windows guy), the path is:
<GAE_SDK_dir>/dev_appserver.py if using the GAE SDK
<gcloud_SDK_dir>/bin/dev_appserver.py if using the gcloud SDK
You'll most likely need to pass the path to your GAE app's app.yaml file, too - as an argument to dev_appserver.py, otherwise you'll see it complain about inability to locate the app or its files (or just having things run badly - if the app.yaml file isn't specified dev_appserver.py attempt to auto-detect it and that doesn't work in all cases). I'd avoid complications and just specify the app.yaml file(s).
Also note that the subprocess.run() args should be a list. Something like this:
subprocess.run(['python', '<sdk_path_to>/dev_appserver.py', '<app_path_to>/app.yaml'])
See also appcfg.py not working in command line - the post is about a different executable, but the answers are equally applicable to dev_appserver.py.
Quoting the App Engine documentation:
To start the local development server:
Run the dev_appserver.py command as follows from the directory that
contains your app's app.yaml configuration file:
Specify the directory
path to your app, for example:
dev_appserver.py [PATH_TO_YOUR_APP].
Alternatively, you can specify the
configuration file of a specific service, for example:
dev_appserver.py app.yaml.
To change the port, you include the --port
option:
dev_appserver.py --port=9999 [PATH_TO_YOUR_APP]
I'm trying to use Spacy on the new AppEngine Standard Python 3.7 runtime.
When I try to deploy I get:
ERROR: (gcloud.app.deploy) Cannot upload file
[/my/project/path/venv/lib/python3.7/site-packages/spacy/lang/tr/lemmatizer.py],
which has size [41523943] (greater than maximum allowed size of
[33554432]). Please delete the file or add to the skip_files entry in
your application .yaml file and try again.
A few oddities:
The docs seem to indicate that I don't need to upload the virtual environment and it will be created from requirements.txt
Looking at the log file, it seems to ignore .pyc files, but not the venv directory
The error message says to add to the skip_files in your application .yaml file and try again., but the docs say the python3.7 runtime doesn't use skip files and to use a .gcloudignore file instead, but adding venv/ or venv/* doesn't work (it appears to be ignored)
To fix this, I needed up update gcloud and reauthenticate:
gcloud components update
gcloud auth login
I want to update the app in google app store.
But I can't download the code...
Is there any way to update the app without downloading the code?
I tried to download with python, google app engine SDK...
But
appcfg.py download_app -A
This command does not work giving this error
NameError: global name 'execfile' is not defined...
Can you help me with this?
The error you have shown may occur due to incorrect PYTHONPATH environment variable.
If you are using the Windows version of the GAE SDK, then do the following:
1) Go to Edit > Preferences
2) Correct your Python Path.
To know the Python Path in windows do the following in the Python IDLE or Python CMD:
import os
import sys
print os.path.dirname(sys.executable)
For downloading your source code try this:
download_app -A app_name -V version C:\path_to_project
You may or may not need to escape the backslash.
Replace app_name, version and C:\path_to_project with appropriate values
To know the version go to the app engine admin website appengine.appspot.com
This seems like it should be very easy but I don't see a link to it anywhere.
How do I download the source code of a google app engine project?
Windows
appengine-java-sdk\bin\appcfg.cmd -A <your_app_id> -V <your_app_version> download_app <output-dir>
Linux
./appengine-java-sdk/bin/appcfg.sh -A <your_app_id> -V <your_app_version> download_app <output-dir>
For completeness, using the Python implementation:
appcfg.py download_app -A $appID -V $appVersionNumber $downloadDirectory --oauth2
--oauth2 is of course optional, you can omit it and provide your email + app-specific password (or your password, and then go implement two-factor authentication right after), but it's easier, and frankly there's no reason not to.
Documentation.
App Engine actually recently added the ability for the developer who uploaded a given app version to download its source code.
As of October 2019 you can simply go to --> App Engine --> Services and in the tool dropdown select 'source' and the source code is there
Posting this since none of the listed methods above didn't take me to the code (by June 2021)
You could try accessing it through;
Google Cloud Platform > Debugger > choosing the version of the
Application from combo at top.
This will list the files of that version on the left pane. There is no way to download it automatically but you can copy-paste the code.
Hope you will find this helpful.
IMHO, the best option today (Aug 2018) is:
Under the main menu, under Products, go to Tools -> Cloud Build -> Build history.
There, click the ID of the build you want (for me - the last one).
Then, in the opened window (Build details), click the "source" link, the download of your compressed code begins.
As simple as that.
HTH.
Working with App engine standard using Go, the debugger isn't available yet.
How I managed to download the source code for an existing service was to use the gcloud tool.
First: Get the version id of your service using the app engine console or running: gcloud app versions list
Second: use the version and service name and run: gcloud app versions describe <versionID> --service=<service name>
the describe parameter will give you the storage locations for your source files that looks like this:
cmd/main.go:
sha1Sum: e3fe5848c2640eca7ac3591490e1debc2d3a9b09
sourceUrl: https://storage.googleapis.com/<project>/<file id>
Third: you can then use the storage console, using the file id, to download the files you are interested in.
this process based on java sdk
Its works for me...
Download Google cloud SDK
gcloud init
enter image description here
Follow through process of logging in using your credentials
Enter following command from SDK
C:\Program Files (x86)\Google\appengine-java-sdk-1.9.49\bin
enter image description here
Enter Following command to download source code
appcfg.sh -A [YOUR_APP_ID] -V [YOUR_APP_VERSION] download_app [OUTPUT_DIR]
Eg: appcfg.sh -A my-project-name-1234 -V 2 download_app C:\Users\india\Desktop\my project
Note: this progress based on java-appengine sdk so we use appcfg.sh instead of appcfg.py
check if your app is uploaded with same email id that is in your app engine. if you are not sure then in app engine > control > Clear deployment credentials and then click on any project, deploy to sign in again then use this
appcfg.py download_app -A {app id from google app engine} -V {1} "{c:\path}" --oauth2_credential_file=C:\Users\{your account name}/.appcfg_oauth2_tokens
change all {} to your needs
Things have changed since this question was asked so I'm adding an updated answer. Note that this only applies to GAE Standard Environment
Google has deprecated appcfg.py and so the previous responses appcfg.py download_app no longer works.
gcloud which is the SDK in use (it replaced appcfg) does not have the functionality to download your source code.
When you deploy your app via gcloud app deploy, it copies your source code to a bucket. The default bucket is staging.<project_name>.appspot.com. Your files will stay in this bucket for a maximum of 15 days before they are deleted. You can modify the rule so that the files are retained for longer or less time.
The file names in the bucket are encoded so you can't figure out what each file is unless you open it (i.e. download it). Google has a mapping of the encoded names to the original file names. To get this mapping, you run the gcloud app versions describe command and it will list the file names and their encoded names. To download the files, you have to manually click each url one by one. So essentially, you have to download each file manually and then use the mapping to rename them (or open the file, check the content and then rename them). Also note that downloading the files manually will not maintain the folder structure in which they were uploaded.
If you do not wish to go through all of the above hassles (imagine having to manually open each url for each file if you have a small to mid-sized project which has hundreds of files), our App - https://nocommandline.com - now supports downloading source code from the default bucket - staging.<project_name>.appspot.com (so far as your files are still there which means any deployment i.e update not older than 15 days from your current date unless you previously increased the deletion age on your staging bucket's lifecycle page).
In simple terms, you enter your project name, the version number and our App will take care of retrieving the original file name to encoded name mapping, automatically downloading the files and renaming them to the original names, while maintaining the folder structure. For more information, refer to https://nocommandline.com/help/#faq_download_source_code_from_gae.
Log in to the console.developers.google.com
Select the project you want to download the code from (Google App Engine Standard Envoronment).
Go to the App Engine Dashboard. Under Summary is Debug and Source. Click on Source.
Select each file one at a time and copy it (highlight the code, copy and paste into your local editor.)
Select the next file....
You need to use svn to checkout the files.
If you are on Windows, you can use tortoise svn for your GUI end.
Here are tutorials on how to do it, here is the related question.