I'm trying to use Spacy on the new AppEngine Standard Python 3.7 runtime.
When I try to deploy I get:
ERROR: (gcloud.app.deploy) Cannot upload file
[/my/project/path/venv/lib/python3.7/site-packages/spacy/lang/tr/lemmatizer.py],
which has size [41523943] (greater than maximum allowed size of
[33554432]). Please delete the file or add to the skip_files entry in
your application .yaml file and try again.
A few oddities:
The docs seem to indicate that I don't need to upload the virtual environment and it will be created from requirements.txt
Looking at the log file, it seems to ignore .pyc files, but not the venv directory
The error message says to add to the skip_files in your application .yaml file and try again., but the docs say the python3.7 runtime doesn't use skip files and to use a .gcloudignore file instead, but adding venv/ or venv/* doesn't work (it appears to be ignored)
To fix this, I needed up update gcloud and reauthenticate:
gcloud components update
gcloud auth login
When trying to deploy my application:
appcfg.py update --oauth2 .
I'm getting:
PM Rolling back the update.
Error 400: --- begin server output ---
Max number of files and blobs is 10000.
--- end server output ---
How do I solve this?
You "solve" it by having less files deployed when you deploy your project. You have a few choices.
You can use Skipfiles in your app.yaml to exclude some files from being deployed:
skip_files:
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
You might want to, for example, exclude .readme files, compiled python files etc.
Or you can use ZipImport and compress your python packages and they will be unzipped on the fly during import.
Or if you are also uploading lots of static assets, put them in a bucket instead as the other answer suggests.
Looks like you are hitting this quota: https://cloud.google.com/appengine/docs/quotas#Deployments
Could you shift many of your files to Cloud Storage?
I run endpoints.sh
get-client-lib com.my.app.FooService and successfully generate the files for Google Cloud Endpoints[1] (2 .discovery files, 1 .api file and 1 .zip file).
The script doesn't add anything to my war folder, so I assume server-side is already handled by magic configured in web.xml and #Api annotations (a'la Spring Framework).
But http://localhost:8080/_ah/api/explorer redirects me to a blank Google cloud console. Uploading my app yields the same result.
Am I missing something? I think the documentation is a bit lacking, it doesn't even explain what the generated files are for.
I'm using Google App Engine Java.
[1] https://developers.google.com/appengine/docs/java/endpoints/gen_clients
To make the endpoints work, need follow the steps stated in the Google GAE page
Write your API backend code first.
Annotate your API backend code, so classes and client libraries can be generated from it. (Alternatively, use the Google Plugin for Eclipse, which annotates automatically for you.)
Generate the client library using the endpoints.sh utility. (Alternatively, use the Google Plugin for Eclipse to generate the client library.)
Write your client app, using the client library when making calls to the API backend.
But the above steps miss out some important steps, which the Google Plugins (Google Plugin for Eclipse) will generated for you automatically.
Configure the servlet in your web.xml. (Replace the your-full-class-name with your own class name)
<servlet>
<servlet-name>com.google.api.server.spi.SystemServiceServlet</servlet-name>
<servlet-class>com.google.api.server.spi.SystemServiceServlet</servlet-class>
<init-param>
<param-name>services</param-name>
<param-value>your-full-class-name</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>com.google.api.server.spi.SystemServiceServlet</servlet-name>
<url-pattern>/_ah/spi/*</url-pattern>
</servlet-mapping>
you need ensure the file (.api) generated by the endpoints.sh is copied into the folder WEB-INF in your web app root folder.
For better understanding, you can invoke the endpoints.sh to see all the available options as the following
Available commands:
get-client-lib: Generates a client library
usage: get-client-lib ...
Options:
--classpath=CLASSPATH Additional class path entries
-cp CLASSPATH (default: ./war/WEB-INF/classes).
--language=LANGUAGE The target output programming language
-l LANGUAGE (java) (default: java).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
--war=WAR_PATH The path to a directory or .war with a WAR
-w WAR_PATH directory layout (default: ./war).
gen-api-config: Generates API configuration files from service classes
usage: gen-api-config ...
Options:
--classpath=CLASSPATH Additional class path entries
-cp CLASSPATH (default: ./war/WEB-INF/classes).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
--war=WAR_PATH The path to a directory or .war with a WAR
-w WAR_PATH directory layout (default: ./war).
gen-discovery-doc: Generates API Discovery document
usage: gen-discovery-doc
Options:
--format=FORMAT The requested API protocol type (rest|rpc)
-f FORMAT (default: rest).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
gen-client-lib: Generates a client library
usage: gen-client-lib
Options:
--language=LANGUAGE The target output programming language
-l LANGUAGE (java) (default: java).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
Then you are able to view the services in the explorer UI as the following.
when you access the http: //localhost:8080/_ah/api/explorer, it will be redirected to the developers.google.com first. But the contents is from the localhost.
You can always rely on your normal Google APE local development server. For example, IntelliJ Google APE plugin start the Google APE local server also can support the "_ah/api/explorer" access
As mentioned by Dan Holevoet in the comment, the script needs to be ran from war. As for my case, I just need to copy the files into that directory.
This seems like it should be very easy but I don't see a link to it anywhere.
How do I download the source code of a google app engine project?
Windows
appengine-java-sdk\bin\appcfg.cmd -A <your_app_id> -V <your_app_version> download_app <output-dir>
Linux
./appengine-java-sdk/bin/appcfg.sh -A <your_app_id> -V <your_app_version> download_app <output-dir>
For completeness, using the Python implementation:
appcfg.py download_app -A $appID -V $appVersionNumber $downloadDirectory --oauth2
--oauth2 is of course optional, you can omit it and provide your email + app-specific password (or your password, and then go implement two-factor authentication right after), but it's easier, and frankly there's no reason not to.
Documentation.
App Engine actually recently added the ability for the developer who uploaded a given app version to download its source code.
As of October 2019 you can simply go to --> App Engine --> Services and in the tool dropdown select 'source' and the source code is there
Posting this since none of the listed methods above didn't take me to the code (by June 2021)
You could try accessing it through;
Google Cloud Platform > Debugger > choosing the version of the
Application from combo at top.
This will list the files of that version on the left pane. There is no way to download it automatically but you can copy-paste the code.
Hope you will find this helpful.
IMHO, the best option today (Aug 2018) is:
Under the main menu, under Products, go to Tools -> Cloud Build -> Build history.
There, click the ID of the build you want (for me - the last one).
Then, in the opened window (Build details), click the "source" link, the download of your compressed code begins.
As simple as that.
HTH.
Working with App engine standard using Go, the debugger isn't available yet.
How I managed to download the source code for an existing service was to use the gcloud tool.
First: Get the version id of your service using the app engine console or running: gcloud app versions list
Second: use the version and service name and run: gcloud app versions describe <versionID> --service=<service name>
the describe parameter will give you the storage locations for your source files that looks like this:
cmd/main.go:
sha1Sum: e3fe5848c2640eca7ac3591490e1debc2d3a9b09
sourceUrl: https://storage.googleapis.com/<project>/<file id>
Third: you can then use the storage console, using the file id, to download the files you are interested in.
this process based on java sdk
Its works for me...
Download Google cloud SDK
gcloud init
enter image description here
Follow through process of logging in using your credentials
Enter following command from SDK
C:\Program Files (x86)\Google\appengine-java-sdk-1.9.49\bin
enter image description here
Enter Following command to download source code
appcfg.sh -A [YOUR_APP_ID] -V [YOUR_APP_VERSION] download_app [OUTPUT_DIR]
Eg: appcfg.sh -A my-project-name-1234 -V 2 download_app C:\Users\india\Desktop\my project
Note: this progress based on java-appengine sdk so we use appcfg.sh instead of appcfg.py
check if your app is uploaded with same email id that is in your app engine. if you are not sure then in app engine > control > Clear deployment credentials and then click on any project, deploy to sign in again then use this
appcfg.py download_app -A {app id from google app engine} -V {1} "{c:\path}" --oauth2_credential_file=C:\Users\{your account name}/.appcfg_oauth2_tokens
change all {} to your needs
Things have changed since this question was asked so I'm adding an updated answer. Note that this only applies to GAE Standard Environment
Google has deprecated appcfg.py and so the previous responses appcfg.py download_app no longer works.
gcloud which is the SDK in use (it replaced appcfg) does not have the functionality to download your source code.
When you deploy your app via gcloud app deploy, it copies your source code to a bucket. The default bucket is staging.<project_name>.appspot.com. Your files will stay in this bucket for a maximum of 15 days before they are deleted. You can modify the rule so that the files are retained for longer or less time.
The file names in the bucket are encoded so you can't figure out what each file is unless you open it (i.e. download it). Google has a mapping of the encoded names to the original file names. To get this mapping, you run the gcloud app versions describe command and it will list the file names and their encoded names. To download the files, you have to manually click each url one by one. So essentially, you have to download each file manually and then use the mapping to rename them (or open the file, check the content and then rename them). Also note that downloading the files manually will not maintain the folder structure in which they were uploaded.
If you do not wish to go through all of the above hassles (imagine having to manually open each url for each file if you have a small to mid-sized project which has hundreds of files), our App - https://nocommandline.com - now supports downloading source code from the default bucket - staging.<project_name>.appspot.com (so far as your files are still there which means any deployment i.e update not older than 15 days from your current date unless you previously increased the deletion age on your staging bucket's lifecycle page).
In simple terms, you enter your project name, the version number and our App will take care of retrieving the original file name to encoded name mapping, automatically downloading the files and renaming them to the original names, while maintaining the folder structure. For more information, refer to https://nocommandline.com/help/#faq_download_source_code_from_gae.
Log in to the console.developers.google.com
Select the project you want to download the code from (Google App Engine Standard Envoronment).
Go to the App Engine Dashboard. Under Summary is Debug and Source. Click on Source.
Select each file one at a time and copy it (highlight the code, copy and paste into your local editor.)
Select the next file....
You need to use svn to checkout the files.
If you are on Windows, you can use tortoise svn for your GUI end.
Here are tutorials on how to do it, here is the related question.
While working on my GAE project under my dev environment, whenever I upload data to my dev datastore, the logfiles are stored in my current directory, for instance:
C:\dev\ls
bulkloader-log-20090912.104643
bulkloader-log-20090912.104648
bulkloader-log-20090912.104731
bulkloader-log-20090912.105526
bulkloader-log-20090912.110428
bulkloader-progress-20090912.104648.sql3
bulkloader-progress-20090912.104731.sql3
bulkloader-progress-20090912.105526.sql3
bulkloader-progress-20090912.110428.sql3
project
project is my GAE app. The above is generated when I run the command appcfg.py upload_data. Is there a way to tell GAE where to store those log files, for instance in a log folder.
Use the --log_file=... option to appcfg.py, as documented here: with this command line option you can give the complete path to the log file, including folder and name. (You cannot give JUST the folder and let it figure out the name; for that, you need to write a tiny script that figures out the name then calls appcfg.py).