When trying to deploy my application:
appcfg.py update --oauth2 .
I'm getting:
PM Rolling back the update.
Error 400: --- begin server output ---
Max number of files and blobs is 10000.
--- end server output ---
How do I solve this?
You "solve" it by having less files deployed when you deploy your project. You have a few choices.
You can use Skipfiles in your app.yaml to exclude some files from being deployed:
skip_files:
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
You might want to, for example, exclude .readme files, compiled python files etc.
Or you can use ZipImport and compress your python packages and they will be unzipped on the fly during import.
Or if you are also uploading lots of static assets, put them in a bucket instead as the other answer suggests.
Looks like you are hitting this quota: https://cloud.google.com/appengine/docs/quotas#Deployments
Could you shift many of your files to Cloud Storage?
Related
Today we start recieving this message when trying to deploy.
...
Scanned 8750 files.
Scanned 9000 files.
Scanned 9250 files.
Scanned 9500 files.
Scanned 9750 files.
Initiating update.
Cloning 711 static files.
Cloning 9233 application files.
Cloned 2000 files.
Cloned 4000 files.
Cloned 6000 files.
Cloned 8000 files.
Deploying:
Uploading 0 files.
Initializing precompilation...
Deploying new version.
Rolling back the update.
com.google.appengine.tools.admin.HttpIoException: Error posting to URL: https://appengine.google.com/api/appversion/deploy?app_id=facturanube&version=1&
500 Internal Server Error
<h3>Server Error</h3><p>A server error has occurred.</p>
Debugging information may be found in C:\Users\usuario\AppData\Local\Temp\appengine-deploy6314649189927600314.log
Debugging information file contents:
Unable to update:
com.google.appengine.tools.admin.HttpIoException: Error posting to URL: https://appengine.google.com/api/appversion/deploy?app_id=si-nube&version=1&
500 Internal Server Error
<h3>Server Error</h3><p>A server error has occurred.</p>
at com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:303)
at com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:254)
at com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:233)
at com.google.appengine.tools.admin.LoggingClientDeploySender.send(LoggingClientDeploySender.java:47)
at com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:1100)
at com.google.appengine.tools.admin.AppVersionUpload.deploy(AppVersionUpload.java:949)
at com.google.appengine.tools.admin.AppVersionUpload.commit(AppVersionUpload.java:850)
at com.google.appengine.tools.admin.AppVersionUpload.uploadFilesTransaction(AppVersionUpload.java:176)
at com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:151)
at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:474)
at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:55)
at com.google.appengine.eclipse.core.proxy.AppEngineBridgeImpl.deploy(AppEngineBridgeImpl.java:400)
at com.google.appengine.eclipse.core.deploy.DeployProjectJob.runInWorkspace(DeployProjectJob.java:148)
at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:38)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Important considerations:
The appengine-web.xml file has no errors.
We tried deploying in different versions with no success.
The application can be deployed in another appengine application, but not in our production application.
We have no cuota-related problems. Our payments have no
problems.
Our current versions:
GWT SDK 2.4.0
App Engine 1.9.54
Java 7
Update
I was able to deploy today (2018/03/18) with no changes to code or configuration files. I guess Google did something to solve the issue.
500 A server error has occurred on deployment is usually related to malformed configs, including app.yaml and cron.yaml (and Java equivalents). Or if unsupported stanzas are included. Please check your GAE app configs and attempt to strip them down to the essentials just to narrow down which stanzas/flags are messing up the deployment.
Since I upgraded my Gradle build to deploy my App Engine application with SDK 1.9.46, I get deployment conflict messages, like the one below.
Beginning interaction for module admin...
0% Created staging directory at: '/var/folders/m8/6z4h4k2x11s3whxrqmd79lym0000gn/T/appcfg2292422843793738444.tmp'
5% Scanning for jsp files.
8% Generated git repository information file.
20% Scanning files on local disk.
25% Scanned 250 files.
28% Scanned 500 files.
31% Scanned 750 files.
33% Scanned 1000 files.
34% Initiating update.
Nov 13, 2016 11:11:41 AM com.google.appengine.tools.admin.AbstractServerConnection send1
WARNING: Error posting to URL: https://appengine.google.com/api/appversion/create?module=admin&app_id=the-outdoor-game&version=dev&
409 Conflict
There is an operation pending for this application: AppLock held for "Updating engine_version_id='dev' within engine_id='default'." acquired by Peter.Fortuin at 2016-11-13 09:15:05.482616 GMT. Please wait and try again or use 'appcfg rollback' to attempt to cleanly abort the current operation.
This is try #0
A rollback if not solving the problem. The only thing that seems to work is downgrading the App Engine SDK to 1.9.44. But that gives me other problems that I don't want to solve.
Anybody has any idea how to solve this? Can it be that this is a bug in the 1.9.46 SDK? Anybody else seeing the same thing?
I have resolved this problem by changing 'service' to 'module' in appengine-web.xml of my module.
See https://github.com/GoogleCloudPlatform/gradle-appengine-plugin/issues/268 for more information.
I am following the instructions on https://cloud.google.com/appengine/docs/go/googlecloudstorageclient/download to begin migrating some code from the, now deprecated, Files API to the new Cloud Storage API without success.
The steps I'm following are ...
I'm running appengine v1.9.23 which is later than the required appengine v1.8.1.
My $GOPATH is set, so I skip step #1.
I proceed to step #2:
goapp get -u golang.org/x/oauth2
goapp get -u google.golang.org/cloud/storage
I am not developing on a managed VM, so I skip step #3.
Now when I run the application, I get:
go-app-builder: Failed parsing input: parser: bad import "syscall" in goapp/src/golang.org/x/net/internal/nettest/error_posix.go
What am I doing wrong?
Steps to reproduce
Download an install the Google Appengine runtime, version 1.9.23 from https://console.cloud.google.com/storage/browser/appengine-sdks/featured/ . Follow the installation instructions documented on https://cloud.google.com/appengine/downloads?hl=en
Create an appengine project directory:
% mkdir $HOME/myapp
Create a new app.yaml file as ~/myapp/app.yaml. Read the directions on the Google website for details: https://cloud.google.com/appengine/docs/go/config/appconfig
I use a version that does not have the static resources:
application: myapp
version: alpha-001
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
Create a location for the Go source files.
% mkdir $HOME/myapp/go
Set your GOPATH to the location of your sources
% export GOPATH=$HOME/myapp/go
Get the Go appengine example project: https://github.com/golang/example
% goapp get github.com/golang/example/appengine-hello
This command will download the example app to the first path entry in the GOPATH
Install the Google Cloud Storage client libraries as directed in https://cloud.google.com/appengine/docs/go/googlecloudstorageclient/download . Reference the steps at the top of this question for more details. Following the directions should result in you running 2 commands:
% go get -u golang.org/x/oauth2
% go get -u google.golang.org/cloud/storage
Attempt to run your go application
% goapp serve
You will see the following compilation error (no stack trace):
2015/12/23 10:37:07 go-app-builder: Failed parsing input: parser: bad import "syscall" in go/src/golang.org/x/net/ipv6/control_unix.go
This error is caused by either of two scenarios:
1) Implicitly importing syscall by importing another package that uses it, as referenced in this related question.
2) Having your package source files in your GOPATH located in a directory at or below the same level as your project's app.yaml (eg. app.yaml in ~/go, and packages sources in ~/go/gopath/src). If a package like x/net/internal/nettest exists in your GOPATH the syscall import will be parsed by goapp at compile time and throw the compilation error.
Avoiding these two scenarios should be sufficient to prevent any bad import "syscall" errors or related compilation errors.
Reproduced the initial steps above and got a similar error, even if not explicitly mentioning syscall. However, running “goapp serve” in the appengine-hello directory results in no error at all.
Adam’s explanation at point 2 applies here correctly: one needs to place the app.yaml file at the right level in the directory structure.
sirupsen/logrus references syscall.
They have an appengine tag specified, not to include syscall so it's usable in AppEngine, something like go build -tags appengine as per issue 310.
However I haven't yet succeeded including it in an AppEngine project so that this build param could be forwarded and specified somewhere so that it goes through. I'll come back to update if I manage.
my python27 app deployment is failing for some reasons:
2012-03-09 16:46:25 Running command: "['C:\\Python27\\pythonw.exe', '-u', 'C:\\Program Files (x86)\\Google\\google_appengine\\appcfg.py', '--no_cookies', u'--email=malartre#scolab.com', '--passin', 'update', 'C:\\Users\\XXXXXXXXXXXXXX\\GoogleAppEngine\\XXXXXXXXXXXXXX']"
Application: XXXXXXXXXXXXXX; version: 1
Host: appengine.google.com
Starting update of app: XXXXXXXXXXXXXXX, version: 1
Getting current resource limits.
Password for malartre#scolab.com: Scanning files on local disk.
Scanned 500 files.
Scanned 1000 files.
Scanned 1500 files.
Scanned 2000 files.
Scanned 2500 files.
Cloning 2973 static files.
Cloned 100 files.
Cloned 200 files.
Cloned 300 files.
Cloned 400 files.
Cloned 500 files.
Cloned 600 files.
Cloned 700 files.
Cloned 800 files.
Cloned 900 files.
Cloned 1000 files.
Cloned 1100 files.
Cloned 1200 files.
Cloned 1300 files.
Cloned 1400 files.
Cloned 1500 files.
Cloned 1600 files.
Cloned 1700 files.
Cloned 1800 files.
Cloned 1900 files.
Cloned 2000 files.
Cloned 2100 files.
Cloned 2200 files.
Cloned 2300 files.
Cloned 2400 files.
Cloned 2500 files.
Cloned 2600 files.
Cloned 2700 files.
Cloned 2800 files.
Cloned 2900 files.
Cloning 3 application files.
Uploading 4 files and blobs.
Uploaded 4 files and blobs
Compilation starting.
Compilation completed.
Starting deployment.
Rolling back the update.
Error 400: --- begin server output ---
Client Error (400)
The request is invalid for an unspecified reason.
--- end server output ---
2012-03-09 16:47:14 (Process exited with code 1)
You can close this window now.
How can I debug that?
Thanks,
Carl
Just recording my experience in this related question.
I got a 'Client Error 400/unspecified reason' because my app.yaml error handlers were pointing to files in a static directory.
I moved the error files to a separate directory and updated app.yaml paths.
Update worked properly.
i know this is an old question, but i thought i would provide the solution that worked for me. Select Control from the GAE Launcher, then Clear Deployment Credential. Deploy again. The deployment script will open a web page where GAE is requesting access to use your Google account to authenticate. the answer from Jen Tong helped me get to my solution.
This is a fall-back error message that is triggered when your deployment sends very unexpected data to App Engine during deployment.
In other words, there's no quick fix here, but there are a couple things you can check:
Check your apps metadata in app.yaml for Python and appengine-web.xml for Java. Double check each field. If the application name is very wrong, it may trigger this error.
Make sure that your deployment credentials are valid. If you're authenticating with OAuth 2.0, try using password based authentication.
Deleting some files worked for me. Looks like there may be a 1000 file hard limit for apps on GAE, and this obscure error code is at least in part used to document being over the limit.
http://www.adampresley.com/2011/06/gae-and-client-error-400.html
I has this because I had an rogue ) in my app.yaml. I'd check the last thing(s) you did for errors because although your app may upload, an error may cause the application not to deploy.
One of the causes of this error is having more than 1000 files in a folder.
For me I had about 1k images but then 9 folders and I think that put it over the top. I haven't checked if the limitation also exists for folder counts but I imagine it does.
This seems like it should be very easy but I don't see a link to it anywhere.
How do I download the source code of a google app engine project?
Windows
appengine-java-sdk\bin\appcfg.cmd -A <your_app_id> -V <your_app_version> download_app <output-dir>
Linux
./appengine-java-sdk/bin/appcfg.sh -A <your_app_id> -V <your_app_version> download_app <output-dir>
For completeness, using the Python implementation:
appcfg.py download_app -A $appID -V $appVersionNumber $downloadDirectory --oauth2
--oauth2 is of course optional, you can omit it and provide your email + app-specific password (or your password, and then go implement two-factor authentication right after), but it's easier, and frankly there's no reason not to.
Documentation.
App Engine actually recently added the ability for the developer who uploaded a given app version to download its source code.
As of October 2019 you can simply go to --> App Engine --> Services and in the tool dropdown select 'source' and the source code is there
Posting this since none of the listed methods above didn't take me to the code (by June 2021)
You could try accessing it through;
Google Cloud Platform > Debugger > choosing the version of the
Application from combo at top.
This will list the files of that version on the left pane. There is no way to download it automatically but you can copy-paste the code.
Hope you will find this helpful.
IMHO, the best option today (Aug 2018) is:
Under the main menu, under Products, go to Tools -> Cloud Build -> Build history.
There, click the ID of the build you want (for me - the last one).
Then, in the opened window (Build details), click the "source" link, the download of your compressed code begins.
As simple as that.
HTH.
Working with App engine standard using Go, the debugger isn't available yet.
How I managed to download the source code for an existing service was to use the gcloud tool.
First: Get the version id of your service using the app engine console or running: gcloud app versions list
Second: use the version and service name and run: gcloud app versions describe <versionID> --service=<service name>
the describe parameter will give you the storage locations for your source files that looks like this:
cmd/main.go:
sha1Sum: e3fe5848c2640eca7ac3591490e1debc2d3a9b09
sourceUrl: https://storage.googleapis.com/<project>/<file id>
Third: you can then use the storage console, using the file id, to download the files you are interested in.
this process based on java sdk
Its works for me...
Download Google cloud SDK
gcloud init
enter image description here
Follow through process of logging in using your credentials
Enter following command from SDK
C:\Program Files (x86)\Google\appengine-java-sdk-1.9.49\bin
enter image description here
Enter Following command to download source code
appcfg.sh -A [YOUR_APP_ID] -V [YOUR_APP_VERSION] download_app [OUTPUT_DIR]
Eg: appcfg.sh -A my-project-name-1234 -V 2 download_app C:\Users\india\Desktop\my project
Note: this progress based on java-appengine sdk so we use appcfg.sh instead of appcfg.py
check if your app is uploaded with same email id that is in your app engine. if you are not sure then in app engine > control > Clear deployment credentials and then click on any project, deploy to sign in again then use this
appcfg.py download_app -A {app id from google app engine} -V {1} "{c:\path}" --oauth2_credential_file=C:\Users\{your account name}/.appcfg_oauth2_tokens
change all {} to your needs
Things have changed since this question was asked so I'm adding an updated answer. Note that this only applies to GAE Standard Environment
Google has deprecated appcfg.py and so the previous responses appcfg.py download_app no longer works.
gcloud which is the SDK in use (it replaced appcfg) does not have the functionality to download your source code.
When you deploy your app via gcloud app deploy, it copies your source code to a bucket. The default bucket is staging.<project_name>.appspot.com. Your files will stay in this bucket for a maximum of 15 days before they are deleted. You can modify the rule so that the files are retained for longer or less time.
The file names in the bucket are encoded so you can't figure out what each file is unless you open it (i.e. download it). Google has a mapping of the encoded names to the original file names. To get this mapping, you run the gcloud app versions describe command and it will list the file names and their encoded names. To download the files, you have to manually click each url one by one. So essentially, you have to download each file manually and then use the mapping to rename them (or open the file, check the content and then rename them). Also note that downloading the files manually will not maintain the folder structure in which they were uploaded.
If you do not wish to go through all of the above hassles (imagine having to manually open each url for each file if you have a small to mid-sized project which has hundreds of files), our App - https://nocommandline.com - now supports downloading source code from the default bucket - staging.<project_name>.appspot.com (so far as your files are still there which means any deployment i.e update not older than 15 days from your current date unless you previously increased the deletion age on your staging bucket's lifecycle page).
In simple terms, you enter your project name, the version number and our App will take care of retrieving the original file name to encoded name mapping, automatically downloading the files and renaming them to the original names, while maintaining the folder structure. For more information, refer to https://nocommandline.com/help/#faq_download_source_code_from_gae.
Log in to the console.developers.google.com
Select the project you want to download the code from (Google App Engine Standard Envoronment).
Go to the App Engine Dashboard. Under Summary is Debug and Source. Click on Source.
Select each file one at a time and copy it (highlight the code, copy and paste into your local editor.)
Select the next file....
You need to use svn to checkout the files.
If you are on Windows, you can use tortoise svn for your GUI end.
Here are tutorials on how to do it, here is the related question.