I have a number of Python 3.7 apps on Google App Engine standard, all building and deploying fine. I'm trying to upgrade some of them to the new Python 3.8 runtime, but when I try to deploy, they fail in Cloud Build.
It looks like they're hitting this open pip bug (more background). Odd that only the Python 3.8 runtime triggers this bug, though, and 3.7 builds fine.
Full log below. (Note that it's happening in Cloud Build, not my local machine, so I can't upgrade pip or otherwise change any of the commands or environment.) Anyone know how I can fix or work around this?
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.beta.app.deploy) Error Response: [9] Cloud build 83e346a0-7e88-43dd-b89c-a4820526e4a1 status: FAILURE
Error ID: f8df99ad
Error type: INTERNAL
Error message: ... (setup.py): started
Building wheel for webapp2 (setup.py): finished with status 'done'
Created wheel for webapp2: filename=webapp2-3.0.0b1-py3-none-any.whl size=68362 sha256=9dd9f3ab6a55404492a88eb9a6bacb00faa37efafbc41f21a24d21cfba0eaea3
Stored in directory: /layers/google.python.pip/pipcache/wheels/55/e9/4d/76b030f418cac0bef4a3dcc15ca95c9671f1e826731ce2bc0f
Building wheel for tlslite-ng (setup.py): started
Building wheel for tlslite-ng (setup.py): finished with status 'done'
Created wheel for tlslite-ng: filename=tlslite_ng-0.7.5-py3-none-any.whl size=199869 sha256=b9ead00f0832041fba1e9d3883e57847995c2d6f83ecb7ea87d09cf82c730e8b
Stored in directory: /layers/google.python.pip/pipcache/wheels/a6/e1/a6/09610854c3405202d0b71d8f869811781e40cd26ffb85eacf8
Successfully built gdata humanize mf2py mf2util python-tumblpy ujson webapp2 tlslite-ng
Installing collected packages: six, ecdsa, tlslite-ng, lxml, gdata, certifi, urllib3, chardet, idna, requests, setuptools, protobuf, googleapis-common-protos, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, pytz, grpcio, google-api-core, google-cloud-core, google-cloud-logging, gunicorn, pbr, extras, linecache2, traceback2, python-mimeparse, argparse, unittest2, testtools, fixtures, mox3, soupsieve, beautifulsoup4, gdata-python3, redis, google-cloud-datastore, google-cloud-ndb, humanize, MarkupSafe, jinja2, webencodings, html5lib, mf2py, mf2util, oauthlib, prawcore, websocket-client, update-checker, praw, requests-oauthlib, python-tumblpy, tweepy, ujson, webob, webapp2, oauth-dropins
Running setup.py develop for oauth-dropins
ERROR: Command errored out with exit status 1:
command: /opt/python3.8/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/workspace/setup.py'"'"'; __file__='"'"'/workspace/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --home /tmp/pip-target-zp53suvg
cwd: /workspace/
Complete output (6 lines):
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --home not recognized
----------------------------------------
ERROR: Command errored out with exit status 1: /opt/python3.8/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/workspace/setup.py'"'"'; __file__='"'"'/workspace/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --home /tmp/pip-target-zp53suvg Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the '/opt/python3.8/bin/python3 -m pip install --upgrade pip' command.
Full build logs: https://console.cloud.google.com/cloud-build/builds/83e346a0-7e88-43dd-b89c-a4820526e4a1?project=216076569502
Here's my requirements.txt file. I suspect the -e . might be the problem...but it works with Python 3.7, so if so, that's disappointing.
git+https://github.com/dvska/gdata-python3.git#egg=gdata
google-cloud-logging~=1.14
gunicorn~=20.0
mox3~=0.28
# this includes everything in setup.py's install_requires.
# https://caremad.io/posts/2013/07/setup-vs-requirement/#developing-reusable-things-or-how-not-to-repeat-yourself
-e .
I checked pypi page of oauth-dropins (at which it is failing) and they're mentioning there exactly this issue being caused by -e
I receive the following error:
google.appengine.api.datastore_errors.BadRequestError: \
Illegal string "dev~None" in dataset id.
When I try to use my production database in my local development environment:
$ appcfg.py download_data \
--application=s~app-id \
--url=http://app-id.appspot.com/_ah/remote_api \
--filename=database.ndb
$ appcfg.py upload_data \
--application=dev~app-id \
--url=http://localhost:53828/_ah/remote_api \
--filename=database.ndb
Any tips? My best guess is that something is broken by the transition from appcfg.py to gcloud. I use the latter most of the time, but I can't find a way to accomplish this task using gcloud. Thanks!
When running local appengine you have to specify -A parameter to set any application ID, otherwise you will be getting Illegal string "dev~None" in dataset id error. Took me a while to figure this out. Old environment variable APPLICATION_ID doesn't work (and application id from yaml file is neither is taken).
I'm now able to run a Dart app using
gcloud --verbosity debug preview app run app.yaml
and also to deploy and run on AppEngine
gcloud --verbosity debug preview app deploy app.yaml
but I haven't found a way to connect a debugger to the Dart app running on the development server.
I found http://dartbug.com/21067 but still couldn't find a way to make it work.
See also https://groups.google.com/a/dartlang.org/forum/#!topic/cloud/OK1nJtg7AjQ
Update 2015-02-27
The app can be run without Docker and then be debugged like any Dart command line application:
Source. https://groups.google.com/a/dartlang.org/d/msg/cloud/zrxgOHFz_lA/q5CdLLQPBAgJ
The API server is part of the App Engine SDK, and we are using it for
running tests in the appengine package. If you look at
https://github.com/dart-lang/appengine/blob/master/tool/run_tests.sh
you will see that it expects the environment variable
APPENGINE_API_SERVER.
The API server is in /platform/google_appengine/api_server.py
and takes a number of arguments. I just tested running it like this:
$ $CLOUD_SDK/platform/google_appengine/api_server.py \ -A
dev~test-application \ --api_port 4444 \ --high_replication \
--datastore_path /tmp/datastore
To run an app engine application outside the normal development server
requires that a number of environment variables are set. This worked
for my application:
$ GAE_LONG_APP_ID=test-application \ GAE_MODULE_NAME=default \
GAE_MODULE_VERSION=version \ GAE_PARTITION=dev \ API_PORT=4444 \
API_HOST=127.0.0.1 \ dart bin/server.dart
In the Dart Editor you cannot set environment variables for each
launch configuration, so they have to be set globally before starting
the Dart Editor. In WebStorm it is possible to have run configuration
specific environment variables.
This simple setup will of cause not support everything the normal
development server support. Some of the issues are:
Only one application at the time as it is always listening on port
8080 (can easily be made configurable) * The users API (mocking this
shouldn't be that difficult) * The modules API * No health-checks
(should not be a problem) * All HTTP headers are direct from the
client (no x-appengine- headers) * The admin web interface is not
available * Probably other stuff as well
This is all experimental, but it is one solution for a simpler
developer setup, which of cause does not match the deployment
environment as closely as the development server.
Running the API Server using Docker is also possible as the image
google/cloud-sdk with the Cloud SDK is on hub.docker.com.
Use the following Dockerfile
FROM google/cloud-sdk EXPOSE 4444 ENTRYPOINT
["/google-cloud-sdk/platform/google_appengine/api_server.py", \
"-A", "dev~test-application", \ "--api_port", "4444", \
"--high_replication", \ "--datastore_path", "/tmp/datastore"]
Build and run
$ docker build -t api_server . $ docker run -d -p 4444:4444 api_server
Change API_HOST above to 192.166.59.103 (of wherever your Docker
containers are) and run.
Regards, Søren Gjesse
Update 2014-11-27
Debugging from DartEditors debugger started working with the bleeding Dart build 1.8.0.edge_042017.
I assume that the next dev build (probably 1.9.0-dev1.0) will include the related fixes as well?
Detailed steps how this works can be found here: https://groups.google.com/a/dartlang.org/d/msg/cloud/OK1nJtg7AjQ/u-GzUDI-0VIJ
Build a custom Docker image with the latest Dart dev build 1.8.0-dev.4.6.
The Dart team is actually preparing an easy way to do this yourself (see https://github.com/dart-lang/dart_docker)
Installe the latest bleeding_edge on the host system (using this script https://gist.github.com/zoechi/d240f56a32ed5649797f or manual download from http://gsdview.appspot.com/dart-archive/channels/be/raw/latest/editor/darteditor-linux-x64.zip)
Add this to the app.yaml file
env_variables:
DBG_ENABLE: 'true'
# disable health-checking because this is so annoying during debugging
vm_health_check:
enable_health_check: False
See How to disable health checking for `gcloud preview app run` for more details about customizing health checking.
Launch the server code of your app with glcoud --verbosity debug app run app.yaml or glcoud --verbosity debug app run app.yaml index.yaml
Wait until the Docker container is ready (check with docker ps if the Command column shows a value starting with /dart_runtime/dart_
Open DartEditor
Open Menu Run > Remote Connection...
Connect to: Command-line VM
Host: localhost if you dont use boot2dockeror the IP address returned by the commandboot2docker ip`
Port: 5005
Select Folder... select the directory which contains the source code of your project.
Click OK
Set breakpoints and continue as usual.
Old
A first step is using the Observatory which includes a browser based debugger UI.
To make this work add the following lines to the app.yaml file
network:
forwarded_ports: ["8181"]
This might be useful as well to make the server.dart wait until we had the chance to set breakpoints using the observatory.
env_variables:
DART_VM_OPTIONS: '--pause-isolates-on-start'
boot2docker gives us the Docker ip (192.168.59.103) and after starting with gcloud preview app run app.yaml we can connect to http://192.168.59.103:8181 which should open the Observatory GUI.
I would like to have Drush on Hostgator shared hosting. I just spent 1 hour trying various outdated tutorials (Drush now requires composer). Does somebody have proved, tested and working solution how to install Drush there? I'm using PHP 5.4.
My last achieved step is drush st error:
Unable to load autoload.php. Drush now requires Composer in order to install its dependencies and autoload classes. Please see README.md
Content-type: text/html
When I run php composer.phar diagnose I see:
Content-type: text/html
Warning: Composer should be invoked via the CLI version of PHP, not the cgi-fcgi SAPI
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity: OK
Checking disk free space: OK
Checking composer version: OK
Incase it helps others, I had been using a variety of google inspired resources to install Drush, which just complicated the situation for me. I would highly recommend following the official documentation which is the main source of information. I even read on there that they only maintain the documentation here, not even on drupal.org.
I was missing this step based on other instructions,
Now add Drush to your system path by placing export
PATH="$HOME/.composer/vendor/bin:$PATH"
into your ~/.bash_profile (Mac OS users) or into your ~/.bashrc (Linux users).
These instructions helped me resolve the error:
curl -sS https://getcomposer.org/installer | php mv composer.phar
/usr/local/bin/composer ln -s /usr/local/bin/composer
/usr/bin/composer
git clone https://github.com/drush-ops/drush.git /usr/local/src/drush
cd /usr/local/src/drush git checkout 7.0.0-alpha5 #or whatever
version you want. ln -s /usr/local/src/drush/drush /usr/bin/drush
composer install drush --version
I think you are trying to use Drush version 7.x.
Try using Drush 6.x, I don't think it requires composer. Drush releases. I have had drush 6.4 installed on shared hosting environment successfully without any problems.
I'm trying to generate an Android client lib from my Python endpoints API by running endpointscfg.py
endpointscfg.py get_client_lib java -o . -f rest services.MyApi
The problem I have is that a none-.*-sources.jar is generated, and the client classes are created under a Java package named "none.MyApi".
How can I override the generated package name to fix this?
This is mentioned in the docs. Try specifying the hostname either as a flag to endpointscfg.py when invoked from command line:
endpointscfg.py get_client_lib java \
> --hostname=some-client-id.appspot.com \
> services.MyApi
or in your API definition:
#endpoints.api(name='apiname', version='version',
hostname='some-client-id.appspot.com',
...)