I have a Flask application deployed on an Amazon Elastic Beanstalk cluster. On my local machine, macOS, I've added an integration with the Google Cloud API, and I've updated my requirements.txt to include the line google-cloud==0.27.0. When I deploy to Elastic Beanstalk with the updated requirements file, my deployment fails during pip install with the error
Running setup.py install for grpcio
Complete output from command /opt/python/run/venv/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-ve1vz0tx/grpcio/setup
.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-aszzosux-rec
ord/install-record.txt --single-version-externally-managed --compile --install-headers /opt/python/run/venv/include/site/python3.4/grpcio:
Failed to import the site module
Traceback (most recent call last):
File "/opt/python/run/venv/lib64/python3.4/site.py", line 890, in <module>
main()
File "/opt/python/run/venv/lib64/python3.4/site.py", line 848, in main
virtualenv_search_paths(sys.prefix)
File "/opt/python/run/venv/lib64/python3.4/site.py", line 638, in virtualenv_search_paths
addsitedir(sitedir, known_paths)
File "/opt/python/run/venv/lib64/python3.4/site.py", line 204, in addsitedir
addpackage(sitedir, name, known_paths)
File "/opt/python/run/venv/lib64/python3.4/site.py", line 173, in addpackage
exec(line)
File "<string>", line 1, in <module>
KeyError: 'google'
I am able to install my requirements locally in a virtualenv running python 3, however, when I create a similar virtualenv on my ec2 and install the requirements, I get the same error I get during deployment. One thing I have read about is that the ec2 might not have google cloud sdk installed, however, I installed it on my ec2 (tested both inside and outside of a virtualenv) using the following commands as described here here:
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init
How can I diagnose this error and prevent it from happening going forward?
My current hypotheses are:
there is still an issue with the way google cloud sdk is install or operating on the ec2
there is some conflict between requirements in my requirements.txt file once I add the google-cloud requirement
I've identified and fixed the problem. I had google==1.9.2 as a package in my requirements.txt and it wasn't playing well with google-cloud==0.27.0. I'm not sure why this occurred though.
Note: when deploying to Elastic Beanstalk, I had to rebuild the environments for the change to take place. It seems like Elastic Beanstalk reuses the Python virtualenv across deploys, so if a server had ever run a version of my application with google==1.9.2 in requirements, that previously installed version of google would interfere with future deploys that excluded it
Related
I am trying to modularize our code, and running into difficulties installing private packages on app engine.
We have a repository on GCP in the same project, so I would think this is not too difficult.
My requirement.txt looks like:
--extra-index-url=https://us-central1-python.pkg.dev/myproject/my-python-repo/simple/
Flask==2.0.1
my-new-package
I can pip install the package locally, and it uses the keyrings.google-artifactregistry-auth package to authenticate.
The deployment fails with:
File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 218, in ask_input
return input(message)
EOFError: EOF when reading a line
This is clearly pip asking for a username, where i
Is google's own keyring package not available in its own environment? This Suggests so. Of course adding it to requirements.txt has no effect, as it is too late.
How can I install packages properly?
A possible solution is to:
--extra-index-url=_json_key_base64:KEY#us-central1-python.pkg.dev/myproject/my-python-repo/simple/
Where KEY is the result of base64 -w 0 < credentials.json
Of course this then requires some rewriting of the requirements.txt file in your deployment pipeline to ensure service account credentials are secured in repository variables.
I am trying to access my Google Cloud Bucket from development environment but when I write import statement I get error
from google.cloud import storage
Command I use to run server with bucket flag
dev_appserver.py app.yaml --default_gcs_bucket_name ABC-test-bucket
Error I get
File "C:\Users\ABC\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\setuptools-0.6c11\pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
DistributionNotFound: google-cloud-storage
I think I completed all the steps like creating bucket, downloading client library using pip i.e.
pip install GoogleAppEngineCloudStorageClient -t <your_app_directory/lib>
I am new to GAE projects(Using Webapp2 Python framework for server) so I will really appreciate any pointers and help
The library you're using is not the correct one. You need the one below, as per the documentation:
pip install --upgrade google-cloud-storage
Setup: Run in Google Cloud Shell Standard Env
I tried to access firestore from the flask app using from google.cloud import firestore
I have installed it using pip install --upgrade google-cloud-firestore -t lib. If I run the script manually, it works fine. But when using dev_appserver.py, it fails. Got the error below.
$ dev_appserver.py app.yaml
INFO 2017-10-06 07:34:35,301 devappserver2.py:105] Skipping SDK update check.
INFO 2017-10-06 07:34:35,391 api_server.py:300] Starting API server at: http://0.0.0.0:34796
WARNING 2017-10-06 07:34:35,391 dispatcher.py:312] Your python27 micro version is below 2.7.12, our current production version.
INFO 2017-10-06 07:34:35,440 dispatcher.py:251] Starting module "default" running at: http://0.0.0.0:8080
INFO 2017-10-06 07:34:35,441 admin_server.py:116] Starting admin server at: http://0.0.0.0:8000
ERROR 2017-10-06 07:34:42,266 wsgi.py:263]
Traceback (most recent call last):
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/user1/projects/probfe/main.py", line 10, in <module>
from google.cloud import firestore
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/sandbox.py", line 1132, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named google.gax
However, in my lib, I can see them.
$ls lib
builtins future-0.16.0.dist-info itsdangerous.pyc requests-2.18.4.dist-info
cachetools futures-3.1.1.dist-info jinja2 rsa
cachetools-2.0.1.dist-info google Jinja2-2.9.6.dist-info rsa-3.4.2.dist-info
certifi googleapis_common_protos-1.5.3.dist-info libfuturize setuptools
certifi-2017.7.27.1.dist-info googleapis_common_protos-1.5.3-py2.7-nspkg.pth libpasteurize setuptools-36.5.0.dist-info
chardet google_auth-1.1.1.dist-info _markupbase six-1.11.0.dist-info
chardet-3.0.4.dist-info google_auth-1.1.1-py2.7-nspkg.pth markupsafe six.py
click google_cloud_core-0.27.1.dist-info MarkupSafe-1.0.dist-info six.pyc
click-6.7.dist-info google_cloud_core-0.27.1-py3.6-nspkg.pth past socketserver
concurrent google_cloud_firestore-0.27.0.dist-info pkg_resources tests
copyreg google_cloud_firestore-0.27.0-py3.6-nspkg.pth ply _thread
dill google_gax-0.15.15.dist-info ply-3.8.dist-info tkinter
dill-0.2.7.1.dist-info google_gax-0.15.15-py2.7-nspkg.pth protobuf-3.4.0.dist-info urllib3
_dummy_thread grpc protobuf-3.4.0-py2.7-nspkg.pth urllib3-1.22.dist-info
easy_install.py grpcio-1.4.0.dist-info pyasn1 werkzeug
easy_install.pyc html pyasn1-0.3.7.dist-info Werkzeug-0.12.2.dist-info
enum http pyasn1_modules winreg
enum34-1.1.6.dist-info idna pyasn1_modules-0.1.4.dist-info xmlrpc
flask idna-2.6.dist-info queue
Flask-0.12.2.dist-info itsdangerous-0.24.dist-info reprlib
future itsdangerous.py requests
Since you're using dev_appserver.py it means you have a standard env application. And in the standard env all external dependencies need to be installed in your app, not on the local python instalation (which is what you did with your pip invocation).
From Installing a third-party library:
Create a directory to store your third-party libraries, such as lib/.
mkdir lib
Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous
step. For example:
pip install -t lib/ <library_name>
I'm trying to deploy my appengine go application to managed vm, and I keep getting this error
Pulling image: google/appengine-go
Traceback (most recent call last):
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/gcloud/gcloud.py", line 170, in <module>
main()
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/gcloud/gcloud.py", line 166, in main
_cli.Execute()
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/calliope/cli.py", line 385, in Execute
post_run_hooks=self.__post_run_hooks, kwargs=kwargs)
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/calliope/frontend.py", line 274, in _Execute
pre_run_hooks=pre_run_hooks, post_run_hooks=post_run_hooks)
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/calliope/backend.py", line 928, in Run
result = command_instance.Run(args)
File "/home/honeybooboo/google-cloud-sdk/lib/googlecloudsdk/appengine/app_commands/setup_managed_vms.py", line 39, in Run
args.image_version)
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/appengine/lib/images/pull.py", line 54, in PullBaseDockerImages
util.PullSpecifiedImages(docker_client, image_names, version, bucket)
File "/home/honeybooboo/google-cloud-sdk/./lib/googlecloudsdk/appengine/lib/images/util.py", line 232, in PullSpecifiedImages
'Error pulling {image}: {e}'.format(image=image_name, e=e))
googlecloudsdk.appengine.lib.images.util.DockerPullError: Error pulling google/appengine-go: 404 Client Error: Not Found ("No such id: localhost:49156/google/appengine-go")
My docker version
Client version: 1.3.0
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): c78088f
OS/Arch (client): linux/amd64
Server version: 1.3.0
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): c78088f
My Gcloud version
Google Cloud SDK 0.9.37
app 2014.11.18
app-engine-go-linux-x86_64 1.9.15
app-engine-java 1.9.15a
app-engine-managed-vms 2014.11.03
app-engine-python 1.9.15a
app-engine-python-extras 1.9.6
bq 2.0.18
bq-nix 2.0.18
compute 2014.11.25
core 2014.11.25
core-nix 2014.10.20
dns 2014.11.06
gae-go 2014.11.25
gae-go-nix 2014.09.10
gae-python 2014.05.06
gcutil 1.16.5
gcutil-nix 1.16.5
gsutil 4.6
gsutil-nix 4.6
preview 2014.11.18
preview-extensions-linux-x86_64 4.1
sql 2014.11.18
Sorry that you're having problems. We're aware of this issue and it is already fixed in the next SDK release (coming out in a week). As a temporary workaround please try to run
gcloud --verbosity debug preview app setup-managed-vms
(and choose Go in the list of options)
several times (until success) to get the base image for go runtime.
Another options is to try pulling the base go image (google/appengine-go) from containers-prod
bucket using google/docker-registry https://registry.hub.docker.com/u/google/docker-registry/
Pull the google/docker-registry
docker pull google/docker-registry
Get your credentials
gcloud auth print-refresh-token
Store your refresh token and your bucket (containers prod in a registry-params.env file)
cat registry-params.env
GCP_OAUTH2_REFRESH_TOKEN=your-refresh-token
GCS_BUCKET=containers-prod
Run registry
docker run -d --env-file=registry-params.env -p 5000:5000 google/docker-registry
Pull the image
docker pull localhost:5000/google/appengine-go
Retag the image
docker tag localhost:5000/google/appengine-go google/appengine-go
Remove old tag containing registry name
docker rmi localhost:5000/google/appengine-go
Check your image is in there. You'll see something like
docker images | grep google
You'll see something like
google/appengine-go latest 35ef8e2a9c5e 13 days ago 206 MB
Don't forget to stop your registry container
docker ps
docker stop <CONTAINER ID>
I am trying to test a Google App Engine app with dev_appserver.py, but when I run import Crypto I get the following excerpted from the IOError (i.e. No access) traceback:
...
import Crypto
...
File "/System/Library/Frameworks/Python.framework/Versions
/2.7/lib/python2.7/zipfile.py", line 867, in read
return self.open(name, "r", pwd).read()
File "/System/Library/Frameworks/Python.framework/Versions
/2.7/lib/python2.7/zipfile.py", line 882, in open
zef_file = open(self.filename, 'rb')
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google
/appengine/tools/dev_appserver_import_hook.py", line 592, in __init__
raise IOError(errno.EACCES, 'file not accessible', filename)
IOError: [Errno 13] file not accessible: '/Library/Python/2.7/site-packages
/pycrypto-2.3-py2.7-macosx-10.7-intel.egg'
I am on Mac OS X 10.7, with Google App Engine 1.6.6 using Python 2.7.
Since PyCrypto is supported on Google App Engine, I would expect it to work on the development server.
I am aware that dev_appserver.py prevents loading external files. However, I noted that appengine/tools/dev_appserver_import_hook.py seems to have all the requisite files in the whitelist (e.g._fastmath).
Note, in app.yaml I have
libraries:
- name: pycrypto
version: latest
It seems as though I am missing something obvious but crucial. Any thoughts would be appreciated.
EDIT For more details see: https://code.google.com/p/googleappengine/issues/detail?id=12129
Yes, you have to install the third-party library yourself.
Google explains exactly which versions the provide on their platform,
so this should not be any problem.
The best way to get through this is create a virtual environment and install the pycrypto inside that. The reason your libraries inside app.yaml is not detected is most probably because you have multiple versions of python installed in your machine and the version you used to run the program might not be the same version where you installed the libraries