I am getting error when using PIL on the cloud in google app engine
ImportError: cannot import name _imaging
at <module>():66 (Image.py:66 in /base/data...../lib/PIL)
at <module>():6 (storage.py:6 in /base/data/home/...../modules/common)
....
at <module>():1 (main.py:1 in /base/data/home/apps/..../....456)
The application seems to load fine locally.
I list PIL as a dependnecy in the app.yaml libraries section
libraries:
- name: webapp2
version: latest
- name: jinja2
version: latest
- name: PIL
version: "1.1.7"
I also used pip install PIL -t /lib to save it to my ./lib folder for local development
Additional info:
I am using PIL because it is an imaging library that has google app engine support https://cloud.google.com/appengine/docs/python/tools/built-in-libraries-27
remove the PIL module from the lib folder that you installed on your own.
go to app.yaml and paste this under the libraries section (if there isn't one paste this whole) -
libraries:
- name: PIL
version: latest
call the appengine based module by -
from PIL import Image
You're effectively attempting to run your copy of the PIL library instead of the one provided (see the file paths in the stack trace, which include details about your app which you edited out).
Not only that you don't need to upload the PIL lib, you actually should not upload it as the one provided is likely a version specifically customized for the GAE python sandbox.
So make sure the local PIL installation doesn't place the library or a link to it anywhere inside the app's dir, to prevent it from being accidentally vendored into your app and uploaded with it.
Related
I am trying to generate signUrl in python 2.7 using v4 signing process as given here
below is the code given on the link:
def generate_singed_url(bucket_name, blob_name):
"""Generates a v4 signed URL for downloading a blob.
Note that this method requires a service account key file. You can not use
this if you are using Application Default Credentials from Google Compute
Engine or from the Google Cloud SDK.
"""
# bucket_name = 'your-bucket-name'
# blob_name = 'your-object-name'
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version="v4",
# This URL is valid for 15 minutes
expiration=datetime.timedelta(minutes=15),
# Allow GET requests using this URL.
method="GET",
)
print("Generated GET signed URL:")
print(url)
print("You can use this URL with any user agent, for example:")
print("curl '{}'".format(url))
return url
This is how I am trying to import the storage:
from google.cloud import storage
But I am getting the error as:
File "<input>", line 1, in <module>
File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
ImportError: No module named google.cloud
I tried installing google-cloud-storage library also tried installing lots of other google specific libraries but it's still giving the same import error.
Tried:
ImportError: No module named google.cloud
ModuleNotFoundError: No module named 'google.cloud' (uninstalling the libraries and again installing)
Edit: how can i generate signurl using python2.7 and app engine ?
I was running two python versions in my mac: python2.7 and python3 , the libraries was already installed for python3 but was missing for python2.7.
Used below command to install the library:
python2.7 -m pip install --upgrade google-cloud-storage --user
....Answering based on your comments....
If you were using Python3, you would have a virtual env and a requirements.txt file and when you deploy your project to Production, GAE would first install the contents of your requirements.txt file before running your App.
Since you're running Python2, you don't have that requirements.txt file and virtual env concept with GAE. Your App has to be uploaded together with any third party library you need (i.e. any library outside of these has to uploaded with your App). You do this via the 'lib' folder that I mentioned in the comments - (instructions for creating the folder can be found here). I believe the instructions are simple enough to follow.
I would say to first try using the lib concept on your local machine (this would mean installing the cloud storage library to that folder). It also means you have to run your app with dev_appserver.py.
Note that when you install google cloud storage client to the lib folder and run your app with dev_appserver.py, GAE will use the package in your lib folder instead of the one installed globally on your laptop.
If it works (i.e. you're able to create signed urls on your local machine), then go ahead and deploy to production.
If you have problems creating the lib folder and installing packages to it, let me know.
I am experiencing a strange issue with dependencies when deploying my application to Google App Engine (Python 2.7).
I have specified my dependencies in the requirements.txt file:
Flask==1.0.2
werkzeug>=0.14
flask-cors
twilio
httplib2
gunicorn
Jinja2
google-cloud-ndb
exponent_server_sdk
These work fine with the local development server when installed locally with pip from the requirements.txt folder.
However, when I deploy to app engine, it appears that the module exponent_server_sdk has not been installed (ImportError: No module named exponent_server_sdk).
I normally deploy by pushing to a git repository which is then deployed with Google Cloud Build ("gcr.io/cloud-builders/gcloud"). There are no errors when I deploy and other dependencies in the requirements.txt file work fine such as twilio or Jinja2.
Whilst investigating this I tried pushing directly from my development machine with gcloud app deploy. When I do this, for some strange reason, exponent_server_sdk is available and works correctly.
This behaviour seems very strange to me and I can't find any documentation of people experiencing similar errors. I wonder if anyone can give me any guidance about what might be causing this issue or places I can look to find errors (for example logs of the process of installing the requirements.txt file during deployment/instance startup).
In response to the comments:
I am running in the standard environment.
My cloudbuild.yaml looks like this:
steps:
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "app.yaml", "dispatch.yaml", "service_1.yaml", "service_2.yaml", "service_3.yaml", "index.yaml"]
timeout: "1600s"
The .yaml file for the service in which I am experiencing the error looks like this:
runtime: python27
api_version: 1
threadsafe: true
service: notifications
libraries:
- name: ssl
version: latest
handlers:
- url: /.*
script: notifications.app
For the (1st gen) standard environment the deployment process does not use a requirements.txt file, the libraries should be installed inside your app. From Copying a third-party library:
Use pip (version 6 or later) with the -t <directory> flag to
copy the libraries into the folder you created in the previous step.
For example:
pip install -t lib/ <library_name>
Yes, if you want you can use a requirements.txt file to do the installation inside your app, but that file itself is not used by the deployment process.
Since you're not deploying directly from the environment in which you had the libraries installed you'd have to add the installation directory to the git repo as well.
I installed the App engine SDK, then did a pip install requests and pip install requests-toolbelt (under VENV). when running the app in the local development server - I get the following error:
ERROR 2017-05-31 18:14:52,315 cgi.py:122] Traceback (most recent call last):
File "/Users/assafshamia/Freebird/Techradar/dev/scraper.py", line 8, in <module>
import requests
ImportError: No module named requests
I followed the steps of installing a 3rd party library (appengine_config.py and install requests under /lib)
what is going on???
I was able to resolve this by adding the following code to my Python app (main.py) in order to access the libraries at /lib:
import sys
sys.path.insert(0, 'lib')
Per the docs, you need to add the requests library code to your application directory. pip install is not enough.
You can include third party Python libraries with your application by
putting the code in your application directory.
Edit:
Additionally:
The include path of the Python module includes your application's root directory, which is the directory containing the app.yaml file. Python modules that you create in your application's root directory are available using a path from the root. Don't forget to create the required init.py files in your sub-directories so that Python recognizes those sub-directories as packages.
Yesterday I updated my google cloud SDK along with the python runtime. Now an app I was able to run locally in the SDK complains about the PIL / Python Imaging Library not being available:
bash-3.2$ dev_appserver.py . --host=0.0.0.0
INFO 2014-05-31 17:07:52,313 devappserver2.py:706] Skipping SDK update check.
WARNING 2014-05-31 17:07:52,319 api_server.py:378] Could not initialize images API; you are likely missing the Python "PIL" module.
Not true, however. I DO have PIL installed and it works swimmingly with jpegs, which is what I expect:
bash-3.2$ ipython
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
In [1]: from PIL import Image
In [2]: my_image = Image.open('motorcycle.jpg')
In [3]: my_image.show()
Works just fine. FWIW, I declare my app.yaml use of PIL as such:
libraries:
...
- name: webapp2
version: latest
- name: PIL
version: latest
I've tested it, so it's not a false negative. When I try to use the GAE images module, it's blows up with the same problem. It works just fine in production. Clearly Python has what it needs to work. Why is App Engine unable to make use of it?
The problem I was having had to do with having a clash between a brew version of python installed versus the system version. Fixing my shell's path to point to the correct python installation sorted this problem, too. For GAE users: I also had to make sure my app engine SDK path was correct as well.
If you're having problems installing or getting PIL to work, understand that there are many dependencies involved with PIL, including what file format codecs you will want to support (there is a PNG library, a TIFF library and a JPEG library), and whether you want to draw text in your images and need truetype font support via the freetype library. Know that you may have PIL installed and no codec support libraries and effectively PIL won't be usable.
A good starting place for this issue might be this other question:
How to deal with Linux/Python dependencies?
I am using Google App engine on Mac OSX 10.7.4. When I load PIL from the commandline, everything works fine. However, when I load it from the GAE local environment, i.e.:
import Image
Gives me the error:
ImportError: No module named Image
This line works fine:
from google.appengine.api import images
So I don't believe there's a problem with my installation. The app works fine when I deploy it to GAE. Also, I load the PIL library in app.yaml:
libraries:
- name: webapp2
version: "2.5.1"
- name: lxml
version: latest
- name: PIL
version: latest
- name: numpy
version: latest
I am also using python in /usr/bin/python, 2.7.1. I have PIL installed using:
brew install pip
pip install pil
You need to use: from PIL import Image.