PyDev and Eclipse - ImportError No Module Named - after refreshing interpreter - google-app-engine

I'm new to Eclipse and PyDev and have been stuck on this for while having had a look at quite a few answers to similar issues on here.
I'm trying to build a simple web app using PyDev, Eclipse, Python 2.7 and Flask (on Windows) and have followed this guide (https://cloud.google.com/appengine/docs/python/getting-started/python-standard-env) which all worked fine.
I made some small changes, but am currently stuck on the first step where I am trying to import pandas in my script (main.py)
I'm getting this error from the debugger when I try to import pandas from in my script
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "C:\Users\LONTI\workspace\Flask-app\main.py", line 3, in <module>
import pandas as pd
ImportError: No module named pandas
I've checked in Preferences > Interpreters > Python Interpreter that C:\Python27\lib\site-packages has been added (where my pandas module sits). I've also tried removing and adding the interpreter again but to no avail.
Also, in the editor I can see that pandas isn't Unresolved, so it seems like it can 'see' it. And in the workspace, under the Python > System Libs > lib/site-packages that pandas is also there.
I'm a bit at a loss where else to check.
main.py looks like this, where I've just cut out as much as possible to try and make sure there wasn't anything else that affect the import:
import logging
from flask import Flask, render_template, request
import pandas as pd
app = Flask(__name__)
#app.route('/form')
def form():
return render_template('form.html')
#app.route('/submitted', methods=['POST'])
def submitted_form():
name = request.form['name']
pc1 = request.form['pc1']
pc2 = request.form['pc2']
pc3 = request.form['pc3']
return render_template(
'submitted_form.html',
name=name,
pc1=pc1,
pc2=pc2,
pc3=pc3)
#app.errorhandler(500)
def server_error(e):
# Log the error and stacktrace.
logging.exception('An error occurred during a request.')
return 'An internal error occurred.', 500
Thanks in advance for your help and let me know if I need to provide any more info .

I think the issue is that you're using google-app-engine, which limits what's allowed to run.
Can Pandas run on Google App Engine for Python? has information which may be useful.

The import error in particular is caused by improper installation of pandas in your application. See Using third-party libraries. From Installing a third-party library:
In order to use a third-party library, copy it into a folder in your
project's source directory. The library must be implemented as pure
Python code with no C extensions. The code is uploaded to App Engine
with your application code, and counts towards file quotas.
This quote also ties into the answer mentioned by Fabio, it's unlikely you'll get this working on the standard GAE environment.
It might work on the flex environment - less restrictions, but that's a significantly different beast.

Related

Dependencies not installing from lib directory in google app engine

As far as I can tell, I have set up my flask app right. lib, containing all dependencies is at my root, and contains what Requirements.txt has in it. my appengine_config.py contains the below
print 'running app config yaya!'
from google.appengine.ext import vendor
import os
import sys
print os.path
print os.path.realpath
print os.path.realpath(__file__)
print os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
sys.path.insert(0,'./lib')
vendor.add('lib')
print 'I am the line after adding lib, it should have worked'
per all those print statements, nothing is erroring, but I am getting
No module named flask_sqlalchemy
after deploying and seeing a 500. What am I missing to get these suckers installed?
EDIT---------------
thanks and here --
running app config yaya!
18:36:29.499
['./lib', '/base/data/home/apps/s~nimble-poet-150223/20161221t183424.397920801519685819', '/base/data/home/runtimes/python27/python27_dist/lib/python27.zip', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/plat-linux2', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-tk', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-old', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-dynload', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/site-packages', '/base/data/home/runtimes/python27/python27_lib/versions/1', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/jinja2-2.6', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/markupsafe-0.15', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/webob-1.1.1', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/yaml-3.10']
18:36:29.499
at print dir(vendor)
['PYTHON_VERSION', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'add', 'os', 'site', 'sys']
18:36:29.503
at final print statement
I am the line after adding lib, it should have worked
18:36:29.948
the error
(/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~nimble-poet-150223/20161221t183424.397920801519685819/main.py", line 6, in <module>
from nimble import *
File "/base/data/home/apps/s~nimble-poet-150223/20161221t183424.397920801519685819/nimble/__init__.py", line 10, in <module>
from flask_sqlalchemy import SQLAlchemy, SignallingSession
ImportError: No module named flask_sqlalchemy
18:36:30.191
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
Not sure if it is related, but I am also getting https://code.google.com/p/google-cloud-sdk/issues/detail?id=729
but that fix doesnt help
---more edits
I moved everything from /site_pakcages up a dir to /lib and I am alot closer, some of my pages are even working! but now I get
ImportError: No module named _sqlite3
but the code reads
except ImportError as e:
try:
from sqlite3 import dbapi2 as sqlite # try 2.5+ stdlib name.
except ImportError as ee:
raise ee
return sqlite
why would there be an underscore in the import attempt here? help?
tldr: use appengine_config.py and copy your virtualenv to a folder called lib, then make SURE you are running the app via dev_appserver.py
(the below is via bash in ubuntu) SO after a long battle, I find that virtual env and gcloud dont play nice -
I copied everything from my virtual env dir
.../.virtualenvs/nimble/local/lib/python2.7/site-packages
into
[projectdir]/lib
and my appengine_config.py finally worked locally like it does in the cloud, but I absolutely HAVE to run
dev_appserver.py [my proj dir here]
or the google.appengine module wont load. did not know I should be using dev server. I feel very dumb.
for reference, heres the appengine_config.py
"""`appengine_config` gets loaded when starting a new application instance."""
print 'running app config yaya!'
from google.appengine.ext import vendor
vendor.add('lib')
print 'I am the line after adding lib, it should have worked'
import os
print os.getcwd()

Error with module using Cloud Storage with Python and his tutorial

I'm trying to test Google Cloud Storage to store images (I need it in an app that I'm developing) and I'm following the Bookshelf App tutorial that they have in his webpage.
I'm using python and the problem is that when I execute the requirementes.txt all packages have been installed fine, but when I try execute the code, I see this error:
...sandbox.py", line 948, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named cryptography.hazmat.bindings._openssl
I have been trying hundred of posibles solutions, reinstalling only the cryptography package, trying to use different versions of the same module, and installing other packages that contains it but anything resolved the problem.
The requirements contains this:
Flask==0.10.1
gcloud==0.9.0
gunicorn==19.4.5
oauth2client==1.5.2
Flask-SQLAlchemy==2.1
PyMySQL==0.7.1
Flask-PyMongo==0.4.0
PyMongo==3.2.1
six==1.10.0
I'm sure that it is a simple error but I don't find the way to solve it.
Any help will be welcome. Thanks.
EDIT:
When I try do this with a python program this work fine:
import os
from gcloud import storage
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key.json'
client = storage.Client(project='xxxxxxx')
bucket = client.get_bucket('yyyyyyy')
f = open('profile.jpg', 'rb')
blob = bucket.blob(f.name)
blob.upload_from_string(f.read(), 'image/jpeg')
url = blob.public_url
print url
Why I don't can use gcloud library without erros in a GAE app?
It seems you're following the bookshelf tutorial, but according to this line in your stacktrace:
...sandbox.py", line 948, in load_module
It hints that you're using dev_appserver.py to run the code. This isn't necessary for Managed VMs/Flexible unless you're using the compat runtime.
If this is the case, the tutorial provides correct instructions:
$ virtualenv env
$ source env/bin/activate
$ pip install -r requirements.txt
$ python main.py
(If this is not the case, please feel free to comment on this with more details about how you're running your application).

import pycrypto in dev_appserver.py on Google App Engine gives IOError

I am trying to test a Google App Engine app with dev_appserver.py, but when I run import Crypto I get the following excerpted from the IOError (i.e. No access) traceback:
...
import Crypto
...
File "/System/Library/Frameworks/Python.framework/Versions
/2.7/lib/python2.7/zipfile.py", line 867, in read
return self.open(name, "r", pwd).read()
File "/System/Library/Frameworks/Python.framework/Versions
/2.7/lib/python2.7/zipfile.py", line 882, in open
zef_file = open(self.filename, 'rb')
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google
/appengine/tools/dev_appserver_import_hook.py", line 592, in __init__
raise IOError(errno.EACCES, 'file not accessible', filename)
IOError: [Errno 13] file not accessible: '/Library/Python/2.7/site-packages
/pycrypto-2.3-py2.7-macosx-10.7-intel.egg'
I am on Mac OS X 10.7, with Google App Engine 1.6.6 using Python 2.7.
Since PyCrypto is supported on Google App Engine, I would expect it to work on the development server.
I am aware that dev_appserver.py prevents loading external files. However, I noted that appengine/tools/dev_appserver_import_hook.py seems to have all the requisite files in the whitelist (e.g._fastmath).
Note, in app.yaml I have
libraries:
- name: pycrypto
version: latest
It seems as though I am missing something obvious but crucial. Any thoughts would be appreciated.
EDIT For more details see: https://code.google.com/p/googleappengine/issues/detail?id=12129
Yes, you have to install the third-party library yourself.
Google explains exactly which versions the provide on their platform,
so this should not be any problem.
The best way to get through this is create a virtual environment and install the pycrypto inside that. The reason your libraries inside app.yaml is not detected is most probably because you have multiple versions of python installed in your machine and the version you used to run the program might not be the same version where you installed the libraries

ImportError while running Google App Engine test project

I am new to Google App Engine and so, I just followed the procedure to test an application which just prints "hello world".
(followed according to Google App Engine Documentation)
The project folder name is "GoogleApp"
This is my Python file:
File Name : sayHello.py
#!/usr/bin/env python
def main():
print "hello"
pass
if __name__ == '__main__':
main()
This the yaml file
File Name : app.yaml
application: GoogleApp
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: sayHello.py
When ever I just run the project in Google App Engine Launcher, this is the log error I am getting
2012-04-19 10:52:23 Running command: "['C:\\Python27\\pythonw.exe', 'C:\\Program Files\\Google\\google_appengine\\dev_appserver.py', '--admin_console_server=', '--port=8080', 'D:\\Code\\Projects\\IRCmathBot\\GoogleApp']"
Traceback (most recent call last):
File "C:\Program Files\Google\google_appengine\dev_appserver.py", line 125, in <module>
run_file(__file__, globals())
File "C:\Program Files\Google\google_appengine\dev_appserver.py", line 121, in run_file
execfile(script_path, globals_)
File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver_main.py", line 157, in <module>
from google.appengine.tools import appcfg
File "C:\Program Files\Google\google_appengine\google\appengine\tools\appcfg.py", line 41, in <module>
import mimetypes
File "C:\Python27\lib\mimetypes.py", line 29, in <module>
import urllib
File "C:\Python27\lib\urllib.py", line 26, in <module>
import socket
File "C:\Python27\lib\socket.py", line 47, in <module>
import _socket
ImportError: Module use of python25.dll conflicts with this version of Python.
2012-04-19 10:52:25 (Process exited with code 1)
Actually, I use Python 2.7. The above log says that python25.dll conflicts with version of python.
I don't know what exactly it means but I could interpret that
It should either support only python 2.5 or
It I should have used python 2.5 code.
as the 2nd isn't possible and Google supports 2.7, I don't know what is this error.
Try to figure out myself but couldn't succeed.
I had a very similar problem (while running the tutorial code), and solved it by checking my PythonPath system environment variable.
For me, OpenCV edited the PythonPath variable to point to its own directory structure instead of the base Python installation itself.
You can edit the environment variables by opening up the 'Advanced System Properties' window and clicking on the 'Environment Variables' button. The 'PythonPath' variable under 'System Variables'. At the minimum, it should include the path of your python executable, such as 'C:\Python27\'.
Have you tried in command-line? Forget the launcher.
I had same problem! I re-installed Python several times, changed versions. It was hilarious.
With version 2.5 launcher can start the server, but the application doesn't work, but I can upload it online through launcher.
With Python ver. 2.7, launcher displays same problems you're dealing with. I also can't get the server started locally. But command-line works well, both starting the server locally and uploading online.
So I'll stick with version 2.7. Hope it helps, let me know..
You have threadsafe enabled, so you should use WSGI handlers. That could be the problem. I would replace your code with the following:
File: helloworld.py
import webapp2
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('Hello, WebApp World!')
app = webapp2.WSGIApplication([('/', MainPage)])
File: app.yaml
application: GoogleApp
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: helloworld.app
If using Python2.7 on Google AppEngine, refer to this official documentation to create your first "Hello World" Application.
I fixed this after 3 hrs of effort. Reinstall cannot solve my problem, messing around the user/system environment variable also not work.
Follow this step work:
Open Google Drive Perference, turn off auto run after win start up
turn off Google Drive
or simply uninstall google drive (Google Drive seems using Python 2.5 _socket library
uninstall python and GAE
reboot
install python27 and GAE
Most importantly, specify a new path, do not use the default c:\python27 again (I changed it to c:\mypython27. same as GAE, use a new path.
Then it works again.
Something's seriously broken with your Python 2.7 installation - parts of it are referencing Python 2.5 libraries. You should uninstall all versions of Python on your machine, and reinstall the version(s) you need.
I guess you've added the GAE path into the environment variable PATH? Just remove it.
I had exactly the same problem with you, it seems to me that Python 2.7 will try to use an incorrect _socket.pyd lib under GAE path, removing GAE path from PATH will fix this.
It needs no extra coding, just a tiny configuration of your win OS.
Since my OS language is Chinese, names of icons/links below are translated from Chinese by myself, which i cannot guarantee to be correct :(
Right click on the "Computer" icon, choose 'Properties', you'll see the system panel showing some basic system information;
Click on the "Advanced System Settings" link, you'll see the system property panel containing several tabs;
Choose the "Advanced" tab, there should be a button named "Environment variables" on the bottom, click on it then you can see a dialog displaying all environment variables of your system;
Find and edit the variable named "PATH" or "Path" (case insensible): if its value contains a path to your GAE, then remove the GAE path.
For me it was that I did set the System Variable PYTHONPATH to '.'. Deleting the variable did the job!
I set PYTHON_PATH to
"C:\Program Files\Google\google_appengine;C:\Program
Files\Google\google_appengine\lib\antlr3;C:\Program
Files\Google\google_appengine\lib\django_0_96;C:\Program
Files\Google\google_appengine\lib\fancy_urllib;C:\Program
Files\Google\google_appengine\lib\graphy;C:\Program
Files\Google\google_appengine\lib\ipaddr;C:\Program
Files\Google\google_appengine\lib\simplejson;C:\Program
Files\Google\google_appengine\lib\webob;C:\Program
Files\Google\google_appengine\lib\yaml\lib"
but then I realize that django 0.96 require python 2.5, so use django 1.4, and it works fine.
"C:\Program Files\Google\google_appengine;C:\Program
Files\Google\google_appengine\lib\antlr3;C:\Program
Files\Google\google_appengine\lib\django-1.4;C:\Program
Files\Google\google_appengine\lib\fancy_urllib;C:\Program
Files\Google\google_appengine\lib\graphy;C:\Program
Files\Google\google_appengine\lib\ipaddr;C:\Program
Files\Google\google_appengine\lib\simplejson;C:\Program
Files\Google\google_appengine\lib\webob;C:\Program
Files\Google\google_appengine\lib\yaml\lib"
If you have same problem as above, just check PATH, PYTHON_PATH to see whether mismatch version between library and python. Hope this help.
Some programs may modify the PYTHONPATH environment variable. If you check it's value (System -> Environment variables) and ensure it is set to the correct python 2.7 path (default C:\Python27) then this should fix the issue. In my case something had set it to my google appengine path.
ImportError: Module use of python25.dll conflicts with this version of Python.
The problem is that python25.dll conflicts with your Python 2.7 installation. This python25.dll comes from the greatly outdated Google App Engine Launcher for Windows and is located in C:\Program Files\Google\google_appengine\launcher
It is not a problem of your Python 2.7 install - it is a problem of Launcher, who injects its own Python path into app environment. To test that, add the following lines at the top of google_appengine/dev_appserver.py after first imports:
from pprint import pprint
pprint(sys.path)
Then re-run you app to get fresh log and see for yourself:
...
'C:\\OpenCV2.3\\opencv\\build\\python\\2.7',
'C:\\Google\\google_appengine\\launcher',
'C:\\Windows\\system32\\python27.zip',
'C:\\Python27\\DLLs',
...
I got the launcher from Python .msi SDK 1.8.8, and it is of course suxx that Google still ships several versions of SDK with broken GUI tool. To fix that, add the following after first imports in dev_appserver.py:
# --- Repair sys.path after broken GAE Windows Launcher
# see http://stackoverflow.com/questions/10222342/
ROOT = os.path.abspath(os.path.dirname(__file__))
LAUNCHPATH = os.path.join(ROOT, 'launcher')
if LAUNCHPATH in sys.path:
sys.stderr.write('[BUG] GAE Windows Launcher detected. Fixing..\n')
sys.path.remove(LAUNCHPATH)
# /--
Unfortunately, you'll need to repeat the procedure when new version of GAE comes out. There is an issue #8568 on AppEngine tracker that you can star and comment to make it fixed someday.
Do you have a PYTHONPATH variable in Environment Variables? delete it and restart the Google App Engine Launcher,it will work!

App Engine python27 urlfetch error: "[Errno 11003] getaddrinfo failed"

I'm trying to switch my app engine app from python 2.5 to 2.7 and urlfetch.fetch() just doesn't want to work (on the dev server at least; I haven't deployed it yet). If I do something as simple as:
file = urlfetch.fetch(url="http://www.google.com")
I get the following error:
File "C:\workspace\DjangoServer\src\mycode.py", line 167, in request
file = urlfetch.fetch(url="http://www.google.com")
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 264, in fetch
return rpc.get_result()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 366, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2 [Errno 11003] getaddrinfo failed
Searching the internet has suggested it might be a firewall or proxy issue, but I turned off the firewall and I'm not behind a proxy.
This works fine in the python 2.5 dev server, but not the python 2.7 dev server. If I open a python 2.7 console and manually runsocket.getaddrinfo("www.google.com", 80) it works there as well.
Any ideas? Has anyone else encountered this?
Ok, I have been able to reproduce this issue and after a lot of going back and forth in git from a version that was working to a version that was not working I found that (at least in my case) the problem was in django-nonrel (because I updated all at once: django-nonrel and appengine).
So, please do the following: edit djangoappengine/settings_base.py and change
try:
from google.appengine.api import apiproxy_stub_map
except ImportError:
from .boot import setup_env
setup_env()
from djangoappengine.utils import on_production_server, have_appserver
to:
from djangoappengine.utils import on_production_server, have_appserver
if not on_production_server:
from .boot import setup_env
setup_env()
The real issue seems to be that in a previous version of django-nonrel, the import was actually written as: from google.appengine.api import api_proxy_stub_map (which was never found and the setup_env() was always called), so, when the import was fixed, the setup_env() was no longer called (which in my case made the error appear).
A 2nd note is that for some reason this was only happening when running inside of Eclipse/PyDev and it did work on the command line (although the environment variables seemed the same for me, so, I'm unsure why running from inside java actually made a difference).

Resources