I have created my GAE application in directory "my_application". Inside this directory I created a .py file and named it "my_scrypt".
The contents of "my_scrypt" in the beginning were as following:
print 'Content-Type: text/plain'
print ''
print 'This is my first application'
Then I ran it locally on my machine (Windows XP) in the installed browser (Mozilla FireFox) with "GAE Launcher" - everything was fine - I could see that sentence ("This is my first application") on the screen.
Then I deployed this application to GAE (again with the help of "GAE Launcher") - everything was fine again - I could see the same sentence on the screen.
Then I changed the contents of "my_scrypt" a bit:
from twill.commands import *
config('use_tidy', '0')
go ("http://us.yahoo.com/")
showlinks()
Downloaded "twill0.9" (from here), chose and copied "twill" folder from there, and pasted it in "my_application" directory.
When I ran this new application locally (with "GAE Launcher") everything was fine - I could see a list of yahoo.com links on the screen, but when I uploaded this application to GAE, I received only an error message.
Why is it so? I don't think it's because the version of mechanize being used by twill here is too old - the code in "my_script" is so simple, any version of mechanize must be able to handle it.
Does GAE accept twill (as an external module) at all?
You can view the stack trace of the error in the "Update 1" section right below (↓).
UPDATE 1:
(This update is my answer to Nick)
Hello, Nick. I checked the admin console, so here is the stack trace:
<type 'exceptions.ImportError'>: No module named fcntl
Traceback (most recent call last):
File "/base/data/home/apps/silkybutton/1.344911014283513184/bumper.py", line 1, in <module>
from twill.commands import *
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/__init__.py", line 52, in <module>
from shell import TwillCommandLoop
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/shell.py", line 9, in <module>
from twill import commands, parse, __version__
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/commands.py", line 70, in <module>
from browser import TwillBrowser
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/browser.py", line 17, in <module>
from _browser import PatchedMechanizeBrowser
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/_browser.py", line 9, in <module>
from utils import FixedHTTPBasicAuthHandler, FunctioningHTTPRefreshProcessor
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/utils.py", line 12, in <module>
import subprocess
File "/base/data/home/apps/silkybutton/1.344911014283513184/twill/other_packages/subprocess.py", line 378, in <module>
import fcntl
Twill is trying to import 'subprocess'. This is a Python module for spawning threads, and it's not available on App Engine. You'll either need to see if you can persuade Twill to work without spawning processes (probably by modifying the code), or you'll need to use mechanize or simply urlfetch directly.
Related
I'm new to Eclipse and PyDev and have been stuck on this for while having had a look at quite a few answers to similar issues on here.
I'm trying to build a simple web app using PyDev, Eclipse, Python 2.7 and Flask (on Windows) and have followed this guide (https://cloud.google.com/appengine/docs/python/getting-started/python-standard-env) which all worked fine.
I made some small changes, but am currently stuck on the first step where I am trying to import pandas in my script (main.py)
I'm getting this error from the debugger when I try to import pandas from in my script
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "C:\Users\LONTI\workspace\Flask-app\main.py", line 3, in <module>
import pandas as pd
ImportError: No module named pandas
I've checked in Preferences > Interpreters > Python Interpreter that C:\Python27\lib\site-packages has been added (where my pandas module sits). I've also tried removing and adding the interpreter again but to no avail.
Also, in the editor I can see that pandas isn't Unresolved, so it seems like it can 'see' it. And in the workspace, under the Python > System Libs > lib/site-packages that pandas is also there.
I'm a bit at a loss where else to check.
main.py looks like this, where I've just cut out as much as possible to try and make sure there wasn't anything else that affect the import:
import logging
from flask import Flask, render_template, request
import pandas as pd
app = Flask(__name__)
#app.route('/form')
def form():
return render_template('form.html')
#app.route('/submitted', methods=['POST'])
def submitted_form():
name = request.form['name']
pc1 = request.form['pc1']
pc2 = request.form['pc2']
pc3 = request.form['pc3']
return render_template(
'submitted_form.html',
name=name,
pc1=pc1,
pc2=pc2,
pc3=pc3)
#app.errorhandler(500)
def server_error(e):
# Log the error and stacktrace.
logging.exception('An error occurred during a request.')
return 'An internal error occurred.', 500
Thanks in advance for your help and let me know if I need to provide any more info .
I think the issue is that you're using google-app-engine, which limits what's allowed to run.
Can Pandas run on Google App Engine for Python? has information which may be useful.
The import error in particular is caused by improper installation of pandas in your application. See Using third-party libraries. From Installing a third-party library:
In order to use a third-party library, copy it into a folder in your
project's source directory. The library must be implemented as pure
Python code with no C extensions. The code is uploaded to App Engine
with your application code, and counts towards file quotas.
This quote also ties into the answer mentioned by Fabio, it's unlikely you'll get this working on the standard GAE environment.
It might work on the flex environment - less restrictions, but that's a significantly different beast.
As far as I can tell, I have set up my flask app right. lib, containing all dependencies is at my root, and contains what Requirements.txt has in it. my appengine_config.py contains the below
print 'running app config yaya!'
from google.appengine.ext import vendor
import os
import sys
print os.path
print os.path.realpath
print os.path.realpath(__file__)
print os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
sys.path.insert(0,'./lib')
vendor.add('lib')
print 'I am the line after adding lib, it should have worked'
per all those print statements, nothing is erroring, but I am getting
No module named flask_sqlalchemy
after deploying and seeing a 500. What am I missing to get these suckers installed?
EDIT---------------
thanks and here --
running app config yaya!
18:36:29.499
['./lib', '/base/data/home/apps/s~nimble-poet-150223/20161221t183424.397920801519685819', '/base/data/home/runtimes/python27/python27_dist/lib/python27.zip', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/plat-linux2', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-tk', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-old', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-dynload', '/base/data/home/runtimes/python27/python27_dist/lib/python2.7/site-packages', '/base/data/home/runtimes/python27/python27_lib/versions/1', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/jinja2-2.6', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/markupsafe-0.15', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/webob-1.1.1', '/base/data/home/runtimes/python27/python27_lib/versions/third_party/yaml-3.10']
18:36:29.499
at print dir(vendor)
['PYTHON_VERSION', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'add', 'os', 'site', 'sys']
18:36:29.503
at final print statement
I am the line after adding lib, it should have worked
18:36:29.948
the error
(/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~nimble-poet-150223/20161221t183424.397920801519685819/main.py", line 6, in <module>
from nimble import *
File "/base/data/home/apps/s~nimble-poet-150223/20161221t183424.397920801519685819/nimble/__init__.py", line 10, in <module>
from flask_sqlalchemy import SQLAlchemy, SignallingSession
ImportError: No module named flask_sqlalchemy
18:36:30.191
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
Not sure if it is related, but I am also getting https://code.google.com/p/google-cloud-sdk/issues/detail?id=729
but that fix doesnt help
---more edits
I moved everything from /site_pakcages up a dir to /lib and I am alot closer, some of my pages are even working! but now I get
ImportError: No module named _sqlite3
but the code reads
except ImportError as e:
try:
from sqlite3 import dbapi2 as sqlite # try 2.5+ stdlib name.
except ImportError as ee:
raise ee
return sqlite
why would there be an underscore in the import attempt here? help?
tldr: use appengine_config.py and copy your virtualenv to a folder called lib, then make SURE you are running the app via dev_appserver.py
(the below is via bash in ubuntu) SO after a long battle, I find that virtual env and gcloud dont play nice -
I copied everything from my virtual env dir
.../.virtualenvs/nimble/local/lib/python2.7/site-packages
into
[projectdir]/lib
and my appengine_config.py finally worked locally like it does in the cloud, but I absolutely HAVE to run
dev_appserver.py [my proj dir here]
or the google.appengine module wont load. did not know I should be using dev server. I feel very dumb.
for reference, heres the appengine_config.py
"""`appengine_config` gets loaded when starting a new application instance."""
print 'running app config yaya!'
from google.appengine.ext import vendor
vendor.add('lib')
print 'I am the line after adding lib, it should have worked'
import os
print os.getcwd()
I attempted to use Appengine SDK version 1.7.6 but was receiving fatal errors (same as outlined below) so reverted to 1.7.5, where my app works fine. However, the errors I saw in 1.7.6 recur when I attempt to run the experimental devappserver in 1.7.5. I am running Python 2.7 on a win 7 machine. Standard install selections for AppEngine.
These error messages refer to permissions relating to the PYTHON_EGG_CACHE, but...
- 1) They do not occur for the 'old' dev_appserver
- 2) The application can successfully extract files to the location noted. In fact, I changed the path of the PYTHON_EGG_CACHE to another location, gave it wide open permissions, and still received the error.
- 3) I have pyyaml in my site-packages, so am not sure why it would need to do a temporary extract anyway.
Any help looking for a resolution to this would be appreciated, since I'll be forced to use the new devappserver in a couple of releases time.
Command line error dump below:
python "C:\Program Files (x
86)\Google\google_appengine\devappserver2.py" .
WARNING 2013-03-25 19:10:41,029 devappserver2.py:497] devappserver2.py is curre
ntly experimental but will eventually replace dev_appserver.py in the App Engine
Python SDK. For more information and to report bugs, please see: http://code.go
ogle.com/p/appengine-devappserver2-experiment/
INFO 2013-03-25 19:10:41,171 appcfg.py:618] Checking for updates to the SDK.
WARNING 2013-03-25 19:10:44,525 simple_search_stub.py:975] Could not read searc
h indexes from c:\users\ben\appdata\local\temp\appengine.dev~upskillme-main.Ben\
search_indexes
INFO 2013-03-25 19:10:44,539 api_server.py:148] Starting API server at: http
://localhost:49302
INFO 2013-03-25 19:10:44,555 dispatcher.py:96] Starting server "default" run
ning at: http://localhost:8080
INFO 2013-03-25 19:10:44,571 admin_server.py:112] Starting admin server at:
http://localhost:8000
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\_python_runtime.py", line
150, in <module>
run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\_python_runtime.py", line
146, in run_file
execfile(script_path, globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
vappserver2\python\runtime.py", line 39, in <module>
from google.appengine.ext.remote_api import remote_api_stub
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remo
te_api\remote_api_stub.py", line 75, in <module>
import yaml
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml-3.10\yaml\__init
__.py", line 14, in <module>
from cyaml import *
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml-3.10\yaml\cyaml.
py", line 5, in <module>
from _yaml import CParser, CEmitter
File "C:\Python27\lib\site-packages\pyyaml-3.10-py2.7-win32.egg\_yaml.py", lin
e 7, in <module>
File "C:\Python27\lib\site-packages\pyyaml-3.10-py2.7-win32.egg\_yaml.py", lin
e 4, in __bootstrap__
File "C:\Python27\lib\site-packages\pkg_resources.py", line 882, in resource_f
ilename
self, resource_name
File "C:\Python27\lib\site-packages\pkg_resources.py", line 1351, in get_resou
rce_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "C:\Python27\lib\site-packages\pkg_resources.py", line 1406, in _extract_
resource
manager.extraction_error() # report a user-friendly error
File "C:\Python27\lib\site-packages\pkg_resources.py", line 928, in extraction
_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Error 5] Access is denied: 'C:\\Users\\Ben\\AppData\\Roaming\\Python-Eggs\\py
yaml-3.10-py2.7-win32.egg-tmp\\_yaml.pyd'
The Python egg cache directory is currently set to:
C:\Users\Ben\AppData\Roaming\Python-Eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
Exception in thread Thread-4:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 551, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
vappserver2\server.py", line 1045, in _loop_adjusting_instances
self._adjust_instances()
If anyone comes along with the same issue, the resolution that worked for me was as follows:
Unzip the PyYAML-3.10-py2.7-win32.egg in your python27 site-packages
directory into a new directory within site-packages. You'll need to
unzip it to into a directory name like NEWPyYAML-3.10-py2.7-win32.egg
to avoid name conflicts.
Rename your original egg. to something else (e.g., OLDPyYAML-3.10-py2.7-win32.egg).
Rename your new directory to PyYAML-3.10-py2.7-win32.egg
All references to the egg will now actually point to a directory in your site-packages containing the unzipped python files. The new devappserver should be able to work with this.
Unsure how to unzip an egg in windows? See here (presuming you have Winzip or similar installed):
http://mail.python.org/pipermail/chicago/2007-July/002301.html
I'm currently trying to access my app using remote_api_shell.py as documented in appengine at http://code.google.com/appengine/articles/remote_api.html. I am using python 27 in windows 7 and I am able to access the shell - I've remotely cleared the memcache for example. I am having trouble accessing the datastore with the error
no module found named <module name>
My directory stucture something like this
/app.yaml
/main.py
/users/models.py
In the windows shell I type the following (in accordance with the walkthrough at the above link)
C:\Users\user>remote_api_shell.py -s appname.appspot.com
App Engine remote_api shell
Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)]
The db, users, urlfetch, and memcache modules are imported.
s~appname> import appname
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: No module named appname
s~appname> from google.appengine.ext import db
s~appname> from users.models import *
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: No module named users.models
s~appname> from users import models
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: No module named users
I've tried a few different iterations but just cannot seem to access any of my data. I've also tried to access another model in a directory /website/models.py but also no success. Any ideas on how to get this to work? cheers.
Try setting the module search path to include the directory where your app lives. This can either be done by setting the environment variable PYTHONPATH (though I've forgotten how to do this on Windows), or, inside your shell session, by appending to sys.path.
The reason is that the remote_api_shell script lives in a different directory than your app, so its directory gets added to the path instead of your app's directory, even if the latter is the current directory.
I guess you could also copy remote_api_shell.py into your app directory...
Obviously you can copy the models.py into remote_api_shell.py directory {:^)
I'm trying to switch my app engine app from python 2.5 to 2.7 and urlfetch.fetch() just doesn't want to work (on the dev server at least; I haven't deployed it yet). If I do something as simple as:
file = urlfetch.fetch(url="http://www.google.com")
I get the following error:
File "C:\workspace\DjangoServer\src\mycode.py", line 167, in request
file = urlfetch.fetch(url="http://www.google.com")
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 264, in fetch
return rpc.get_result()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 366, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2 [Errno 11003] getaddrinfo failed
Searching the internet has suggested it might be a firewall or proxy issue, but I turned off the firewall and I'm not behind a proxy.
This works fine in the python 2.5 dev server, but not the python 2.7 dev server. If I open a python 2.7 console and manually runsocket.getaddrinfo("www.google.com", 80) it works there as well.
Any ideas? Has anyone else encountered this?
Ok, I have been able to reproduce this issue and after a lot of going back and forth in git from a version that was working to a version that was not working I found that (at least in my case) the problem was in django-nonrel (because I updated all at once: django-nonrel and appengine).
So, please do the following: edit djangoappengine/settings_base.py and change
try:
from google.appengine.api import apiproxy_stub_map
except ImportError:
from .boot import setup_env
setup_env()
from djangoappengine.utils import on_production_server, have_appserver
to:
from djangoappengine.utils import on_production_server, have_appserver
if not on_production_server:
from .boot import setup_env
setup_env()
The real issue seems to be that in a previous version of django-nonrel, the import was actually written as: from google.appengine.api import api_proxy_stub_map (which was never found and the setup_env() was always called), so, when the import was fixed, the setup_env() was no longer called (which in my case made the error appear).
A 2nd note is that for some reason this was only happening when running inside of Eclipse/PyDev and it did work on the command line (although the environment variables seemed the same for me, so, I'm unsure why running from inside java actually made a difference).