I create an app-engine endpoint api, which I am loading using GoogleAppengineLauncher. The api launches fine. But when I try to load api explorer for testing, I get an error due to the line from multiprocessing import Process. My research led me to this site. But that's not working for me. Does anyone know how to fix this?
from multiprocessing import Process
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 84, in <module>
import _multiprocessing
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 861, in load_module
raise ImportError
ImportError
INFO 2013-03-25 23:46:32,229 server.py:528] "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 500 -
INFO 2013-03-25 23:46:32,229 server.py:528] "GET /_ah/api/discovery/v1/apis HTTP/1.1" 500 60
In this group thread, one of the Python 2.7 App Engine runtime engineer point out to alternatives (namely the futures package) that should work with the new Python 2.7 threading support.
Alternatively you could use the fetch_data_async functions to read from a blob without blocking.
fetch_data_rpc = blobstore.fetch_data_async(...)
other_processing()
upload_url = fetch_data_rpc.get_result()
Related
Attempting to use them gives me the following error:
Traceback (most recent call last):
File "/env/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/env/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 196, in init_process
self.patch()
File "/env/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 75, in patch
fileno=s.sock.fileno()))
File "/env/lib/python3.7/site-packages/gevent/_socket3.py", line 114, in __init__
self._sock = self._gevent_sock_class(family, type, proto, fileno)
OSError: [Errno 92] Protocol not available: 'protocol'
Are sockets not supported in the python 3 environment? Is this the reason it won't work?
My application spends a lot of time blocked on other web servers, so using async workers is necessary for decent performance. In the meantime I will try using eventlet workers instead.
EDIT: Including some more context around usage
app.yaml
runtime: python37
instance_class: F2
entrypoint: gunicorn -c gunicorn.conf.py -b :$PORT my_project.server:app
gunicorn.conf.py
# Recommended number of workers based on instance size:
# https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
workers = 2
# Use an asynchronous worker as most of the work is waiting for websites to load
worker_class = 'gevent'
Since it is during gunicorn launch I don't believe the app definition context will be useful. However for completeness I'm launching a basic flask application. Something like:
my_project/server.py
from flask import Flask
from my_project.handlers import run
app = Flask(__name__)
app.register_blueprint(run, url_prefix='/run')
I had the same error. I think we are limited to GAE Flexible or Cloud Run on GKE if we want to use gevent.
GAE Standard runs in a sandbox (gVisor) and some operations are limited (I noticed this trying Cloud Run which had the same issue with gevent)
The error you are seeing is this one:
Container Sandbox Limitation: Unsupported syscall
getsockopt(0x5,0x1,0x26,0x7e11bd8ddf20,0x7e11bd8ddf24,0x4)
I'm new to Eclipse and PyDev and have been stuck on this for while having had a look at quite a few answers to similar issues on here.
I'm trying to build a simple web app using PyDev, Eclipse, Python 2.7 and Flask (on Windows) and have followed this guide (https://cloud.google.com/appengine/docs/python/getting-started/python-standard-env) which all worked fine.
I made some small changes, but am currently stuck on the first step where I am trying to import pandas in my script (main.py)
I'm getting this error from the debugger when I try to import pandas from in my script
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "C:\Users\LONTI\workspace\Flask-app\main.py", line 3, in <module>
import pandas as pd
ImportError: No module named pandas
I've checked in Preferences > Interpreters > Python Interpreter that C:\Python27\lib\site-packages has been added (where my pandas module sits). I've also tried removing and adding the interpreter again but to no avail.
Also, in the editor I can see that pandas isn't Unresolved, so it seems like it can 'see' it. And in the workspace, under the Python > System Libs > lib/site-packages that pandas is also there.
I'm a bit at a loss where else to check.
main.py looks like this, where I've just cut out as much as possible to try and make sure there wasn't anything else that affect the import:
import logging
from flask import Flask, render_template, request
import pandas as pd
app = Flask(__name__)
#app.route('/form')
def form():
return render_template('form.html')
#app.route('/submitted', methods=['POST'])
def submitted_form():
name = request.form['name']
pc1 = request.form['pc1']
pc2 = request.form['pc2']
pc3 = request.form['pc3']
return render_template(
'submitted_form.html',
name=name,
pc1=pc1,
pc2=pc2,
pc3=pc3)
#app.errorhandler(500)
def server_error(e):
# Log the error and stacktrace.
logging.exception('An error occurred during a request.')
return 'An internal error occurred.', 500
Thanks in advance for your help and let me know if I need to provide any more info .
I think the issue is that you're using google-app-engine, which limits what's allowed to run.
Can Pandas run on Google App Engine for Python? has information which may be useful.
The import error in particular is caused by improper installation of pandas in your application. See Using third-party libraries. From Installing a third-party library:
In order to use a third-party library, copy it into a folder in your
project's source directory. The library must be implemented as pure
Python code with no C extensions. The code is uploaded to App Engine
with your application code, and counts towards file quotas.
This quote also ties into the answer mentioned by Fabio, it's unlikely you'll get this working on the standard GAE environment.
It might work on the flex environment - less restrictions, but that's a significantly different beast.
Im having trouble getting app engine to accept a 3rd party library.
ive copied it into the the app engine directory, but i keep getting the error, "invalid object, the library tweepy is not supported" in blah blah blah/app.yaml
could somebody point out the obvious mistake please?
thanks
ok, ive made a few changes to the code and now i get this error, hopefully it is more useful
2012-10-15 20:09:36 Running command: "['C:\\Python27\\pythonw.exe', 'C:\\Program Files\\Google\\google_appengine\\dev_appserver.py', '--admin_console_server=', '--port=8083', 'C:\\Documents and Settings\\ladds\\My Documents\\udacity\\whycantisigh']"
WARNING 2012-10-15 19:09:54,140 rdbms_mysqldb.py:74] The rdbms API is not available because the MySQLdb library could not be loaded.
WARNING 2012-10-15 19:09:56,171 datastore_file_stub.py:513] Could not read datastore data from c:\docume~1\ladds\locals~1\temp\dev_appserver.datastore
WARNING 2012-10-15 19:09:56,203 dev_appserver.py:3394] Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO 2012-10-15 19:09:56,328 dev_appserver_multiprocess.py:647] Running application dev~whycantisigh on port 8083: #http://localhost:8083
INFO 2012-10-15 19:09:56,328 dev_appserver_multiprocess.py:649] Admin console is available at: #http://localhost:8083/_ah/admin
WARNING 2012-10-15 19:10:26,171 py_zipimport.py:139] Can't open zipfile C:\Python27\lib\site-packages\oauth2-1.0.2-py2.7.egg: IOError: [Errno 13] file not accessible: 'C:\Python27\lib\site-packages\oauth2-1.0.2-py2.7.egg'
WARNING 2012-10-15 19:10:26,171 py_zipimport.py:139] Can't open zipfile C:\Python27\lib\site-packages\ply-3.4-py2.7.egg: IOError: [Errno 13] file not accessible: 'C:\Python27\lib\site-packages\ply-3.4-py2.7.egg'
WARNING 2012-10-15 19:10:26,187 py_zipimport.py:139] Can't open zipfile C:\Python27\lib\site-packages\tweepy-1.11-py2.7.egg: IOError: [Errno 13] file not accessible: 'C:\Python27\lib\site-packages\tweepy-1.11-py2.7.egg'
INFO 2012-10-15 19:10:30,171 dev_appserver.py:2884] "GET / HTTP/1.1" 405 -
INFO 2012-10-15 19:18:04,250 py_zipimport.py:148] zipimporter('C:\Python27\lib\site-packages\oauth2-1.0.2-py2.7.egg', '')
INFO 2012-10-15 19:18:04,250 py_zipimport.py:148] zipimporter('C:\Python27\lib\site-packages\ply-3.4-py2.7.egg', '')
INFO 2012-10-15 19:18:04,250 py_zipimport.py:148] zipimporter('C:\Python27\lib\site-packages\tweepy-1.11-py2.7.egg', '')
INFO 2012-10-15 19:18:06,640 dev_appserver.py:2884] "GET / HTTP/1.1" 405 -
im sorry its not formatted clearly, but it will only let me format the first line.
anyway, the gist of the problem seems to be that 3rd party libraries i have downloaded into python(and that work in the shell) arent accessible by app engine.
thanks
https://developers.google.com/appengine/kb/libraries
Many Native C Python modules are disabled with Google App Engine. The Python 2.7 runtime supports more modules than the Python 2.5 runtime.
It would help if you showed your app.yaml and the actual error message instead of blah blah blah.
But most likely you put tweepy in app.yaml and you don't need to.
Edit
To use 3rd party libraries with GAE, you need to put a copy of the library in your project folder. Otherwise, it won't get uploaded to the server on deploy. Installing it to your python path won't work.
On March, 13 www.cliws.com hosted on Google App Engine stopped to fetch HN's RSS feed located at http://news.ycombinator.com/rss
www.cliws.com is RSS-reader like Google Reader but a bit better ;) It regularly checks various RSS-feeds, so from logs I can see exact date when Google stopped to download HN's RSS feed (it worked fine before).
The problematic RSS-feed url is fetched without any issues locally in the development SDK, but can't be downloaded in the production.
Please see bellow demonstration of the problem:
s~cliwws> from google.appengine.api import urlfetch
s~cliwws> print urlfetch.fetch('http://news.ycombinator.com/rss').status_code
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/dogada/sources/python/google_appengine/google/appengine/api/urlfetch.py", line 263, in fetch
return rpc.get_result()
File "/home/dogada/sources/python/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "/home/dogada/sources/python/google_appengine/google/appengine/api/urlfetch.py", line 365, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2
s~cliwws> print urlfetch.fetch('http://www.osnews.com/feed/kind/News').status_code
200
s~cliwws> print urlfetch.fetch('http://googleappengine.blogspot.com/atom.xml').status_code
200
s~cliwws> print urlfetch.fetch('http://google.com').status_code
200
I also filled bug for this issue: 'http://code.google.com/p/googleappengine/issues/detail?id=7181'
What are possible reasons for this strange behavior?
Just guessing here, but I assume that the web server stopped accepting your requests due to a flagged user agent, IP address or other meta data.
Since you can not influence the outgoing IP on GAE, try setting the user agent to "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21" to test if that changes anything.
I'm trying to switch my app engine app from python 2.5 to 2.7 and urlfetch.fetch() just doesn't want to work (on the dev server at least; I haven't deployed it yet). If I do something as simple as:
file = urlfetch.fetch(url="http://www.google.com")
I get the following error:
File "C:\workspace\DjangoServer\src\mycode.py", line 167, in request
file = urlfetch.fetch(url="http://www.google.com")
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 264, in fetch
return rpc.get_result()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 366, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2 [Errno 11003] getaddrinfo failed
Searching the internet has suggested it might be a firewall or proxy issue, but I turned off the firewall and I'm not behind a proxy.
This works fine in the python 2.5 dev server, but not the python 2.7 dev server. If I open a python 2.7 console and manually runsocket.getaddrinfo("www.google.com", 80) it works there as well.
Any ideas? Has anyone else encountered this?
Ok, I have been able to reproduce this issue and after a lot of going back and forth in git from a version that was working to a version that was not working I found that (at least in my case) the problem was in django-nonrel (because I updated all at once: django-nonrel and appengine).
So, please do the following: edit djangoappengine/settings_base.py and change
try:
from google.appengine.api import apiproxy_stub_map
except ImportError:
from .boot import setup_env
setup_env()
from djangoappengine.utils import on_production_server, have_appserver
to:
from djangoappengine.utils import on_production_server, have_appserver
if not on_production_server:
from .boot import setup_env
setup_env()
The real issue seems to be that in a previous version of django-nonrel, the import was actually written as: from google.appengine.api import api_proxy_stub_map (which was never found and the setup_env() was always called), so, when the import was fixed, the setup_env() was no longer called (which in my case made the error appear).
A 2nd note is that for some reason this was only happening when running inside of Eclipse/PyDev and it did work on the command line (although the environment variables seemed the same for me, so, I'm unsure why running from inside java actually made a difference).