Trying to import testbed from GAE SDK 1.95 with Python2.7.8 on osX Maverics 10.9.5
and I'm getting a InvalidCertificateException error.
from google.appengine.ext import testbed
File "/usr/local/google_appengine/google/appengine/ext/testbed/__init__.py", line 120, in <module>
from google.appengine.api import urlfetch_stub
File "/usr/local/google_appengine/google/appengine/api/urlfetch_stub.py", line 34, in <module>
_fancy_urllib_InvalidCertException = fancy_urllib.InvalidCertificateException
AttributeError: 'module' object has no attribute 'InvalidCertificateException'
I looked at the fancy_url module and the InvalidCertificateException class is there, so I don't understand why it's not importing.
Apparently others have had the same error, so I attempt to correct it by deleting: urlfetch_cacerts.txt AND cacerts.txt from:
GoogleAppEngineLauncher/Contents/Resources/GoogleAppEngineDefault.bundle/Contents/Resources/google_appengine/lib/cacerts/
Apparently the GAE installer creates a nested directory, this was fixed by copying the contents in:
cd /usr/local/google_appengine/lib
cp fancy_urllib/fancy_urllib/__init__.py fancy_urllib/__init__.py
This is how the module is incorrectly structured, it looks like these 2 init.py files are duplicate:
/usr/local/google_appengine/lib/fancy_urllib/__init__.py # this file is empty
/usr/local/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py # this file contains the functions.
FIXED THE ERROR
Related
I have created a Volttron agent and trying to use pandas library.
But I am getting error of pandas not being installed, while its there.
ERROR:volttron.platform.packaging:b'Traceback (most recent call last):\n File "setup.py", line 11, in <module>
_temp = __import__(agent_module, globals(), locals(), [\'__version__\'], 0)
File "/tmp/tmpd_5srlls/pkg/weather/agent.py", line 12, in <module>
import pandas as pd
File "/home/pi/volttron/env/lib/python3.7/site-packages/pandas/__init__.py", line 17, in <module>
"Unable to import required dependencies:\\n" + "\\n".join(missing_dependencies)
ImportError: Unable to import required dependencies:\nnumpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from "/home/pi/volttron/env/bin/python"
* The NumPy version is: "1.19.5"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.Original error was: libf77blas.so.3: cannot open shared object file: No such file or directory\n\n'
Traceback (most recent call last):
File "scripts/install-agent.py", line 340, in <module>
if not os.path.isfile(opts.package):
File "/usr/lib/python3.7/genericpath.py", line 30, in isfile
st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
The shared library cannot be located. Please try:
sudo apt-get install libatlas-base-dev
I tried install a package in python3 called Spexxy, using this:
pip3 install -r requirements.txt
python3 setup.py install
Using iPython in the shell, I can well import the package with import spexxy but when I try to use anything from the package, first I have to use an exclamation point before the command like for example !spexxytools grid create, otherwise I have the error:
File "<ipython-input-3-b744a07ba8b3>", line 1
spexxytools grid create --from-filename "lte(?P<Teff>\d{5})-(?P<logg>\d\.\d\d)(?P<FeH>[+-]\d\.\d)(\.Alpha=(?P<Alpha>[+-]\d\.\d\d))?\.PHOENIX"
^
SyntaxError: invalid syntax
But even when using the exclamation point, I now have the error:
File "/home/geoffrey/.local/bin/spexxy", line 4, in <module>
__import__('pkg_resources').run_script('spexxy==2.3', 'spexxy')
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 658, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1429, in run_script
.format(**locals()),
pkg_resources.ResolutionError: Script 'scripts/spexxy' not found in metadata at '/home/geoffrey/.local/lib/python3.6/site-packages/spexxy-2.3.dist-info'
Can you help me find what's wrong ?
setuptools.find_packages()
Packages are only recognized if they include an init.py file.
So if you put
...
packages = setuptools.find_packages(),
...
scripts=['scripts/spexxy'],
you will get this error.
To fix it either write spexxy as a module in your source or declare packages = manually instead of using find_packages().
Edit
spexxy setup.py is using
packages=find_packages(include=['spexxy', 'spexxy.*']),
scripts=['bin/spexxy', 'bin/spexxytools'],
But the bin directory is missing, so it should be:
packages=find_packages(include=['spexxy', 'spexxy.*', 'bin/*']),
scripts=['bin/spexxy', 'bin/spexxytools'],
I am trying to import some files from my Google Cloud Storage bucket to a Google Colab Notebook. However, I face the following error using the snippet suggested by google.
Code:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
# Download a file based on its file ID.
#
# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz
file_id = '1NKNtpuP95NKNMTVao6qt8G-OJs_xmbnC'
downloaded = drive.CreateFile({'id': file_id})
print('Downloaded content "{}"'.format(downloaded.GetContentString()))
Error:
W0331 16:31:52.011976 139994604533632 __init__.py:44] file_cache is unavailable when using oauth2client >= 4.0.0
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
from . import file_cache
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
'file_cache is unavailable when using oauth2client >= 4.0.0')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0
Downloaded content "{
"attention_probs_dropout_prob": 0.1,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"type_vocab_size": 2,
"vocab_size": 28996
}
"
I understand that I have some modules missing and I need to install them. However, I tried
!export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
!echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
!apt update && apt install google-cloud-sdk
And faced the following error:
deb http://packages.cloud.google.com/apt main
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1326 100 1326 0 0 53040 0 --:--:-- --:--:-- --:--:-- 53040
OK
E: Malformed entry 1 in list file /etc/apt/sources.list.d/google-cloud-sdk.list (Component)
E: The list of sources could not be read.
I am new to Google Colab Notebooks and while I understand that it's a simple interactive notebook I am not able to get the hang of this.
Thanks,
It is a bug in googleapiclient, the code bellow will sove it.
import logging
logging.getLogger('googleapiclient.discovery_cache').setLevel(logging.ERROR)
Ref : Github Reference Error
I run the sourcecode of TWE model. I need to compile the C extension of python. I have installed the Microsoft Visual C++ Compiler for Python 2.7 and Cython.
First, I need to run the TWE/train.py:
import gensim
sentence_word = gensim.models.word2vec.LineSentence("tmp/word.file")
print "Training the word vector..."
w = gensim.models.Word2Vec(sentence_word,size=400, workers=20)
sentence = gensim.models.word2vec.CombinedSentence("tmp/word.file","tmp/topic.file")
print "Training the topic vector..."
w.train_topic(topic_number, sentence)
print "Saving the topic vectors..."
w.save_topic("output/topic_vector.txt")
print "Saving the word vectors..."
w.save_wordvector("output/word_vector.txt")`
Second, TWE/gensim/models/wor2vec.py:
try:
raise ImportError # ignore for now
from gensim_addons.models.word2vec_inner import train_sentence_sg,train_sentence_cbow, FAST_VERSION, train_sentence_topic
except ImportError:
try:
import pyximport
print 'import pyximport'
models_dir = os.path.dirname(__file__) or os.getcwd()
print 'models_dir'
pyximport.install(setup_args={"include_dirs": [models_dir, get_include()]})
print 'pyximport' # is the follow code's problem
from word2vec_inner import train_sentence_sg, train_sentence_cbow,
FAST_VERSION, train_sentence_topic
print 'from word2vec'
except:
FAST_VERSION = -1
def train_sentence_sg(model, sentence, alpha, work=None):
...
def train_sentence_cbow(model, sentence, alpha, work=None, neu1=None):
...
class Word2Vec(utils.SaveLoad):
...
def train(self, sentences, total_words=None, word_count=0, chunksize=100):
if FAST_VERSION < 0:
import warnings
warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")
logger.info("training model with %i workers on %i vocabulary and %i features, "
"using 'skipgram'=%s 'hierarchical softmax'=%s 'subsample'=%s and 'negative sampling'=%s" %
(self.workers, len(self.vocab), self.layer1_size, self.sg, self.hs, self.sample, self.negative))
def worker_train():
...
if self.sg:
job_words = sum(train_sentence_topic(self, sentence, alpha, work) for sentence in job)
else:
ob_words = sum(train_sentence_cbow(self, sentence, alpha, work, neu1) for sentence in job)`
...
Thrid, I haved compiled the TWE/gensim/models/word2vec_inner.pyx with a setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy
extensions = [
Extension("word2vec_inner", ["word2vec_inner.pyx"],
include_dirs=[numpy.get_include()])
]
setup(
name="word2vec_inner",
ext_modules=cythonize(extensions),
)
by using the command 'python setup.py install', I have complied the word2vec_inner.pyx. But it appears the follow errors:
E:\Python27\python.exe D:/pycharm/TWE/TWE1/train.py wordmap.txt model-final.tassign 100
import pyximport
models_dir
pyximport
word2vec_inner.c
e:\python27\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg : Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\Users\hp\.pyxbld\temp.win32-2.7\Release\pyrex\gensim\models\word2vec_inner.c(15079) : warning C4244:
'initializing' : conversion from 'double' to 'float', possible loss of data
C:\Users\hp\.pyxbld\temp.win32-2.7\Release\pyrex\gensim\models\word2vec_inner.c(15085) : warning C4244 : 'initializing' : conversion from 'double' to 'float', possible loss of data
LINK : fatal error LNK1104: cannot open file 'C:\Users\hp\.pyxbld\lib.win32-2.7\gensim\models\word2vec_inner.pyd'
Training the word vector...
D:\pycharm\TWE\TWE1\gensim\models\word2vec.py:410: UserWarning: Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`
warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")
PROGRESS: at 100.00% words, alpha 0.02500, 2556 words/s
Training the topic vector...
D:\pycharm\TWE\TWE1\gensim\models\word2vec.py:882: UserWarning: Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`
warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")
Exception in thread Thread-23:
Traceback (most recent call last):
File "E:\Python27\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "E:\Python27\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "D:\pycharm\TWE\TWE1\gensim\models\word2vec.py", line 909, in worker_train
job_words = sum(train_sentence_topic(self, sentence, alpha, work) for sentence in job)
File "D:\pycharm\TWE\TWE1\gensim\models\word2vec.py", line 909, in <genexpr>
job_words = sum(train_sentence_topic(self, sentence, alpha, work) for sentence in job)
NameError: global name 'train_sentence_topic' is not defined
Saving the topic vectors...
Saving the word vectors...
Process finished with exit code 0
I have checked that the .pyx file was compiled correctly and also installed the cython. in conclusion, it can't import train_sentence_sg,train_sentence_cbow, FAST_VERSION, train_sentence_topic from gensim/models/word2vec_inner or gensim_addons/models/word2vec_inner. So it appears these problems. But why? I have compiled the .pyx file correctly in both two directionarys. Anyone can help me? This problem haved troubled me several days. Please help me, thank you!
I ran into the same problem with PyCharm 2018.1 + Python 3.6.2.
This line is the key:
LINK : fatal error LNK1104: cannot open file 'C:\Users\hp\.pyxbld\lib.win32-2.7\gensim\models\word2vec_inner.pyd'
This error message is misleading. With Python, this error actually means:
cannot open file to write to it.
The file is probably locked for writing by some process, so the linker cannot complete its job.
Solution 1
A previous Python line import word2vec_inner locked writing to the file. Reset the Python console to remove the lock:
Solution 2
Use Process Explorer to find out which program is locking the file. Use Ctrl-F, then type the name of the locked file in question, and it will give you the process that locked the file.
Solution 3
Exit Pycharm, then precompile the Cython file into a package on the command line. In case this link changes, here is a mirror:
Imagine a simple “hello world” script in a file hello.pyx:
def say_hello_to(name):
print("Hello %s!" % name)
The following could be a corresponding setup.py script:
from distutils.core import setup
from Cython.Build import cythonize
setup(
name = 'Hello world app',
ext_modules = cythonize("hello.pyx"),
)
To build, run python setup.py build_ext --inplace. Then simply start
a Python session and do from hello import say_hello_to and use the
imported function as you see fit.
One caveat if you use setuptools instead of distutils, the default
action when running python setup.py install is to create a zipped egg
file which will not work with cimport for pxd files when you try to
use them from a dependent package. To prevent this, include
zip_safe=False in the arguments to setup().
I am getting the following error when trying to run the nosetest from my GAE project:
nosetests --nologcapture --with-gae --without-sandbox --gae-lib-root=/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine
but I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/nosetests", line 8, in <module>
load_entry_point('nose==1.3.4', 'console_scripts', 'nosetests')()
File "/Library/Python/2.7/site-packages/nose/core.py", line 121, in __init__
**extra_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/Library/Python/2.7/site-packages/nose/core.py", line 145, in parseArgs
self.config.configure(argv, doc=self.usage())
File "/Library/Python/2.7/site-packages/nose/config.py", line 346, in configure
self.plugins.configure(options, self)
File "/Library/Python/2.7/site-packages/nose/plugins/manager.py", line 284, in configure
cfg(options, config)
File "/Library/Python/2.7/site-packages/nose/plugins/manager.py", line 99, in __call__
return self.call(*arg, **kw)
File "/Library/Python/2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(*arg, **kw)
File "/Library/Python/2.7/site-packages/nosegae.py", line 87, in configure
from google.appengine.tools import old_dev_appserver as dev_appserver
ImportError: cannot import name old_dev_appserver
The sys.path reads:
'/Users/dsinha/Downloads/eclipse/plugins/org.python.pydev_3.9.0.201411111611/pysrc', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/antlr3', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/ipaddr', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/yaml-3.10', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/rsa', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/pyasn1', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/simplejson', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/django-1.4', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/goo...
so the app engine libraries should be getting pulled in. Actually its first failing to pull in dev_appserver, and then trying and failing to pull old_dev_appserver
The dir inside the appengine/lib file is:
bash-3.2$ ls
__init__.py appcfg.py backends_xml_parser.py dispatch_xml_parser.py handler_generator.py php_cli.py value_mixin.pyc
__init__.pyc appcfg_java.py boolean_action.py docker handler_generator.pyc queue_xml_parser.py web_xml_parser.py
adaptive_thread_pool.py appengine_rpc.py boolean_action.pyc dos_xml_parser.py jarfile.py queue_xml_parser.pyc web_xml_parser.pyc
api_server.py appengine_rpc.pyc bulkload_client.py download_appstats.py java_quickstart.py remote_api_shell.py xml_parser_utils.py
app_engine_config_exception.py appengine_rpc_httplib2.py bulkloader.py endpointscfg.py java_quickstart.pyc requeue.py xml_parser_utils.pyc
app_engine_config_exception.pyc augment_mimetypes.py cron_xml_parser.py gen_protorpc.py java_utils.py sdk_update_checker.py yaml_translator.py
app_engine_web_xml_parser.py augment_mimetypes.pyc dev-channel-js.js handler.py java_utils.pyc sdk_update_checker.pyc yaml_translator.pyc
app_engine_web_xml_parser.pyc backends_conversion.py devappserver2 handler.pyc os_compat.py value_mixin.py
bash-3.2$ pwd
/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/google/appengine/tools
I also tried to find the modules available inside the google.appengine.tools package:
>>> import pkgutil
>>> [name for _, name, _ in pkgutil.iter_modules(['testpkg'])]
[]
This problem started occuring after I upgraded to App Engine 1.9.10 (to use the async search features). In a problem that I think is related, when I try to run the debug server from PyDev, it just silently terminated on any page request (localhost:8080).
Running dev_appserver . from the command line works fine though.
Nose-GAE broke with App Engine 1.9.17: https://github.com/Trii/NoseGAE/issues/6
Downgrading to 1.9.15 made the problem go away temporarily while waiting for the issue to be resolved by nose-gae