Installing google appengine on Google Colab Notebook - google-app-engine

I am trying to import some files from my Google Cloud Storage bucket to a Google Colab Notebook. However, I face the following error using the snippet suggested by google.
Code:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
# Download a file based on its file ID.
#
# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz
file_id = '1NKNtpuP95NKNMTVao6qt8G-OJs_xmbnC'
downloaded = drive.CreateFile({'id': file_id})
print('Downloaded content "{}"'.format(downloaded.GetContentString()))
Error:
W0331 16:31:52.011976 139994604533632 __init__.py:44] file_cache is unavailable when using oauth2client >= 4.0.0
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
from . import file_cache
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
'file_cache is unavailable when using oauth2client >= 4.0.0')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0
Downloaded content "{
"attention_probs_dropout_prob": 0.1,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"type_vocab_size": 2,
"vocab_size": 28996
}
"
I understand that I have some modules missing and I need to install them. However, I tried
!export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
!echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
!apt update && apt install google-cloud-sdk
And faced the following error:
deb http://packages.cloud.google.com/apt main
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1326 100 1326 0 0 53040 0 --:--:-- --:--:-- --:--:-- 53040
OK
E: Malformed entry 1 in list file /etc/apt/sources.list.d/google-cloud-sdk.list (Component)
E: The list of sources could not be read.
I am new to Google Colab Notebooks and while I understand that it's a simple interactive notebook I am not able to get the hang of this.
Thanks,

It is a bug in googleapiclient, the code bellow will sove it.
import logging
logging.getLogger('googleapiclient.discovery_cache').setLevel(logging.ERROR)
Ref : Github Reference Error

Related

Error Importing Pandas in Volttron Platform

I have created a Volttron agent and trying to use pandas library.
But I am getting error of pandas not being installed, while its there.
ERROR:volttron.platform.packaging:b'Traceback (most recent call last):\n File "setup.py", line 11, in <module>
_temp = __import__(agent_module, globals(), locals(), [\'__version__\'], 0)
File "/tmp/tmpd_5srlls/pkg/weather/agent.py", line 12, in <module>
import pandas as pd
File "/home/pi/volttron/env/lib/python3.7/site-packages/pandas/__init__.py", line 17, in <module>
"Unable to import required dependencies:\\n" + "\\n".join(missing_dependencies)
ImportError: Unable to import required dependencies:\nnumpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from "/home/pi/volttron/env/bin/python"
* The NumPy version is: "1.19.5"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.Original error was: libf77blas.so.3: cannot open shared object file: No such file or directory\n\n'
Traceback (most recent call last):
File "scripts/install-agent.py", line 340, in <module>
if not os.path.isfile(opts.package):
File "/usr/lib/python3.7/genericpath.py", line 30, in isfile
st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
The shared library cannot be located. Please try:
sudo apt-get install libatlas-base-dev

Using wagtail start mysite returns syntax error

I am using pip, virtualenv and Python 3 to install wagtail. I am working on Ubuntu.
To backtrack a bit:
I was trying to install initially and got this error:
unicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 47: ordinal not in range(128)
I fixed this by changing my env to use Python 3.
This allows wagtail to start installing, but returned this list of errors while installing:
SyntaxError: invalid syntax
*** Error compiling '/home/ubuntu/workspace/env/build/Django/django/utils/feedgenerator.py'...
File "/home/ubuntu/workspace/env/build/Django/django/utils/feedgenerator.py", line 82
**kwargs,
^
SyntaxError: invalid syntax
*** Error compiling '/home/ubuntu/workspace/env/build/Django/django/utils/functional.py'...
File "/home/ubuntu/workspace/env/build/Django/django/utils/functional.py", line 12
return _curried_func(*args, *moreargs, **{**kwargs, **morekwargs})
^
SyntaxError: invalid syntax
*** Error compiling '/home/ubuntu/workspace/env/build/Django/django/utils/log.py'...
File "/home/ubuntu/workspace/env/build/Django/django/utils/log.py", line 229
)
^
SyntaxError: invalid syntax
*** Error compiling '/home/ubuntu/workspace/env/build/Django/django/utils/tree.py'...
l File "/home/ubuntu/workspace/env/build/Django/django/utils/tree.py", line 76
return hash((self.__class__, self.connector, self.negated, *make_hashable(self.children)))
^
SyntaxError: can use starred expression only as assignment target
It gave me a success message despite the syntax errors.
Successfully installed wagtail html5lib Django six Unidecode draftjs-exporter djangorestframework django-taggit pytz django-modelcluster Willow Pillow beautifulsoup4 requests django-treebeard webencodings urllib3 chardet idna certifi
Cleaning up...
After trying to start wagtail I have this final error message:
Traceback (most recent call last):
File "/home/ubuntu/workspace/env/bin/wagtail", line 7, in <module>
from wagtail.bin.wagtail import main
File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/wagtail/bin/wagtail.py", line 10, in <module>
from django.core.management import ManagementUtility
File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 11, in <module>
from django.conf import settings
File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/django/conf/__init__.py", line 18, in <module>
from django.utils.functional import LazyObject, empty
File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/django/utils/functional.py", line 12
return _curried_func(*args, *moreargs, **{**kwargs, **morekwargs})
This final error is the one I have not been able to solve. I opened the file functional.py and reviewed the contents and it looks correct to me. Any ideas on how to fix the error or what the root cause could be?
You are running Python 3.4, which is not supported by the current version of Django (2.1). You should either upgrade to Python 3.5 or above, or downgrade Django to 2.0.x by running pip install "Django>=2.0,<2.1".

Nose GAE : Cannot import dev_appserver, but app engine is still in PYTHONPATH

I am getting the following error when trying to run the nosetest from my GAE project:
nosetests --nologcapture --with-gae --without-sandbox --gae-lib-root=/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine
but I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/nosetests", line 8, in <module>
load_entry_point('nose==1.3.4', 'console_scripts', 'nosetests')()
File "/Library/Python/2.7/site-packages/nose/core.py", line 121, in __init__
**extra_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/Library/Python/2.7/site-packages/nose/core.py", line 145, in parseArgs
self.config.configure(argv, doc=self.usage())
File "/Library/Python/2.7/site-packages/nose/config.py", line 346, in configure
self.plugins.configure(options, self)
File "/Library/Python/2.7/site-packages/nose/plugins/manager.py", line 284, in configure
cfg(options, config)
File "/Library/Python/2.7/site-packages/nose/plugins/manager.py", line 99, in __call__
return self.call(*arg, **kw)
File "/Library/Python/2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(*arg, **kw)
File "/Library/Python/2.7/site-packages/nosegae.py", line 87, in configure
from google.appengine.tools import old_dev_appserver as dev_appserver
ImportError: cannot import name old_dev_appserver
The sys.path reads:
'/Users/dsinha/Downloads/eclipse/plugins/org.python.pydev_3.9.0.201411111611/pysrc', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/antlr3', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/ipaddr', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/yaml-3.10', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/rsa', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/pyasn1', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/simplejson', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/lib/django-1.4', '/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/goo...
so the app engine libraries should be getting pulled in. Actually its first failing to pull in dev_appserver, and then trying and failing to pull old_dev_appserver
The dir inside the appengine/lib file is:
bash-3.2$ ls
__init__.py appcfg.py backends_xml_parser.py dispatch_xml_parser.py handler_generator.py php_cli.py value_mixin.pyc
__init__.pyc appcfg_java.py boolean_action.py docker handler_generator.pyc queue_xml_parser.py web_xml_parser.py
adaptive_thread_pool.py appengine_rpc.py boolean_action.pyc dos_xml_parser.py jarfile.py queue_xml_parser.pyc web_xml_parser.pyc
api_server.py appengine_rpc.pyc bulkload_client.py download_appstats.py java_quickstart.py remote_api_shell.py xml_parser_utils.py
app_engine_config_exception.py appengine_rpc_httplib2.py bulkloader.py endpointscfg.py java_quickstart.pyc requeue.py xml_parser_utils.pyc
app_engine_config_exception.pyc augment_mimetypes.py cron_xml_parser.py gen_protorpc.py java_utils.py sdk_update_checker.py yaml_translator.py
app_engine_web_xml_parser.py augment_mimetypes.pyc dev-channel-js.js handler.py java_utils.pyc sdk_update_checker.pyc yaml_translator.pyc
app_engine_web_xml_parser.pyc backends_conversion.py devappserver2 handler.pyc os_compat.py value_mixin.py
bash-3.2$ pwd
/Users/dsinha/Dropbox/code/google-cloud-sdk/platform/google_appengine/google/appengine/tools
I also tried to find the modules available inside the google.appengine.tools package:
>>> import pkgutil
>>> [name for _, name, _ in pkgutil.iter_modules(['testpkg'])]
[]
This problem started occuring after I upgraded to App Engine 1.9.10 (to use the async search features). In a problem that I think is related, when I try to run the debug server from PyDev, it just silently terminated on any page request (localhost:8080).
Running dev_appserver . from the command line works fine though.
Nose-GAE broke with App Engine 1.9.17: https://github.com/Trii/NoseGAE/issues/6
Downgrading to 1.9.15 made the problem go away temporarily while waiting for the issue to be resolved by nose-gae

GAE SDK 1.9.5 and an InvalidCertificateException

Trying to import testbed from GAE SDK 1.95 with Python2.7.8 on osX Maverics 10.9.5
and I'm getting a InvalidCertificateException error.
from google.appengine.ext import testbed
File "/usr/local/google_appengine/google/appengine/ext/testbed/__init__.py", line 120, in <module>
from google.appengine.api import urlfetch_stub
File "/usr/local/google_appengine/google/appengine/api/urlfetch_stub.py", line 34, in <module>
_fancy_urllib_InvalidCertException = fancy_urllib.InvalidCertificateException
AttributeError: 'module' object has no attribute 'InvalidCertificateException'
I looked at the fancy_url module and the InvalidCertificateException class is there, so I don't understand why it's not importing.
Apparently others have had the same error, so I attempt to correct it by deleting: urlfetch_cacerts.txt AND cacerts.txt from:
GoogleAppEngineLauncher/Contents/Resources/GoogleAppEngineDefault.bundle/Content‌​s/Resources/google_appengine/lib/cacerts/
Apparently the GAE installer creates a nested directory, this was fixed by copying the contents in:
cd /usr/local/google_appengine/lib
cp fancy_urllib/fancy_urllib/__init__.py fancy_urllib/__init__.py
This is how the module is incorrectly structured, it looks like these 2 init.py files are duplicate:
/usr/local/google_appengine/lib/fancy_urllib/__init__.py # this file is empty
/usr/local/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py # this file contains the functions.
FIXED THE ERROR

SyntaxError when trying to create a new app using django nonrel

I'm playing around with django-nonrel and google app engine when I get the following error trying to create a new app.
Traceback (most recent call last):
File "./manage.py", line 2, in <module>
from django.core.management import execute_manager
File "/home/levi/Projects/DoneBox/django/__init__.py", line 14
if svn_rev != u'SVN-unknown':
^
SyntaxError: invalid syntax
The file that caused the exception is included below.
VERSION = (1, 3, 0, 'final', 0)
def get_version():
version = '%s.%s' % (VERSION[0], VERSION[1])
if VERSION[2]:
version = '%s.%s' % (version, VERSION[2])
if VERSION[3:] == ('alpha', 0):
version = '%s pre-alpha' % version
else:
if VERSION[3] != 'final':
version = '%s %s %s' % (version, VERSION[3], VERSION[4])
from django.utils.version import get_svn_revision
svn_rev = get_svn_revision()
if svn_rev != u'SVN-unknown':
version = "%s %s" % (version, svn_rev)
return version
I've looked at this file in emacs and I can't see the problem, and I've tried searching google with no luck. Can someone please point me in the right direction?
(For those interested in what I'm doing, please see http://www.allbuttonspressed.com/projects/djangoappengine.)
running this code in buffer with ipython shebang
! /usr/bin/env ipython
-- coding: utf-8 --
extending the end like this:
print version
return version
get_version()
get "1.3" at the ipython prompt
similar thing with python shell fails like this
from django.utils.version import get_svn_revision
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named utils.version
assume some install/path issue
Just an FYI, the latest version of django-appengine is found here:
https://github.com/django-nonrel
However, the allbuttonspressed version should be working. You may not have your python environment set properly.

Resources