Running GAE GCS on PyCharm under OS X, runtime error "No module named cloudstorage" - google-app-engine

I am trying to add Google Cloud Storage functionality to a Python GAE app that is already running with significant functionality. I work entirely within PyCharm on my development computer, which is a Mac running OS X 10.9.5.
I have created a new Python module that contains this statement:
import cloudstorage as gcs
as shown in the sample code at https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/getstarted
When I first added that line, PyCharm said "No module named cloudstorage" in the editor.
I then followed both the "pip" and the "svn" instructions at https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/download
to download the GCS Client Library.
In trying to follow those instructions, taking into account my prior experience with this programming environment, I actually tried using "pip" three times:
Once without the "-t" option, since I've never needed that option with "pip" before
Once using the "-t" option to specify my application directory's "lib" subdirectory
Once using:
pip install GoogleAppEngineCloudStorageClient -t /Applications/GoogleAppEngineLauncher.app//Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib, since I wasn't sure what the instructions meant by "<your_app_directory/lib>"
As mentioned, I also executed the "svn" command. Then, as mentioned in install python google cloud storage client on Ubuntu 14.04, I ran "cd gcs-client/src" and "sudo python setup.py install". I ran these commands in my user root directory.
After each of those successful but different installations of the GCS Client Library, I looked at the PyCharm editor window for my module, and it always had the same "No module named cloudstorage" error. But as an experiment, I would also try restarting PyCharm, and also try running my app.
At some point, the editor window stopped showing the error. It was not immediately after one of those steps above, but after I would go away to read various webpages and then come back to look at the error again. I don't know which of the installations was the one that got rid of the error message in the PyCharm editor.
In any case, whenever I try to run the app (again, inside PyCharm), I always get the runtime error "ImportError: No module named cloudstorage" on the same import statement.
The Run/Debug Configuration page for this app has both "Add content roots to PYTHONPATH" and "Add source roots to PYTHONPATH" checked.
Of course the main help I want is how to get past the "No module named cloudstorage" runtime error, even though the import statement no longer shows an error.
I think I also have as many as three spurious versions of the GCS Client Library. I'm much more concerned with getting past "Module not found", since it's a show-stopper, but if you have any idea how I can delete the spurious versions so that they're not just lying around, I'd be most grateful for that help as well.

If the "cloudstorage" directory is at <app>/lib/cloudstorage, then the import statement has to specify "lib":
import lib.cloudstorage
In my case, it's:
import lib.cloudstorage as gcs
By the way, the <app>/lib/GoogleAppEngineCloudStorageClient-1.9.5.0-py2.7.egg-info directory does not seem to be needed and can be deleted.

Actually, you also need to
touch __init__.py
in the lib directory. This will make the cloudstorage module visible to the "import lib.cloudstorage" command.
Dear Google: The distributions should include this file (or the procedure should account for it), and the demo script should be changed to reflect the expected usage. But more importantly why are you distributing/PROLIFERATING library code like this??!!! Why is this not distributed via gcloud? How am I ever going to pick up a patch for this library?

The accepted answer's solution
import lib.module_name
definitely can solve the problem. But I don't like add lib in front of every single module and happened to see how Google suggest import third party libs like this.
appengine will automatically run a file called appengine_config.py. So you can create such a file and put
from google.appengine.ext import vendor
vendor.add('lib')
inside that file. This will help you tell appengine to find dependencies in that folder, so you can simply write
import cloudstorage as gcs

I solve the missing module issue by adding the following to my main application file (main.py):
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), "lib"))
I think this is the way Guido intended. Now my code simply says import cloudstorage as gcs. None of the lib.cloudstorage or lib/__init__.py business.
From
https://stackoverflow.com/a/37645984/1740008

Related

Using go delve with google app engine standard & dev_appserver.py

In the google release notes it says:
November 15, 2017
Go runtime notes
Updated Go SDK to 1.9.61
Add --go_debugging flag to dev_appserver.py to enable Delve debugging.
I'm using dev_appserver.py to fire up several services simultaneously, so that they all share the same datastore emulator, and it works great.
However, when I add that --go_debugging flag, I get lots of errors which I don't understand. If anyone has insight, I'd be grateful.
I get these kinds of errors for each service it tries to build:
can't load package: package -N: unknown import path "-N": cannot find module providing package -N
can't load package: package -l: unknown import path "-l": cannot find module providing package -l
Is there maybe a problem with launching several services at once?
Thanks!
Update Nov 2020:
With all the new changes to app engine being able to use standard go libraries at google APIs, now I can just run my go app locally without using dev_appserver, and let it attach to my local datastore emulator etc. Because of that, and thanks to vscode, delve debugging works great (with the go extension). So now I can step through my app engine standard go code.

`cannot find package "appengine" error` when using VS code

By following this tutorial I created a go lang project and opened it by Visual Studio Code.
The code itself works fine I can run the server, but somehow VS Code shows
cannot find package "appengine" in any of:
/usr/local/Cellar/go/1.8.3/libexec/src/appengine (from $GOROOT)
/Users/ironsand/go/src/appengine (from $GOPATH)
I thought I must set GOROOT for the Google App Engine, but according to this stackoverflow question I shouldn't.
How to make VS Code recognize google app engine library properly?
More info
The appengine package exists in ~/dev/google-cloud-sdk/platform/google_appengine/goroot-1.8/‌​src/appengine
I'm using macOS Sierra 10.12.6.
I did use that tutorial and it's working on my Mac via terminal and via VS Code.
You may follow this tutorial to be able use VS code to deploy GAE app with python:
Deploy GAE app with VS code
• Dont forget to update path of dev_appserver.py in tasks.json.
If that didn't work out, you would need to re-install/init google-cloud-sdk.
Note that it's necessary to provide root privileges during installing/initializing in order to allow installer script to add paths properly.
Last thing: avoid tilde expansion ~ in path. Use absolute path in config.( in general absolute path must be used always unless there is a reason to use relative address with tilde )
Good luck,'.

Golang Appengine Project won't build

I have an appengine project using golang that I haven't touched in about a year. I cannot get it to build now on the machine where it was building before. I get the following error
go-app-builder: Failed parsing input: parser: bad import "syscall" in src/golang.org/x/net/internal/nettest/rlimit_unix.go
I am running on OS X if that helps at all. I am not a Golang expert, by any stretch, and my attempts to find anything even related to this have all turned up nothing. Any thoughts?
That usage is in net/internal. You are probably importing net in your app engine project someplace, and thus doing the illegal import.
If you constrain your usage to net/http on an app engine project, you should be just fine.
I think the problem is this:
You are not permitted to import the syscall package in a Go App Engine
app for security reasons. Whatever file is importing it needs to be
changed to not do that, or be excluded from the build.
as per this
So you might want to try this workaround. If you are importing the 'os' module. It fixes this
It would be helpful if you can post your code too.

PyDev PDFMiner GAE: ImportError: No module named pdfminer.converter

I work on GAE project in PyDev
I'd like to use the PDFminer library in order to convert a pdf file to a text file.
My problem is when i run the application it dosn't work and it displays this error message :
ImportError: No module named pdfminer.converter
I tested the same code in a normale python project and it works fine.
I used the same code in python console and it works too
I add the pdfminer folder to the python interpreter, i removed the interpreter and i add it again but i have always the same error.
Really i don't what i have to do, can anybody help me please ?
The problem comes from the fact that the PDFMiner is a third party library i copied the PDFMiner's files into the project and i works fine GAE don't import lib if isn't a pure python even if it exits in PYTHONPATH I found the solution in this post: Google App Engine "no module named requests" PyDev

No module named Crypto.Cipher on local mac AppEngine

No module named Crypto.Cipher
when I try to import
from Crypto.Cipher import AES
My folder structure looks like this:
test/
test/main.py
test/pycrypto-2.3
I ran the build and install inside of the pycrypto folder but I keep getting the error, any ideas?
It does not work with the usual easy_install, or running setup.py, depending on your Mac OS X version. This is due to how Google App Engine only allows for libraries loaded from certain directories. See here for more discussion: http://code.google.com/p/googleappengine/issues/detail?id=1627
You don't need to include pycrypto in your app; you need to install it the standard way for a Python library. If you've done that, the most likely reason it's not working is because you installed it (Eg, using easy_install) for a different version or installation of Python than the one that you're running the dev_appserver with. Macs are particularly notorious for this. Make sure you installed it in the same version of Python as the one the dev_appserver is running.
Please set PYTHONHOME if you have installed pycrypto in a user-defined directory. Also
/home/user# python -v
>>> from Crypto.Cipher import AES
will give you some idea what's the exact error.

Resources