I'm having issues getting the full text search to work in the GAE production environment. It works fine on my dev machine, but when I deploy it to production, I'm getting this error:
com.google.apphosting.api.ApiProxy$CallNotFoundException: The API package 'search' or call 'IndexDocument()' was not found.
This looks to be a runtime issue where it can't find the correct class, but I have the appengine-api-1.0-sdk-1.7.0.jar included in WEB-INF/lib so I was assuming that when deploying to prod, it would be using that version of GAE. Am I missing something on how GAE deploys libs to prod? GAE SDK 1.7 should be available in production now right?
These are my imports:
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.search.Document;
import com.google.appengine.api.search.Field;
import com.google.appengine.api.search.GeoPoint;
import com.google.appengine.api.search.Index;
import com.google.appengine.api.search.IndexSpec;
import com.google.appengine.api.search.SearchServiceFactory;
//my instantiation
private static final Index INDEX = SearchServiceFactory.getSearchService()
.getIndex(IndexSpec.newBuilder().setName("shared_index"));
It does appear to be using version 1.7.0. Here is my logging of the version that I'm using and the stack trace of the error right after it:
E 2012-08-07 19:30:37.352
version: Google App Engine/1.7.0
W 2012-08-07 19:30:37.702
com.google.apphosting.api.ApiProxy$CallNotFoundException: The API package 'search' or call 'IndexDocument()' was not found.
at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:782)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:824)
at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:809)
at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893)
at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798)
at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1059)
at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:907)
at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2255)
at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2062)
at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143)
at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3117)
at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599)
at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417)
at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260)
at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121)
at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578)
at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002)
at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884)
at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136)
at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855)
I am not sure having appengine-api-1.0-sdk-1.7.0.jar included in WEB-INF/lib will deploy it as that version.
If you have using eclipse, have you selected a 1.7 sdk from preferences-->google-->appengine
Also, how are you deploying? If you are using a command line, make sure you are using the /1.7sdk/bin/appcfg file to do the uploading. If you are using eclipse, then set the preferences as above.
NEVERMIND THE ABOVE...
I think you need to build your IndexSpec
private static final Index INDEX = SearchServiceFactory.getSearchService()
.getIndex(IndexSpec.newBuilder().setName("shared_index").build());
Related
i am using google cloud storage and google cloud datastore api (locally, for now), in a Standard Environment app.
I am having a very weird behaviour trying to running my tests (with pytest): i have discovered that, during tests, the dev_appserver.fix_sys_path command run by pytest-beds that i am using (https://github.com/kaste/pytest-beds/blob/master/testbeds/plugin.py#L35) puts the bundled google-cloud libraries before my custom versions:
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/antlr3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib
/Users/username/google-cloud-sdk/platform/google_appengine/lib/ipaddr
/Users/username/google-cloud-sdk/platform/google_appengine/lib/yaml-3.10
/Users/username/google-cloud-sdk/platform/google_appengine/lib/rsa
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules
/Users/username/google-cloud-sdk/platform/google_appengine/lib/httplib2
/Users/username/google-cloud-sdk/platform/google_appengine/lib/oauth2client_devserver
/Users/username/google-cloud-sdk/platform/google_appengine/lib/six-1.9.0
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/simplejson
/Users/username/google-cloud-sdk/platform/google_appengine/lib/django-1.4
/Users/username/google-cloud-sdk/platform/google_appengine/lib/endpoints-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/jinja2-2.6
/Users/username/google-cloud-sdk/platform/google_appengine/lib/protorpc-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/PyAMF-0.6.1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/markupsafe-0.15
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webob-1.2.3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2
/Users/username/dev/project
/Users/username/.virtualenvs/project/bin
/Users/username/dev/project/app
/Users/username/.virtualenvs/project/lib/python27.zip
/Users/username/.virtualenvs/project/lib/python2.7
/Users/username/.virtualenvs/project/lib/python2.7/plat-darwin
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/Extras/lib/python
/Users/username/.virtualenvs/project/lib/python2.7/lib-tk
/Users/username/.virtualenvs/project/lib/python2.7/lib-old
/Users/username/.virtualenvs/project/lib/python2.7/lib-dynload
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/lib/python2.7/site-packages
This causes the import of google.auth.crypt.base module import to fail, because it does not exist in the package version shipped with google-cloud-sdk, while i can see that version 1.3.0 on github, that is apparently the same version of the one shipped with the sdk, does.
Same for pyasn1_modules package: version 0.2.1 existing, shipped 0.11
The import errors I am having are:
from google.auth.crypt import base
ImportError: cannot import name base
and:
from pyasn1_modules import pem
ImportError: cannot import name pem
Last, but not least, the environment: MACOSX.
Google Cloud SDK 184.0.0
app-engine-python 1.9.65
app-engine-python-extras 1.9.63
beta 2017.09.15
bq 2.0.28
cloud-datastore-emulator 1.3.0
core 2018.01.05
gcloud
gsutil 4.28
UPDATE: apparently, i was using the google.cloud.storage while instead i should be using, according to the examples for standard environment (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/appengine-client/main.py#L23), the package cloudstorage. I will change the code, try and update the post here.
UPDATE 2: i have created a specific github repo that should replicate the problem, just in case. Here it is:
https://github.com/brunoripa/gae_import_error
If the problem is entirely on localhost during testing, you can run:
pip install cryptography
so that the Google libraries won't even need pem to begin with.
We typically add libs to remote API with vendor system.
from google.appengine.ext import vendor
vendor.add('lib')
Module pyasn1_modules gets installed as a dependency when you install google-cloud and I am not sure of the reason behind it but, in remote API, I have observed it misses pem.py. The simple workaround would be:
# within remote API console
>>> import sys
>>> sys.modules.pop('pyasn1_modules')
<module 'pyasn1_modules' from '/Users/pjamkhande001/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules/pyasn1_modules/__init__.pyc'>
About your other issue of not being able to import from google.auth.crypt import base, you are required to gcloud auth login.
gcloud auth application-default login restores your default authentication credentials but if you want to login to the application as user, you have to gcloud auth login. Hope this helps.
So I have spent the better part of two days trying to figure this one out and no matter what I do I can't get things straightened out. Here is what is going on:
Using Go and Appengine. I am running into issues when trying to
get proper unit tests working.
I have tried lots of structures but here is a sample of where I am now: https://github.com/markhayden/SampleIssue
I am running into dependency issues in either goapp serve or goapp test -v ./src/lib1 depending on how I have my import paths set.
If I use "src/lib1" for my import path and then goapp serve. My app boots and runs fine, but when I run tests I get the following failure:
src/lib1/lib1.go:5:2: cannot find package "src/lib2" in any of:
/Users/USERNAME/go_appengine/goroot/src/pkg/src/lib2 (from $GOROOT)
/Users/markhayden/Projects/go/src/src/lib2 (from $GOPATH)
Likewise, if I use "dummy/src/lib1" as my path, my tests are happy and run fine but upon goapp serve ing the app I now get:
2014/11/06 20:33:34 go-app-builder: Failed parsing input: app file lib1.go conflicts with same file imported from GOPATH
Have fiddled with all sorts of different options and can't figure out how to handle dependencies and still have solid testing. Maybe its a appengine / golang bug? Or am I missing something?
Any help would be very much appreciated. Thanks in advance!
Updated everything based on first comment feedback. I can run tests (as I was able to do before) but I still can not serve the app. Here is what I get when running goapp serve
INFO 2014-11-07 17:24:48,727 devappserver2.py:745] Skipping SDK update check.
INFO 2014-11-07 17:24:48,748 api_server.py:172] Starting API server at: http://localhost:60732
INFO 2014-11-07 17:24:48,751 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO 2014-11-07 17:24:48,754 admin_server.py:118] Starting admin server at: http://localhost:8000
ERROR 2014-11-07 17:24:49,041 go_runtime.py:171] Failed to build Go application: (Executed command: /Users/markhayden/go_appengine/goroot/bin/go-app-builder -app_base /Users/markhayden/Projects/go/src/github.com/markhayden/SampleIssue -arch 6 -dynamic -goroot /Users/markhayden/go_appengine/goroot -nobuild_files ^^$ -unsafe -gopath /Users/markhayden/Projects/go -print_extras_hash lib1/lib1.go lib2/lib2_test.go main_test.go main.go lib1/lib1_test.go lib2/lib2.go)
2014/11/07 09:24:49 go-app-builder: Failed parsing input: app file lib2.go conflicts with same file imported from GOPATH
$GOPATH = /Users/markhayden/Projects/go
$GOROOT = not set (according to docs it doesnt need to be if you dont use a custom directory)
App Structure:
$GOPATH/src/github.com/markhayden/SampleIssue/
- app.yaml
- /lib1
- lib1_test.go
- lib1.go
- /lib2
- lib2_test.go
- lib2.go
- main_test.go
- main.go
In main.go:
import (
"fmt"
"github.com/markhayden/SampleIssue/lib1"
"net/http"
)
In lib1/lib1.go:
import (
"fmt"
"github.com/markhayden/SampleIssue/lib2"
)
Appengine "conflicts with same file imported from GOPATH" issue:
Appengine is importing things underneath the root directory (i.e. where the app.yaml is). This will cause two imports, one by appengine when it scans the directories, and a second by your source when it is explicitly imported.
You have two choices:
Don't use the full import path (for sub-folder packages) with appengine.
Remove the source repository part of import. So instead of
"github.com/blah/blah" it would be "blah/blah".
Note: This kinda sucks as it makes your build and software appengine specific. You could make this a little better -maybe- by using build constraints. e.g. +build !appengine or
+build !appengine to include/remove certain files from the build depending on if you are targeting appengine.
Move your modules/dependencies (sub-folders) to a separate and independent project to make it work with the full path import convention:
Get rid of all directories / dependencies in the main project (where
your app.yaml is), so that appengine can't scan and find them.
Move them to another independent project (I did SampleIssueDeps)
with no app.yaml that is not a sub-directory (e.g.
/MarkHayden/SampleIssueDeps).
Then pull those dependencies via
full path import. e.g. github.com/MarkHayden/SampleIssueDeps/lib1.
Summary: For sub-folder packages in an appengine project don't include the "source repository" part of the import path OR only use appengine to init() and move all of your other code to separate projects and use like external dependencies.
I came up with another option that isn't discussed here and in my opinion is much easier to deal with (and keep your app less appengine specific). Lets say you have the repo at github.com/blah/blah and right now the root folder of the repo defines your app engine server.
First, move the app.yaml and other app engine specific files (NOT .go files) into github.com/blah/blah/appengine/app.yaml.
Next, wherever you run your init function for app engine, rename it to something like func Run() { ... }, and then in github.com/blah/blah/whatever.go write something like this:
package appengine
import "github.com/blah/blah"
func init() {
blah.Run()
}
From my experience this has resolved the issue and made things much easier. I'll update this if I run into any major issues that make this a bad solution.
I had a lot of trouble following various answers and understanding how to solve the problem.
But after a lot of research, I believe I understand both cause and solution:
Google app-builder tooling does some path-munging and is causing this.
They are aware of the bug but no ETA to fix it.
Problem summary:
any .go files inside or below the directory holding main.go / app.yaml will be double imported…
In summary, just make sure that ALL of our files/packages are siblings and not decedents of the directory holding those two files...
Ok I have to rewrite my question after further investigation.
I run into below problem in my GWT/GAE project :
java.lang.RuntimeException: Class com.google.appengine.tools.development.agent.runtime.Runtime$21 can not access a member of class com.ibm.icu.text.CollatorServiceShim with modifiers ""
at com.ibm.icu.text.Collator.getShim(Collator.java:456)
at com.ibm.icu.text.Collator.getInstance(Collator.java:478)
at com.google.visualization.datasource.datatable.value.TextValue$1.<init>(TextValue.java:126)
at com.google.visualization.datasource.datatable.value.TextValue.getTextLocalizedComparator(TextValue.java:125)
at com.google.visualization.datasource.datatable.value.Value$1.<init>(Value.java:141)
at com.google.visualization.datasource.datatable.value.Value.getLocalizedComparator(Value.java:140)
at com.google.visualization.datasource.query.engine.TableRowComparator.<init>(TableRowComparator.java:66)
at com.google.visualization.datasource.query.engine.QueryEngine.performSort(QueryEngine.java:234)
at com.google.visualization.datasource.query.engine.QueryEngine.executeQuery(QueryEngine.java:128)
at com.google.visualization.datasource.DataSourceHelper.applyQuery(DataSourceHelper.java:410)
at com.klawt.server.resources.chart.InvoicesChartServerResource.retrieve(InvoicesChartServerResource.java:129)
Some more investigation revealed that gwt-dev.jar contains a copy of the icu library :
public abstract class com.ibm.icu.text.Collator implements
java.util.Comparator, com.ibm.icu.util.Freezable
is part of gwt-dev.jar
My project has a dependency on icu4j 4.0.1 (for the Google Visualization Datasource library). But GAE SDK insists on using the version included in gwt-dev.jar;
I guess there must a way to setup my development environment so this kinda works, it did before.
I have tried moving GWT SDK to the bottom in the 'Order and export' tab of my build path and the icu jar to the top, but no luck.
Switching from OpenJDK to Oracle JDK fixes it !! It is not the first time I ran into issues with OpenJDK !
I'm using PyCharm 2.6.3 and Python 2.7.5 for my Google Application Engine project.
Today occurred strange error when I was trying to debug my GAE project:
ImportError: No module named pydevd_vm_type
Module exists but doesn't want to be imported somewhere in debug tools.
When I trying to type "import pydevd_vm_type", autocomplete tool gives me right path to that module. How can I solve this issue?
Problem solved!
File Path_on_disk\PyCharm 2.6.3\helpers\pydev\pydevd.py
Error occurs in class
class PyDBCommandThread(PyDBDaemonThread)
in method
def OnRun(self)
...
import pydevd_vm_type
if pydevd_vm_type.GetVmType() == pydevd_vm_type.PydevdVmType.JYTHON and sys.hexversion <= 0x020201f0:
# a lot of comments about Jython bug
run_traced = False
and next in
import pydevd_tracing
if run_traced:
pydevd_tracing.SetTrace(None) # no debugging on this thread
I've deleted those two imports. PROFIT!!!
I am still new to Python and GAE. I have an application on local server that is running just fine. I can add entity to my datastore, I can view my website, etc: everything is fine.
Now I am trying to use bulkloader to add entities to my datastore. I followed the tutorial at https://developers.google.com/appengine/docs/python/tools/uploadingdata. My loader is below:
from google.appengine.ext import ndb
from google.appengine.tools import bulkloader
import my_model
class ArticleLoader(bulkloader.Loader):
def __init__(self):
bulkloader.Loader.__init__(self, 'Article',
[('title', str),
('author', str)
])
loaders = [ArticleLoader]
I am getting the error:
No module named my_model
Does anyone have a fix for this?
Note: I am only using one directory. So my loader is in the same location as the other file that imports the my_model module.
This can also happen if your PYTHONPATH is not properly set up. If you're on Linux, try running this before you run the Bulkloader:
export PYTHONPATH=$PYTHONPATH:.
This appends your current directory to your PYTHONPATH and should make your my_model module visible. Since my memory is terrible and I always forget to do it, I've ended up using a simple shell script that includes this at the beginning and then the bulkload command itself.
If you're on Windows, you should be able to modify your path by using sys.path.append. Haven't tested this, but you could try adding this to your script (note that this should work on Linux as well):
import sys
# ...
sys.path.append('.')
Your code should be located in a file named my_model.py. You are getting that error because there is no module named my_module. Might be worth a read of the Python module and package docs.