i am using google cloud storage and google cloud datastore api (locally, for now), in a Standard Environment app.
I am having a very weird behaviour trying to running my tests (with pytest): i have discovered that, during tests, the dev_appserver.fix_sys_path command run by pytest-beds that i am using (https://github.com/kaste/pytest-beds/blob/master/testbeds/plugin.py#L35) puts the bundled google-cloud libraries before my custom versions:
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/antlr3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib
/Users/username/google-cloud-sdk/platform/google_appengine/lib/ipaddr
/Users/username/google-cloud-sdk/platform/google_appengine/lib/yaml-3.10
/Users/username/google-cloud-sdk/platform/google_appengine/lib/rsa
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules
/Users/username/google-cloud-sdk/platform/google_appengine/lib/httplib2
/Users/username/google-cloud-sdk/platform/google_appengine/lib/oauth2client_devserver
/Users/username/google-cloud-sdk/platform/google_appengine/lib/six-1.9.0
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/simplejson
/Users/username/google-cloud-sdk/platform/google_appengine/lib/django-1.4
/Users/username/google-cloud-sdk/platform/google_appengine/lib/endpoints-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/jinja2-2.6
/Users/username/google-cloud-sdk/platform/google_appengine/lib/protorpc-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/PyAMF-0.6.1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/markupsafe-0.15
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webob-1.2.3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2
/Users/username/dev/project
/Users/username/.virtualenvs/project/bin
/Users/username/dev/project/app
/Users/username/.virtualenvs/project/lib/python27.zip
/Users/username/.virtualenvs/project/lib/python2.7
/Users/username/.virtualenvs/project/lib/python2.7/plat-darwin
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/Extras/lib/python
/Users/username/.virtualenvs/project/lib/python2.7/lib-tk
/Users/username/.virtualenvs/project/lib/python2.7/lib-old
/Users/username/.virtualenvs/project/lib/python2.7/lib-dynload
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/lib/python2.7/site-packages
This causes the import of google.auth.crypt.base module import to fail, because it does not exist in the package version shipped with google-cloud-sdk, while i can see that version 1.3.0 on github, that is apparently the same version of the one shipped with the sdk, does.
Same for pyasn1_modules package: version 0.2.1 existing, shipped 0.11
The import errors I am having are:
from google.auth.crypt import base
ImportError: cannot import name base
and:
from pyasn1_modules import pem
ImportError: cannot import name pem
Last, but not least, the environment: MACOSX.
Google Cloud SDK 184.0.0
app-engine-python 1.9.65
app-engine-python-extras 1.9.63
beta 2017.09.15
bq 2.0.28
cloud-datastore-emulator 1.3.0
core 2018.01.05
gcloud
gsutil 4.28
UPDATE: apparently, i was using the google.cloud.storage while instead i should be using, according to the examples for standard environment (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/appengine-client/main.py#L23), the package cloudstorage. I will change the code, try and update the post here.
UPDATE 2: i have created a specific github repo that should replicate the problem, just in case. Here it is:
https://github.com/brunoripa/gae_import_error
If the problem is entirely on localhost during testing, you can run:
pip install cryptography
so that the Google libraries won't even need pem to begin with.
We typically add libs to remote API with vendor system.
from google.appengine.ext import vendor
vendor.add('lib')
Module pyasn1_modules gets installed as a dependency when you install google-cloud and I am not sure of the reason behind it but, in remote API, I have observed it misses pem.py. The simple workaround would be:
# within remote API console
>>> import sys
>>> sys.modules.pop('pyasn1_modules')
<module 'pyasn1_modules' from '/Users/pjamkhande001/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules/pyasn1_modules/__init__.pyc'>
About your other issue of not being able to import from google.auth.crypt import base, you are required to gcloud auth login.
gcloud auth application-default login restores your default authentication credentials but if you want to login to the application as user, you have to gcloud auth login. Hope this helps.
Related
I've been updating my entire go gae standard project to use go 1.11's modules.
Main directory structure
app.yaml
app.go
go.mod
go.sum
app.go
package main
import "bitbucket.org/myPrivateRepo"
func main() {
myImportantModule.Run()
}
go.mod
module myProject
require bitbucket.org/myPrivateRepo v0.0.1
The Error
If I try to gcloud app deploy:
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build <GUI>
status: FAILURE.
Build error details: go: bitbucket.org/myPrivateRepo#v0.0.1:
https://api.bitbucket.org/2.0/repositories/myPrivateRepo?fields=scm:
403 Forbidden
(Note: obviously the repo I'm using has a real name).
So can I do it this way? I'll admit to not fully understanding the migration documentation, particularly when it talked about "Moving files to your GOPATH". https://cloud.google.com/appengine/docs/standard/go111/go-differences
I mean, I thought one of the benefits of the new module system is that you don't need everything under the go path. When I read https://github.com/golang/go/wiki/Modules for example, it very early on says "Create a directory outside of your GOPATH:"
So, to be clear, right now all of my code is outside the go path, but everything builds locally just fine.
I think it all works becausego automatically downloads and caches things within the go path when I run go mod tidy / go build etc.
Yet it fails when I try to gcloud app deploy. How would the google cloud build system ever have access to my private repositories anyway? I'm obviously missing something important. I also read you are not supposed to combine vendoring with the new module system so that can't be it.
I will be very happy if this works, as using DEP forced me to use goapp deploy very awkwardly.
Thanks!
UPDATE: Google has some better documentation now that go 1.14 is out: https://cloud.google.com/appengine/docs/standard/go/specifying-dependencies
My solution:
Instead of dealing with credentials, I'm using go's module replace functionality to point GAE to use my local code. This is working well.
Directory structure:
myService/
src/
service.go // has a run() function to set up routers etc.
go.mod // depends on my private module in bitbucket and other things
… // other source files
build/
gae/
src/ // simlink to ../../src
modules/ // git ignored, I clone or copy my modules in build scripts.
app.go // see below…
go.mod // has main() which calls service.run() and appEngine.Main()
app.yaml
Method
I use git module replace so that GAE uses my local code. Before building I parse myService/src/go.mod to find the correct version of my private module, then I clone it into the modules folder. I also made an option to copy wip module source code for debugging locally without committing to my module repositories.
go.mod from gae directory:
module myServiceGAE
require (
bitbucket.org/me/myService v0.0.0
google.golang.org/appengine v1.4.0
)
replace bitbucket.org/me/myService => ./src
replace bitbucket.org/me/myModule => ./modules/utils
Pros
The package under myService has no references or knowledge of GAE, so I can easily build it into a docker etc. I think parsing the service go.mod files would be like creating my own dependency manager, defeating the benefits of go modules.
Cons
If I had a private module which depended on another private module, I think things would get too complicated.
Another alternative is to also use Google Cloud Secret Manager
https://cloud.google.com/cloud-build/docs/access-private-github-repos
Google Cloud will have an SSH key to access and pull your private repository.
Set git credentials before deploying:
git config credential.helper '!f() { sleep 1; echo "username=${GIT_USER}\npassword=${GIT_PASSWORD}"; }; f'
export GIT_USER=put_git_user_here
export GIT_PASSWORD=put_git_password_here
gcloud app deploy
I am going to integrate salesforce library into liferay 7 mvc portlet, the following steps are what i did:
Add libraries to class path. In Eclipse, go to Project > Properties > Java Build Path > Libraries > Add External JARs, then add the sfdc-wsc JAR to this list
Add below line to build.gradle:
compile group: 'com.force.api', name: 'force-wsc', version: '40.1.1'
The java source code is ok until i use gradle build to build project, the following error occur:
error: package com.sforce.soap.enterprise does not exist import com.sforce.soap.enterprise.EnterpriseConnection;
error: package com.sforce.soap.enterprise does not exist import com.sforce.soap.enterprise.QueryResult;
error: package com.sforce.soap.enterprise does not exist import com.sforce.soap.enterprise.SaveResult;
I also set the bnd file as follow according to blog post of DAVID H NEBINGER: https://web.liferay.com/web/user.26526/blog/-/blogs/osgi-module-dependencies
But nothing is improved
Bundle-ClassPath:.,\lib/externalLib.jar
-includeresource:\
lib/externalLib.jar=externalLib.jar,\
lib/commons-lang.jar=commons-lang=[0-9]*.jar
Please give any suggestion to correct.
Thanks in advance
I recently developed such a solution, but I used a different approach. I've implemented an OSGi bundle that exports Salesforce's SOAP APIs. This way you can use Salesforce APIs in any other Liferay bundle.
On this Salesforce SOAP API Client OSGi Bundle repository you find the sources. The OSGi bundle is also available on Maven Central.
Once you install the Salesforce SOAP API Client OSGi bundle, you can use it in any other Liferay bundle, such as your MVC Portlet. This sample project Salesforce Liferay Gogo Shell Command Client implements a set of Gogo Shell commands that allow us to interact with the Salesforce CRM system.
In your particular case, if you want to include external libraries via Gradle, then you can declare your dependency through the key compileInclude.
dependencies {
compileOnly group: "org.osgi", name: "org.osgi.core", version: "6.0.0"
compileOnly group: "org.osgi", name: "org.osgi.service.component.annotations", version: "1.3.0"
compileOnly group: "com.liferay.portal", name: "com.liferay.portal.kernel", version: "2.6.0"
compileOnly group: "org.apache.felix", name: "org.apache.felix.gogo.runtime", version: '1.0.6'
compileInclude group: 'org.fusesource.jansi', name: 'jansi', version: '1.16'
compileInclude 'de.vandermeer:asciitable:0.3.2'}
This way you do not have to do anything on the bnd file. The external jar, like magic will be placed inside your bundle and MANIFEST will be correct.
If you want generate your stubs then you go at Force.com Web Service Connector (WSC)
I'm trying to deploy an API on AppEngine but I'm not sure why it's not working, here's an extract of the code:
package main
import (
"encoding/json"
"io/ioutil"
"net/http"
"fmt"
"google.golang.org/appengine"
)
func main () {
InitPubSub()
http.HandleFunc("/", HandlerBase)
http.HandleFunc("/user", HandlerUser)
http.HandleFunc("/event", HandlerEvent)
appengine.Main()
}
I'm getting the following error when I try to deploy the app (gcloud app deploy): main.go:9: can't find import: "google.golang.org/appengine"
I've followed the installation guide: https://cloud.google.com/appengine/docs/standard/go/download
And downloaded the appengine package: go get google.golang.org/appengine
Thanks!
Quoted from the Google App Engine forum:
"Developers are aware of this issue and are actively contributing towards a solid solution to it. In the meanwhile, you should be able to use "goapp deploy" instead of the gcloud command."
I had this problem too when I was following: https://cloud.google.com/appengine/docs/standard/go/tools/using-local-server
What worked for me was replacing
import "google.golang.org/appengine"
with:
import "appengine"
I'm not sure if the instruction is outdated.
To run a go appengine project, I found this project structure for a single service more than enough.
your/project/path/src/mainapp.go
your/project/path/src/app.yaml
or
your/project/path/src/mainfolder/mainapp.go
your/project/path/src/mainfolder/app.yaml
your/project/path/src/lib/lib1/watever.go
your/project/path/src/lib/lib2/whatever2.go
your/project/path/src/google.golang.org/ - for the golang appengine package
your/project/path/src/golang.org/ - golang packages
your/project/path/src/github.org/ - github packages
The subfolders/package folders, must not be above the main app.yaml file. Furthermore, the golang appengine package must be placed right inside "src/" and can only be imported directly from the "google.golang.org" import path and below that, not above. i.e. "customFolder/google.golang.org" is not allowed, "appengine", is however, allowed, provided that the app.yaml file is directly under src folder : src/app.yaml - src/mainapp.go.
So I have spent the better part of two days trying to figure this one out and no matter what I do I can't get things straightened out. Here is what is going on:
Using Go and Appengine. I am running into issues when trying to
get proper unit tests working.
I have tried lots of structures but here is a sample of where I am now: https://github.com/markhayden/SampleIssue
I am running into dependency issues in either goapp serve or goapp test -v ./src/lib1 depending on how I have my import paths set.
If I use "src/lib1" for my import path and then goapp serve. My app boots and runs fine, but when I run tests I get the following failure:
src/lib1/lib1.go:5:2: cannot find package "src/lib2" in any of:
/Users/USERNAME/go_appengine/goroot/src/pkg/src/lib2 (from $GOROOT)
/Users/markhayden/Projects/go/src/src/lib2 (from $GOPATH)
Likewise, if I use "dummy/src/lib1" as my path, my tests are happy and run fine but upon goapp serve ing the app I now get:
2014/11/06 20:33:34 go-app-builder: Failed parsing input: app file lib1.go conflicts with same file imported from GOPATH
Have fiddled with all sorts of different options and can't figure out how to handle dependencies and still have solid testing. Maybe its a appengine / golang bug? Or am I missing something?
Any help would be very much appreciated. Thanks in advance!
Updated everything based on first comment feedback. I can run tests (as I was able to do before) but I still can not serve the app. Here is what I get when running goapp serve
INFO 2014-11-07 17:24:48,727 devappserver2.py:745] Skipping SDK update check.
INFO 2014-11-07 17:24:48,748 api_server.py:172] Starting API server at: http://localhost:60732
INFO 2014-11-07 17:24:48,751 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO 2014-11-07 17:24:48,754 admin_server.py:118] Starting admin server at: http://localhost:8000
ERROR 2014-11-07 17:24:49,041 go_runtime.py:171] Failed to build Go application: (Executed command: /Users/markhayden/go_appengine/goroot/bin/go-app-builder -app_base /Users/markhayden/Projects/go/src/github.com/markhayden/SampleIssue -arch 6 -dynamic -goroot /Users/markhayden/go_appengine/goroot -nobuild_files ^^$ -unsafe -gopath /Users/markhayden/Projects/go -print_extras_hash lib1/lib1.go lib2/lib2_test.go main_test.go main.go lib1/lib1_test.go lib2/lib2.go)
2014/11/07 09:24:49 go-app-builder: Failed parsing input: app file lib2.go conflicts with same file imported from GOPATH
$GOPATH = /Users/markhayden/Projects/go
$GOROOT = not set (according to docs it doesnt need to be if you dont use a custom directory)
App Structure:
$GOPATH/src/github.com/markhayden/SampleIssue/
- app.yaml
- /lib1
- lib1_test.go
- lib1.go
- /lib2
- lib2_test.go
- lib2.go
- main_test.go
- main.go
In main.go:
import (
"fmt"
"github.com/markhayden/SampleIssue/lib1"
"net/http"
)
In lib1/lib1.go:
import (
"fmt"
"github.com/markhayden/SampleIssue/lib2"
)
Appengine "conflicts with same file imported from GOPATH" issue:
Appengine is importing things underneath the root directory (i.e. where the app.yaml is). This will cause two imports, one by appengine when it scans the directories, and a second by your source when it is explicitly imported.
You have two choices:
Don't use the full import path (for sub-folder packages) with appengine.
Remove the source repository part of import. So instead of
"github.com/blah/blah" it would be "blah/blah".
Note: This kinda sucks as it makes your build and software appengine specific. You could make this a little better -maybe- by using build constraints. e.g. +build !appengine or
+build !appengine to include/remove certain files from the build depending on if you are targeting appengine.
Move your modules/dependencies (sub-folders) to a separate and independent project to make it work with the full path import convention:
Get rid of all directories / dependencies in the main project (where
your app.yaml is), so that appengine can't scan and find them.
Move them to another independent project (I did SampleIssueDeps)
with no app.yaml that is not a sub-directory (e.g.
/MarkHayden/SampleIssueDeps).
Then pull those dependencies via
full path import. e.g. github.com/MarkHayden/SampleIssueDeps/lib1.
Summary: For sub-folder packages in an appengine project don't include the "source repository" part of the import path OR only use appengine to init() and move all of your other code to separate projects and use like external dependencies.
I came up with another option that isn't discussed here and in my opinion is much easier to deal with (and keep your app less appengine specific). Lets say you have the repo at github.com/blah/blah and right now the root folder of the repo defines your app engine server.
First, move the app.yaml and other app engine specific files (NOT .go files) into github.com/blah/blah/appengine/app.yaml.
Next, wherever you run your init function for app engine, rename it to something like func Run() { ... }, and then in github.com/blah/blah/whatever.go write something like this:
package appengine
import "github.com/blah/blah"
func init() {
blah.Run()
}
From my experience this has resolved the issue and made things much easier. I'll update this if I run into any major issues that make this a bad solution.
I had a lot of trouble following various answers and understanding how to solve the problem.
But after a lot of research, I believe I understand both cause and solution:
Google app-builder tooling does some path-munging and is causing this.
They are aware of the bug but no ETA to fix it.
Problem summary:
any .go files inside or below the directory holding main.go / app.yaml will be double imported…
In summary, just make sure that ALL of our files/packages are siblings and not decedents of the directory holding those two files...
I'm having issues getting the full text search to work in the GAE production environment. It works fine on my dev machine, but when I deploy it to production, I'm getting this error:
com.google.apphosting.api.ApiProxy$CallNotFoundException: The API package 'search' or call 'IndexDocument()' was not found.
This looks to be a runtime issue where it can't find the correct class, but I have the appengine-api-1.0-sdk-1.7.0.jar included in WEB-INF/lib so I was assuming that when deploying to prod, it would be using that version of GAE. Am I missing something on how GAE deploys libs to prod? GAE SDK 1.7 should be available in production now right?
These are my imports:
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.search.Document;
import com.google.appengine.api.search.Field;
import com.google.appengine.api.search.GeoPoint;
import com.google.appengine.api.search.Index;
import com.google.appengine.api.search.IndexSpec;
import com.google.appengine.api.search.SearchServiceFactory;
//my instantiation
private static final Index INDEX = SearchServiceFactory.getSearchService()
.getIndex(IndexSpec.newBuilder().setName("shared_index"));
It does appear to be using version 1.7.0. Here is my logging of the version that I'm using and the stack trace of the error right after it:
E 2012-08-07 19:30:37.352
version: Google App Engine/1.7.0
W 2012-08-07 19:30:37.702
com.google.apphosting.api.ApiProxy$CallNotFoundException: The API package 'search' or call 'IndexDocument()' was not found.
at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:782)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:824)
at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:809)
at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893)
at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798)
at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1059)
at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:907)
at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2255)
at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2062)
at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143)
at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3117)
at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599)
at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417)
at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260)
at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121)
at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578)
at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002)
at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884)
at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136)
at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855)
I am not sure having appengine-api-1.0-sdk-1.7.0.jar included in WEB-INF/lib will deploy it as that version.
If you have using eclipse, have you selected a 1.7 sdk from preferences-->google-->appengine
Also, how are you deploying? If you are using a command line, make sure you are using the /1.7sdk/bin/appcfg file to do the uploading. If you are using eclipse, then set the preferences as above.
NEVERMIND THE ABOVE...
I think you need to build your IndexSpec
private static final Index INDEX = SearchServiceFactory.getSearchService()
.getIndex(IndexSpec.newBuilder().setName("shared_index").build());