I'm trying to deploy an API on AppEngine but I'm not sure why it's not working, here's an extract of the code:
package main
import (
"encoding/json"
"io/ioutil"
"net/http"
"fmt"
"google.golang.org/appengine"
)
func main () {
InitPubSub()
http.HandleFunc("/", HandlerBase)
http.HandleFunc("/user", HandlerUser)
http.HandleFunc("/event", HandlerEvent)
appengine.Main()
}
I'm getting the following error when I try to deploy the app (gcloud app deploy): main.go:9: can't find import: "google.golang.org/appengine"
I've followed the installation guide: https://cloud.google.com/appengine/docs/standard/go/download
And downloaded the appengine package: go get google.golang.org/appengine
Thanks!
Quoted from the Google App Engine forum:
"Developers are aware of this issue and are actively contributing towards a solid solution to it. In the meanwhile, you should be able to use "goapp deploy" instead of the gcloud command."
I had this problem too when I was following: https://cloud.google.com/appengine/docs/standard/go/tools/using-local-server
What worked for me was replacing
import "google.golang.org/appengine"
with:
import "appengine"
I'm not sure if the instruction is outdated.
To run a go appengine project, I found this project structure for a single service more than enough.
your/project/path/src/mainapp.go
your/project/path/src/app.yaml
or
your/project/path/src/mainfolder/mainapp.go
your/project/path/src/mainfolder/app.yaml
your/project/path/src/lib/lib1/watever.go
your/project/path/src/lib/lib2/whatever2.go
your/project/path/src/google.golang.org/ - for the golang appengine package
your/project/path/src/golang.org/ - golang packages
your/project/path/src/github.org/ - github packages
The subfolders/package folders, must not be above the main app.yaml file. Furthermore, the golang appengine package must be placed right inside "src/" and can only be imported directly from the "google.golang.org" import path and below that, not above. i.e. "customFolder/google.golang.org" is not allowed, "appengine", is however, allowed, provided that the app.yaml file is directly under src folder : src/app.yaml - src/mainapp.go.
Related
I am trying to add Login functionality to a React app I'm building using Amplify and AWS Cognito, but when I add the following line in my Login.js file:
Import {Auth} from "aws-amplify-react";
and try to compile, I get the following error:
Failed to compile.
./node_modules/aws-amplify-react/lib-esm/Analytics/trackLifecycle.js
Module not found: Can't resolve '#aws-amplify/analytics' in 'my_react_path/node_modules/aws-amplify-react/lib-esm/Analytics'
I tried installing Amplify with homebrew and npm, but I can't get around this, any tips?
EDIT:
Initially I had:
Import {Auth} from "aws-amplify";
But The compiler couldn't resolve aws-amplify, and I could only find aws-amplify-react in the node_modules folder so I changed it.
However, in "node_modules/aws-amplify-react/lib-esm/Analytics/trackLifecycle.js" in line 39:
import Analytics from '#aws-amplify/analytics';
The problem is that it can't find aws-amplify... Did I make a mess while installing maybe?
Assuming that you are not worried about bundle size at this stage, probably your easiest course of action is to remove your node_modules entirely, then:
npm i aws-amplify
import like this:
import Amplify, { Auth } from 'aws-amplify'
You only need to install/import aws-amplify-react if you plan on using the OOTB UI components. If you are making your own UI components for login you don't need that package at this stage.
I get a similar error
Failed to compile.
./node_modules/aws-amplify-react/lib-esm/Analytics/trackLifecycle.js
Module not found: Can't resolve '#aws-amplify/analytics' in '/Users/myuser/myproject/node_modules/aws-amplify-react/lib-esm/Analytics'
when following the Amplify tutorial.
The workaround I found is to execute:
npm i -S #aws-amplify/analytics #aws-amplify/interactions #aws-amplify/storage #aws-amplify/ui #aws-amplify/xr aws-amplify
Obviously, I would love to know how to avoid all these extra libraries.
I would revert to your previous version before attempting to add amplify (old package.json & node_modules)
Follow this walkthrough: https://docs.amplify.aws/start/getting-started/installation/q/integration/react-native
When you get to the "Add Authentication" step, skip it and go straight to "Next Steps" -> "Authentication" and follow until you get to:
import Amplify, {Auth} from 'aws-amplify';
import config from './aws-exports';
Amplify.configure(config);
I've been updating my entire go gae standard project to use go 1.11's modules.
Main directory structure
app.yaml
app.go
go.mod
go.sum
app.go
package main
import "bitbucket.org/myPrivateRepo"
func main() {
myImportantModule.Run()
}
go.mod
module myProject
require bitbucket.org/myPrivateRepo v0.0.1
The Error
If I try to gcloud app deploy:
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build <GUI>
status: FAILURE.
Build error details: go: bitbucket.org/myPrivateRepo#v0.0.1:
https://api.bitbucket.org/2.0/repositories/myPrivateRepo?fields=scm:
403 Forbidden
(Note: obviously the repo I'm using has a real name).
So can I do it this way? I'll admit to not fully understanding the migration documentation, particularly when it talked about "Moving files to your GOPATH". https://cloud.google.com/appengine/docs/standard/go111/go-differences
I mean, I thought one of the benefits of the new module system is that you don't need everything under the go path. When I read https://github.com/golang/go/wiki/Modules for example, it very early on says "Create a directory outside of your GOPATH:"
So, to be clear, right now all of my code is outside the go path, but everything builds locally just fine.
I think it all works becausego automatically downloads and caches things within the go path when I run go mod tidy / go build etc.
Yet it fails when I try to gcloud app deploy. How would the google cloud build system ever have access to my private repositories anyway? I'm obviously missing something important. I also read you are not supposed to combine vendoring with the new module system so that can't be it.
I will be very happy if this works, as using DEP forced me to use goapp deploy very awkwardly.
Thanks!
UPDATE: Google has some better documentation now that go 1.14 is out: https://cloud.google.com/appengine/docs/standard/go/specifying-dependencies
My solution:
Instead of dealing with credentials, I'm using go's module replace functionality to point GAE to use my local code. This is working well.
Directory structure:
myService/
src/
service.go // has a run() function to set up routers etc.
go.mod // depends on my private module in bitbucket and other things
… // other source files
build/
gae/
src/ // simlink to ../../src
modules/ // git ignored, I clone or copy my modules in build scripts.
app.go // see below…
go.mod // has main() which calls service.run() and appEngine.Main()
app.yaml
Method
I use git module replace so that GAE uses my local code. Before building I parse myService/src/go.mod to find the correct version of my private module, then I clone it into the modules folder. I also made an option to copy wip module source code for debugging locally without committing to my module repositories.
go.mod from gae directory:
module myServiceGAE
require (
bitbucket.org/me/myService v0.0.0
google.golang.org/appengine v1.4.0
)
replace bitbucket.org/me/myService => ./src
replace bitbucket.org/me/myModule => ./modules/utils
Pros
The package under myService has no references or knowledge of GAE, so I can easily build it into a docker etc. I think parsing the service go.mod files would be like creating my own dependency manager, defeating the benefits of go modules.
Cons
If I had a private module which depended on another private module, I think things would get too complicated.
Another alternative is to also use Google Cloud Secret Manager
https://cloud.google.com/cloud-build/docs/access-private-github-repos
Google Cloud will have an SSH key to access and pull your private repository.
Set git credentials before deploying:
git config credential.helper '!f() { sleep 1; echo "username=${GIT_USER}\npassword=${GIT_PASSWORD}"; }; f'
export GIT_USER=put_git_user_here
export GIT_PASSWORD=put_git_password_here
gcloud app deploy
i am using google cloud storage and google cloud datastore api (locally, for now), in a Standard Environment app.
I am having a very weird behaviour trying to running my tests (with pytest): i have discovered that, during tests, the dev_appserver.fix_sys_path command run by pytest-beds that i am using (https://github.com/kaste/pytest-beds/blob/master/testbeds/plugin.py#L35) puts the bundled google-cloud libraries before my custom versions:
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/antlr3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib
/Users/username/google-cloud-sdk/platform/google_appengine/lib/ipaddr
/Users/username/google-cloud-sdk/platform/google_appengine/lib/yaml-3.10
/Users/username/google-cloud-sdk/platform/google_appengine/lib/rsa
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules
/Users/username/google-cloud-sdk/platform/google_appengine/lib/httplib2
/Users/username/google-cloud-sdk/platform/google_appengine/lib/oauth2client_devserver
/Users/username/google-cloud-sdk/platform/google_appengine/lib/six-1.9.0
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/simplejson
/Users/username/google-cloud-sdk/platform/google_appengine/lib/django-1.4
/Users/username/google-cloud-sdk/platform/google_appengine/lib/endpoints-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/jinja2-2.6
/Users/username/google-cloud-sdk/platform/google_appengine/lib/protorpc-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/PyAMF-0.6.1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/markupsafe-0.15
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webob-1.2.3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2
/Users/username/dev/project
/Users/username/.virtualenvs/project/bin
/Users/username/dev/project/app
/Users/username/.virtualenvs/project/lib/python27.zip
/Users/username/.virtualenvs/project/lib/python2.7
/Users/username/.virtualenvs/project/lib/python2.7/plat-darwin
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/Extras/lib/python
/Users/username/.virtualenvs/project/lib/python2.7/lib-tk
/Users/username/.virtualenvs/project/lib/python2.7/lib-old
/Users/username/.virtualenvs/project/lib/python2.7/lib-dynload
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/lib/python2.7/site-packages
This causes the import of google.auth.crypt.base module import to fail, because it does not exist in the package version shipped with google-cloud-sdk, while i can see that version 1.3.0 on github, that is apparently the same version of the one shipped with the sdk, does.
Same for pyasn1_modules package: version 0.2.1 existing, shipped 0.11
The import errors I am having are:
from google.auth.crypt import base
ImportError: cannot import name base
and:
from pyasn1_modules import pem
ImportError: cannot import name pem
Last, but not least, the environment: MACOSX.
Google Cloud SDK 184.0.0
app-engine-python 1.9.65
app-engine-python-extras 1.9.63
beta 2017.09.15
bq 2.0.28
cloud-datastore-emulator 1.3.0
core 2018.01.05
gcloud
gsutil 4.28
UPDATE: apparently, i was using the google.cloud.storage while instead i should be using, according to the examples for standard environment (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/appengine-client/main.py#L23), the package cloudstorage. I will change the code, try and update the post here.
UPDATE 2: i have created a specific github repo that should replicate the problem, just in case. Here it is:
https://github.com/brunoripa/gae_import_error
If the problem is entirely on localhost during testing, you can run:
pip install cryptography
so that the Google libraries won't even need pem to begin with.
We typically add libs to remote API with vendor system.
from google.appengine.ext import vendor
vendor.add('lib')
Module pyasn1_modules gets installed as a dependency when you install google-cloud and I am not sure of the reason behind it but, in remote API, I have observed it misses pem.py. The simple workaround would be:
# within remote API console
>>> import sys
>>> sys.modules.pop('pyasn1_modules')
<module 'pyasn1_modules' from '/Users/pjamkhande001/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules/pyasn1_modules/__init__.pyc'>
About your other issue of not being able to import from google.auth.crypt import base, you are required to gcloud auth login.
gcloud auth application-default login restores your default authentication credentials but if you want to login to the application as user, you have to gcloud auth login. Hope this helps.
I am trying to use Go with App Engine SDK for Go and when I run goapp serve, I get the following error in printed the browser after going to http://localhost:8080 (I get similar errors in the terminal):
The Go application could not be built.
(Executed command: C:\go_appengine\goroot\bin\go-app-builder.exe
-app_base C:\Projects\Go\Davilex -arch 6 -dynamic -goroot C:\go_appengine\goroot -nobuild_files ^^$ -unsafe -gopath
C:\Projects\Go -print_extras_hash controllers\form\form.go main.go
controllers\pages\pages.go models\form.go)
2016/01/20 22:10:42 go-app-builder: Failed parsing input: parser: bad
import "syscall" in colorable_windows.go from GOPATH
Seems there is a somewhat similar question here a few days ago, with no resolve: bad import "syscall" for cloud storage APIs
At first I thought it was because I had both Python 2.7.x and 3.5.x installed and the the docs specifically say it will only work with v 2.7.x, so I uninstalled all 3.5.x references, removed the SDK and reinstalled it/unzipped/added its path to my PATH.
I can run gcloud, goapp and go all fine so everything should be set up correctly as far as I can tell, and just doing regular go run main.go runs fine and works, but I can't seem to be able to use goapp serve and I have no clue where to go about the error above.
EDIT/SOLUTION
I created a dummy project and just added this simple set of code into it:
package main
import "net/http"
func init() {
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello world!"))
}
And the error seems to have gone away and everything works fine. So the problem must be occuring from either my code or my use of https://github.com/gin-gonic/gin package.
Hope that may help anyone with a similar problem. Try creating another project and see if it works. If it does, you now know where the issue is coming from.
I created a dummy project and just added this simple set of code into it:
package main
import "net/http"
func init() {
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello world!"))
}
And the error seems to have gone away and everything works fine. So the problem must be occuring from either my code or my use of https://github.com/gin-gonic/gin package.
Hope that may help anyone with a similar problem. Try creating another project and see if it works. If it does, you now know where the issue is coming from.
So I have spent the better part of two days trying to figure this one out and no matter what I do I can't get things straightened out. Here is what is going on:
Using Go and Appengine. I am running into issues when trying to
get proper unit tests working.
I have tried lots of structures but here is a sample of where I am now: https://github.com/markhayden/SampleIssue
I am running into dependency issues in either goapp serve or goapp test -v ./src/lib1 depending on how I have my import paths set.
If I use "src/lib1" for my import path and then goapp serve. My app boots and runs fine, but when I run tests I get the following failure:
src/lib1/lib1.go:5:2: cannot find package "src/lib2" in any of:
/Users/USERNAME/go_appengine/goroot/src/pkg/src/lib2 (from $GOROOT)
/Users/markhayden/Projects/go/src/src/lib2 (from $GOPATH)
Likewise, if I use "dummy/src/lib1" as my path, my tests are happy and run fine but upon goapp serve ing the app I now get:
2014/11/06 20:33:34 go-app-builder: Failed parsing input: app file lib1.go conflicts with same file imported from GOPATH
Have fiddled with all sorts of different options and can't figure out how to handle dependencies and still have solid testing. Maybe its a appengine / golang bug? Or am I missing something?
Any help would be very much appreciated. Thanks in advance!
Updated everything based on first comment feedback. I can run tests (as I was able to do before) but I still can not serve the app. Here is what I get when running goapp serve
INFO 2014-11-07 17:24:48,727 devappserver2.py:745] Skipping SDK update check.
INFO 2014-11-07 17:24:48,748 api_server.py:172] Starting API server at: http://localhost:60732
INFO 2014-11-07 17:24:48,751 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO 2014-11-07 17:24:48,754 admin_server.py:118] Starting admin server at: http://localhost:8000
ERROR 2014-11-07 17:24:49,041 go_runtime.py:171] Failed to build Go application: (Executed command: /Users/markhayden/go_appengine/goroot/bin/go-app-builder -app_base /Users/markhayden/Projects/go/src/github.com/markhayden/SampleIssue -arch 6 -dynamic -goroot /Users/markhayden/go_appengine/goroot -nobuild_files ^^$ -unsafe -gopath /Users/markhayden/Projects/go -print_extras_hash lib1/lib1.go lib2/lib2_test.go main_test.go main.go lib1/lib1_test.go lib2/lib2.go)
2014/11/07 09:24:49 go-app-builder: Failed parsing input: app file lib2.go conflicts with same file imported from GOPATH
$GOPATH = /Users/markhayden/Projects/go
$GOROOT = not set (according to docs it doesnt need to be if you dont use a custom directory)
App Structure:
$GOPATH/src/github.com/markhayden/SampleIssue/
- app.yaml
- /lib1
- lib1_test.go
- lib1.go
- /lib2
- lib2_test.go
- lib2.go
- main_test.go
- main.go
In main.go:
import (
"fmt"
"github.com/markhayden/SampleIssue/lib1"
"net/http"
)
In lib1/lib1.go:
import (
"fmt"
"github.com/markhayden/SampleIssue/lib2"
)
Appengine "conflicts with same file imported from GOPATH" issue:
Appengine is importing things underneath the root directory (i.e. where the app.yaml is). This will cause two imports, one by appengine when it scans the directories, and a second by your source when it is explicitly imported.
You have two choices:
Don't use the full import path (for sub-folder packages) with appengine.
Remove the source repository part of import. So instead of
"github.com/blah/blah" it would be "blah/blah".
Note: This kinda sucks as it makes your build and software appengine specific. You could make this a little better -maybe- by using build constraints. e.g. +build !appengine or
+build !appengine to include/remove certain files from the build depending on if you are targeting appengine.
Move your modules/dependencies (sub-folders) to a separate and independent project to make it work with the full path import convention:
Get rid of all directories / dependencies in the main project (where
your app.yaml is), so that appengine can't scan and find them.
Move them to another independent project (I did SampleIssueDeps)
with no app.yaml that is not a sub-directory (e.g.
/MarkHayden/SampleIssueDeps).
Then pull those dependencies via
full path import. e.g. github.com/MarkHayden/SampleIssueDeps/lib1.
Summary: For sub-folder packages in an appengine project don't include the "source repository" part of the import path OR only use appengine to init() and move all of your other code to separate projects and use like external dependencies.
I came up with another option that isn't discussed here and in my opinion is much easier to deal with (and keep your app less appengine specific). Lets say you have the repo at github.com/blah/blah and right now the root folder of the repo defines your app engine server.
First, move the app.yaml and other app engine specific files (NOT .go files) into github.com/blah/blah/appengine/app.yaml.
Next, wherever you run your init function for app engine, rename it to something like func Run() { ... }, and then in github.com/blah/blah/whatever.go write something like this:
package appengine
import "github.com/blah/blah"
func init() {
blah.Run()
}
From my experience this has resolved the issue and made things much easier. I'll update this if I run into any major issues that make this a bad solution.
I had a lot of trouble following various answers and understanding how to solve the problem.
But after a lot of research, I believe I understand both cause and solution:
Google app-builder tooling does some path-munging and is causing this.
They are aware of the bug but no ETA to fix it.
Problem summary:
any .go files inside or below the directory holding main.go / app.yaml will be double imported…
In summary, just make sure that ALL of our files/packages are siblings and not decedents of the directory holding those two files...