How to authenticate a private Go Module using go 1.11 and Google App Engine Standard - google-app-engine

Question

I've been updating my entire go gae standard project to use go 1.11's modules.
Main directory structure
app.yaml
app.go
go.mod
go.sum
app.go
package main
import "bitbucket.org/myPrivateRepo"
func main() {
myImportantModule.Run()
}
go.mod
module myProject
require bitbucket.org/myPrivateRepo v0.0.1
The Error
If I try to gcloud app deploy:
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build <GUI>
status: FAILURE.
Build error details: go: bitbucket.org/myPrivateRepo#v0.0.1:
https://api.bitbucket.org/2.0/repositories/myPrivateRepo?fields=scm:
403 Forbidden
(Note: obviously the repo I'm using has a real name).
So can I do it this way? I'll admit to not fully understanding the migration documentation, particularly when it talked about "Moving files to your GOPATH". https://cloud.google.com/appengine/docs/standard/go111/go-differences
I mean, I thought one of the benefits of the new module system is that you don't need everything under the go path. When I read https://github.com/golang/go/wiki/Modules for example, it very early on says "Create a directory outside of your GOPATH:"
So, to be clear, right now all of my code is outside the go path, but everything builds locally just fine.
I think it all works becausego automatically downloads and caches things within the go path when I run go mod tidy / go build etc.
Yet it fails when I try to gcloud app deploy. How would the google cloud build system ever have access to my private repositories anyway? I'm obviously missing something important. I also read you are not supposed to combine vendoring with the new module system so that can't be it.
I will be very happy if this works, as using DEP forced me to use goapp deploy very awkwardly.
Thanks!

Answer 1

UPDATE: Google has some better documentation now that go 1.14 is out: https://cloud.google.com/appengine/docs/standard/go/specifying-dependencies
My solution:
Instead of dealing with credentials, I'm using go's module replace functionality to point GAE to use my local code. This is working well.
Directory structure:
myService/
src/
service.go // has a run() function to set up routers etc.
go.mod // depends on my private module in bitbucket and other things
… // other source files
build/
gae/
src/ // simlink to ../../src
modules/ // git ignored, I clone or copy my modules in build scripts.
app.go // see below…
go.mod // has main() which calls service.run() and appEngine.Main()
app.yaml
Method
I use git module replace so that GAE uses my local code. Before building I parse myService/src/go.mod to find the correct version of my private module, then I clone it into the modules folder. I also made an option to copy wip module source code for debugging locally without committing to my module repositories.
go.mod from gae directory:
module myServiceGAE
require (
bitbucket.org/me/myService v0.0.0
google.golang.org/appengine v1.4.0
)
replace bitbucket.org/me/myService => ./src
replace bitbucket.org/me/myModule => ./modules/utils
Pros
The package under myService has no references or knowledge of GAE, so I can easily build it into a docker etc. I think parsing the service go.mod files would be like creating my own dependency manager, defeating the benefits of go modules.
Cons
If I had a private module which depended on another private module, I think things would get too complicated.

Answer 2

Another alternative is to also use Google Cloud Secret Manager
https://cloud.google.com/cloud-build/docs/access-private-github-repos
Google Cloud will have an SSH key to access and pull your private repository.

Answer 3

Set git credentials before deploying:
git config credential.helper '!f() { sleep 1; echo "username=${GIT_USER}\npassword=${GIT_PASSWORD}"; }; f'
export GIT_USER=put_git_user_here
export GIT_PASSWORD=put_git_password_here
gcloud app deploy