I'm trying to deploy a golang app to app engine. Now I'm able to do it via the gcloud CLI on my mac, and this works fine (running gcloud app deploy app.yaml). However, I'm getting the following error on Bitbucket Pipelines:
+ gcloud --quiet --verbosity=error app deploy app.yaml --promote
You are about to deploy the following services:
- some-project/default/20171128t070345 (from [/go/src/bitbucket.org/acme/some-app/app.yaml])
Deploying to URL: [https://project-url.appspot.com]
Beginning deployment of service [default]...
ERROR: (gcloud.app.deploy) Staging command [/tmp/google-cloud-sdk/platform/google_appengine/goroot/bin/go-app-stager /go/src/bitbucket.org/acme/some-app/app.yaml /tmp/tmpLbUCA5] failed with return code [1].
------------------------------------ STDOUT ------------------------------------
------------------------------------ STDERR ------------------------------------
2017/11/28 07:03:45 failed analyzing /go/src/bitbucket.org/acme/some-app: cannot find package "github.com/gorilla/context" in any of:
($GOROOT not set)
/go/src/github.com/gorilla/context (from $GOPATH)
GOPATH: /go
--------------------------------------------------------------------------------
Here's my bitbucket-pipelines.yaml content:
image: golang:onbuild
pipelines:
branches:
develop:
- step:
script: # Modify the commands below to build your repository.
# Downloading the Google Cloud SDK
- curl -o /tmp/google-cloud-sdk.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-155.0.0-linux-x86_64.tar.gz
- tar -xvf /tmp/google-cloud-sdk.tar.gz -C /tmp/
- /tmp/google-cloud-sdk/install.sh -q
- source /tmp/google-cloud-sdk/path.bash.inc
- PACKAGE_PATH="${GOPATH}/src/bitbucket.org/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}"
- mkdir -pv "${PACKAGE_PATH}"
- tar -cO --exclude-vcs --exclude=bitbucket-pipelines.yml . | tar -xv -C "${PACKAGE_PATH}"
- cd "${PACKAGE_PATH}"
- go get -v
- go get -u github.com/golang/dep/cmd/dep
- go build -v
- go install
- go test -v
- echo $GOOGLE_CLIENT_SECRET | base64 --decode --ignore-garbage > ./gcloud-api-key.json
- gcloud auth activate-service-account --key-file gcloud-api-key.json
- gcloud components install app-engine-go
#- GOROOT="/tmp/go"
# Linking to the Google Cloud project
- gcloud config set project $CLOUDSDK_CORE_PROJECT
# Deploying the application
- gcloud --quiet --verbosity=error app deploy app.yaml --promote
- echo $GCLOUD_API_KEYFILE | base64 --decode --ignore-garbage > ./gcloud-api-key.json
#- gcloud auth activate-service-account --key-file gcloud-api-key.json
And, though it shouldn't be an issue since deploying to the cloud works fine, my app.yaml file as well:
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
nobuild_files:
- vendor
skip_files:
- |
^(.*/)?(
(#.*#)|
(.*\.mapping)|
(.*\.po)|
(.*\.pot)|
(.*\.py[co])|
(.*\.sw?)|
(.*\.yaml)|
(.*_test\.go)|
(.*~)|
(LICENSE)|
(Makefile.*)|
(\..*)|
(vendor/.*)|
)$
I'm fairly certain my issue is with how my bitbucket yaml file or the docker image I'm starting with, but I'm stuck. Any thoughts?
Is github.com/gorilla/context only used within your test files?
go get, will not by default get test dependencies.
You can exclusively add go get github.com/gorilla/context to your pipeline script.
Related
I have buildspec.yml file. I have to get env value from AWS secret manager and add it during codebuild. Later image will be hosted in App runner.
`
version: 0.2
env:
secrets-manager:
REACT_APP_NAME: "AWS_SECRET:AWS_SECRET_VALUE_KEY"
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
- REPOSITORY_URI=ecr_image_url
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- IMAGE_TAG=build-$(echo $CODEBUILD_BUILD_ID | awk -F":" '{print $2}')
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
- echo Writing image definitions file...
- printf '[{"name":"nodeapp","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
- cat imagedefinitions.json
artifacts:
files: imagedefinitions.json
`
i have followed AWS documentation to add env value from secret manager and given necesarry permisons. Application hosted is working fine without any error except ENV getting loaded.
I've switched deployment config (app.yaml) from standard environment to flex with Dockerfile.
runtime: custom
env: flex
service: my-backend-service
When gitlab CI executes app deploy app.yaml, I see a successful build and deployment log. New service and version are being created in the App engine dashboard, but there're no logs/files in Debug mode and instance is not responding. Debugger gives an error "The debugger could not find a debug target for the application".
Final part of CI-CD log.
You can stream logs from the command line by running:
$ gcloud app logs tail -s my-backend-service
To view your application in the web browser run:
$ gcloud app browse -s my-backend-service --project=some-project
Job succeeded
Dockerfile (works well with Cloud run deployment)
FROM python:3.10-slim-bullseye
ENV APP_HOME /app
ENV PYTHONPATH=${PYTHONPATH}:${PWD}
RUN apt-get update \
&& apt-get -y install libpq-dev gcc \
&& pip install psycopg2
RUN mkdir /app
COPY pyproject.toml /app
WORKDIR $APP_HOME
# set env variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
COPY . ./
RUN pip3 install poetry
RUN poetry config virtualenvs.create false
RUN poetry install --no-dev
CMD exec gunicorn main:app -c gunicorn_config.py
There were no such issues with standard app.yaml environment (python). What I'm missing?
I am trying to set up bitbucket-pipelines.yml file to do the build and then deploy react project. There is my code below.
image: node:10.15.1
pipelines:
default: # Pipelines that are triggered manually via the Bitbucket GUI
- step:
name: Build
script:
- yarn
- yarn build
- step:
name: Deploy
script:
- apt-get update
- apt-get install ncftp
- ncftpput -v -u "$FTP_USERNAME" -p "$FTP_PASSWORD" -R $FTP_HOST $FTP_SITE_ROOT_DEV build/*
- echo Finished uploading /build files to $FTP_HOST$FTP_SITE_ROOT
I am getting the result:
+ ncftpput -v -u "$FTP_USERNAME" -p "$FTP_PASSWORD" -R $FTP_HOST $FTP_SITE_ROOT_DEV build/*
could not stat build/*: No such file or directory.
ncftpput build/*: no valid files were specified.
It says that there is no build file or directory. but yarn build is actually build folder creates: react-scripts build
From Atlassian documentation
Key concepts
A pipeline is made up of a set of steps.
Each step in your pipeline runs a separate Docker container. If you
want, you can use different types of container for each step, by
selecting different images
So, when you try to send it in Deploy Step it's not there because you built it in another container.
To pass files between steps you have to use Artifacts
image: node:10.15.1
pipelines:
default: # Pipelines that are triggered manually via the Bitbucket GUI
- step:
name: Build
script:
- yarn
- yarn build
artifacts: # defining build/ as an artifact
- build/**
- step:
name: Deploy
script:
- apt-get update
- apt-get install ncftp
- ncftpput -v -u "$FTP_USERNAME" -p "$FTP_PASSWORD" -R $FTP_HOST $FTP_SITE_ROOT_DEV build/*
- echo Finished uploading /build files to $FTP_HOST$FTP_SITE_ROOT
Is it possible? Here is my app.yaml:
runtime: nodejs8
env_variables:
NODE_ENV: production
PORT: 8080
API_KEY: ${API_KEY}
${API_KEY} is like a placeholder.
When I run API_KEY=xdfj212c gcloud app deploy app.yaml command, I want to pass API_KEY=xdfj212c to app.yaml and replace the placeholder with
xdfj212c.
Expect result:
runtime: nodejs8
env_variables:
NODE_ENV: production
PORT: 8080
API_KEY: xdfj212c
Or, After I run
export API_KEY=xdfj212c
gcloud app deploy
I want the same behavior.
Is this make sense for google app engine deployment workflow?
In app.yaml you can include another YAML config
includes:
- extra_env_vars.yaml
that you can create on the fly while inserting the environment variables
# Unix-like OS
export DB_PASSWORD=your_password
export DB_HOST=your_host
echo -e "env_variables:\n DB_PASSWORD: $DB_PASSWORD\n DB_HOST: $DB_HOST" > extra_env_vars.yaml
# Windows
set DB_PASSWORD=your_password
set DB_HOST=your_host
(echo env_variables: & echo. DB_PASSWORD: %DB_PASSWORD% & echo. DB_HOST: %DB_HOST%) > extra_env_vars.yaml
The resulting extra_env_vars.yaml looks like this:
env_variables:
DB_PASSWORD: your_password
DB_HOST: your_host
Finally, ignore extra_env_vars.yaml in your version control system.
You could always use sed:
$ sed -i 's/${API_KEY}/xdfj212c/g' app.yaml && gcloud app deploy
The 'bad' thing is that this stores the key back, but you can always append a new sed command to replace the key again with the placeholder, or use your VCS mechanism to just reset the change the file.
Another option is saving your app.yaml file as something like app_template.yaml and do this for your deployments:
$ sed 's/${API_KEY}/xdfj212c/g' app_template.yaml | tee app.yaml; gcloud app deploy
This will do the replacement in a new file, app.yaml, and then do the deployment.
When I try to update my appengine project, it gives me an error.
$ sudo gcloud app deploy
ERROR: (gcloud.app.deploy) Staging command [/usr/local/google-cloud-sdk/platform/google_appengine/goroot-1.6/bin/go-app-stager /Users/baz/app/app.yaml /tmp/tmpSZLK/tmpIQ4] failed with return code [1].
------------------------------------ STDOUT ------------------------------------
------------------------------------ STDERR ------------------------------------
2017/07/08 16:22:06 failed analyzing /Users/baz/app: cannot find package "bitbucket.org/foo/bar" in any of:
($GOROOT not set)
($GOPATH not set)
GOPATH:
$ echo $GOPATH
/Users/baz/go
$ go env GOPATH
/Users/baz/go
Clearly GOPATH is set (I export it in my .bash_profile).
TL;DR - Do not use sudo for deploying the app.
Longer version
You're running the gcloud app deploy command using sudo. sudo by the nature of the command will not retain the current set of environment variables when running the command as root user.
Try deploying your app engine app without sudo:
gcloud app deploy
If you do need to run you command as sudo (which is a smell that your setup is broken somehow), you can look at this post. Although, I would recommend looking into the possibility of running your command without sudo.