Below is my app.yaml
runtime: python39
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
env_variables:
SEC: %sec%
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
This is my cloudbuild.yaml for app engine trying to pass a secrete vale to app.yaml as env veriable
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args: ['-c', "export _VAL=$(echo $$SEC)
&& echo $$SEC;echo $_VAL && sed -i 's/%sec%/'$$SEC'/g' app.yaml
&& gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy"
]
secretEnv: ["SEC"]
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/db_secret/versions/3
env: 'SEC'
timeout: '1600s'
I've done this in the past by just using ">>" to append the env vars to the bottom of my app.yaml. With this method, the env_variables section of your app.yaml needs to be last.
I don't use this method anymore though, the secrets show up in your cloudbuild log. I just import the secret manager package to grab my secrets inside the application these days.
cloudbuild.yaml
steps:
- name: "gcr.io/cloud-builders/gcloud"
secretEnv: ['SECRET_ONE','SECRET_TWO']
entrypoint: 'bash'
args:
- -c
- |
echo $'\n SECRET_ONE: '$$SECRET_ONE >> app.yaml
echo $'\n SECRET_TWO: '$$SECRET_TWO >> app.yaml
gcloud -q app deploy
availableSecrets:
secretManager:
- versionName: projects/012345678901/secrets/SECRET_ONE
env: 'SECRET_ONE'
- versionName: projects/012345678901/secrets/SECRET_TWO
env: 'SECRET_TWO'
app.yaml
runtime: go116
main: cmd
service: serviceone
env_variables:
PROJECT_ID: project-a0a00
PORT: 8080
You are in the right way, you have made a quote mistake because you try to run a inline command.
Prefer the | structure, more readable and therefore easy to debug and to write. Each line is executed sequentially, similar to the && in your inline command
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
- '-c'
- |
export _VAL=$(echo $$SEC)
echo $$SEC
echo $_VAL
sed -i "s/%sec%/$$SEC/g" app.yaml
gcloud config set app/cloud_build_timeout 1600
gcloud app deploy
secretEnv: ["SEC"]
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/db_secret/versions/3
env: 'SEC'
timeout: '1600s'
EDIT 1:
If your password contain special characters that break the SED expression, you will have an error. You can use an alternative solution to replace the whole line, for example
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
- '-c'
- |
export _VAL=$(echo $$SEC)
echo $$SEC
echo $_VAL
cat app.yaml | awk "{ if (NR == $(grep -n '%sec%' app.yaml | cut -d : -f 1)) print \" SEC: $$SEC\"; else print }" > app.yaml
gcloud config set app/cloud_build_timeout 1600
gcloud app deploy
secretEnv: ["SEC"]
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/db_secret/versions/3
env: 'SEC'
timeout: '1600s'
Have a try on it
Related
I am deploying FastAPI app using PyTorch as requirements. The build fails most of the time and succeed when I remove PyTorch from requirements.txt.
Here is my configuration:
app.yaml
env: standard
runtime: python310
instance_class: F2
entrypoint: gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app
automatic_scaling:
target_cpu_utilization: 0.65
min_instances: 5
max_instances: 100
min_pending_latency: 30ms
max_pending_latency: automatic
max_concurrent_requests: 50
cloudbuild.yaml
# [START cloudbuild]
steps:
# Install dependencies
- name: 'python'
entrypoint: pip3
args: ['install', '-r', 'requirements.txt']
# Run custom commands
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args: ['-c', 'gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy']
timeout: '6000s'
# [END cloudbuild]
requirements.txt
torch
fastapi
uvicorn[standard]
gunicorn
sqlalchemy
google-api-python-client
google-auth-httplib2
google-auth-oauthlib
pydantic
transformers
cloud-sql-python-connector
pg8000
What should be changed to the configuration to make the deployment successful?
I am trying to add Cloud Build on top of my App Engine Flask app. Everything works, but for some reason, I can't access the substitution variables I declared in the trigger.
Env vars are still being fetched from app.yaml. And they are parsed literally, not as variables. When I remove it from app.yaml Python throws a NoneType error.
[Trigger][1]: https://i.stack.imgur.com/Ii6Jv.png
[App.yaml][2]: https://i.stack.imgur.com/bg646.png
runtime: python310
instance_class: F4
automatic_scaling:
max_instances: 8
env_variables:
_CONFIG_TYPE: ${_CONFIG_TYPE}
[cloudbuild][3] https://i.stack.imgur.com/jo0PN.png
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args: ['-c', 'gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy']
timeout: '1600s'
substitutions:
_CONFIG_TYPE: ${_CONFIG_TYPE}
It won't work because gcloud app deploy command start a new Cloud Build, behind the scene, to build a container with your code and to deploy it. Your env var will change anything
The solution is to perform a bash replacement, with sed for instance.
app.yaml file
runtime: python310
instance_class: F4
automatic_scaling:
max_instances: 8
env_variables:
_CONFIG_TYPE: ##_CONFIG_TYPE##
Cloud Build Step with env var usage
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
env-vars:
- CONFIG_TYPE: ${_CONFIG_TYPE}
args:
- '-c'
- |
sed -i "s/##_CONFIG_TYPE##/$${CONFIG_TYPE}/g" app.yaml
gcloud config set app/cloud_build_timeout 1600
gcloud app deploy
timeout: '1600s'
substitutions:
_CONFIG_TYPE: ${_CONFIG_TYPE}
Cloud Build Step without env var usage
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
- '-c'
- |
sed -i "s/##_CONFIG_TYPE##/${_CONFIG_TYPE}/g" app.yaml
gcloud config set app/cloud_build_timeout 1600
gcloud app deploy
timeout: '1600s'
substitutions:
_CONFIG_TYPE: ${_CONFIG_TYPE}
I have a secret token on my App Engine app.yaml
env_variables:
TOKEN: super-secret-token
And obviously this token is out of git. Using Google Cloud Build, how can I set this parameter TOKEN at build time or before?
You can use Secret Manager within Cloud Build to get the actual secret and replace the super-secret-token placeholder value in app.yaml prior to deploying your app to App Engine. That would look something like this:
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args: [ '-c', "gcloud secrets versions access latest --secret=secret-name --format='get(payload.data)' | tr '_-' '/+' | base64 -d > decrypted-data.txt" ]
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: /bin/sh
args:
- '-c'
- |
sed "s/super-secret-token/g" $(cat decrypted-data.txt)
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args: ['-c', 'gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy']
timeout: '1600s'
Having said that, your secret token will still be available unencrypted in your App Engine's environment variable which is not optimal security-wise. Instead you may want to query Secret Manager from within your App Engine code directly. You'll find code samples to do so here.
I try to make a full process of integrated deployment with Gitlab and Google AppEngine.
But the deployment fail because it seems I miss rights. I really don't know which right is needed here.
I activated the App Engine API and the Cloud Build API.
The rights my service has:
my gitlab-ci.yml file:
Staging Deployment:
stage: Deploy
environment:
name: Staging
before_script:
- echo "deb http://packages.cloud.google.com/apt cloud-sdk-jessie main" | tee /etc/apt/sources.list.d/google-cloud-sdk.list
- curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
- apt-get update
- apt-get -qq -y install google-cloud-sdk
script:
- echo $DEPLOY_KEY_FILE_PRODUCTION > /tmp/$CI_PIPELINE_ID.json
- gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
- gcloud config list
- gcloud config set project $PROJECT_ID_PRODUCTION
- gcloud --quiet --project $PROJECT_ID_PRODUCTION app deploy gae/test.yml
only:
refs:
- master
And finally my test.yml:
# [START django_app]
runtime: python37
service: testing
entrypoint: gunicorn -b :$PORT manta.wsgi --timeout 120
default_expiration: "5m"
env_variables:
DJANGO_SETTINGS_MODULE: manta.settings.dev
handlers:
# This configures Google App Engine to serve the files in the app's
# static directory.
- url: /static
static_dir: static/
# This handler routes all requests not caught above to the main app.
# It is required when static routes are defined, but can be omitted
# (along with the entire handlers section) when there are no static
# files defined.
- url: /.*
script: auto
# [END django_app]
And the error I have during deployment:
ERROR: (gcloud.app.deploy) Permissions error fetching application [apps/testing]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
I'm trying to dockerize a MERN stack but when the time comes when react has to start, the container exit with status 0.
This is the log error:
This is the structure of my project:
- project
- server
- api
api.yml
server.js
Dockerfile
- www
app.yml
Dockerfile
docker-compose.yml
The www folder contains the starter files the will be generate from npx create-react-app www.
The content of server/Dockerfile is:
FROM node:latest
RUN mkdir -p /usr/server
WORKDIR /usr/server
RUN npm install -g nodemon
EXPOSE 3000
CMD [ "npm", "start" ]
the content of www/Dockerfile is:
FROM node:latest
RUN mkdir -p /usr/www/src/app
WORKDIR /usr/www/src/app
EXPOSE 3000
CMD [ "npm", "start" ]
and, in the end the content of docker-compose.yml is:
version: '3.7'
services:
mongodb:
image: mongo
ports:
- 27017:27017
api:
build: ./server/
ports:
- "6200:6200"
volumes:
- ./server:/usr/server
depends_on:
- mongodb
www:
build: ./www/
ports:
- 3000:3000
volumes:
- ./www:/usr/www/src/app
depends_on:
- api
Now, in the title I mention Google Cloud, because I tried to distribute the different "server" and "www" parts in my production environment. The distribution of server works correctly butwww fails with this error:
The errors that are generated by Docker and Google Cloud seem very similar or am I wrong? Could it be a reaction problem or am I wrong in both cases?
I also leave the contents of the app.yaml andapi.yaml files.
app.yaml
runtime: nodejs
env: flex
# Only for developing
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
handlers:
- url: /.*
static_files: build/index.html
upload: build/index.html
- url: /
static_dir: build
The content of the api.yaml file is the same as that of app.yaml but without the handlers section.