Google App Engine does not deploy two services (seem to be overriding each other) - google-app-engine

I have a Django backend and an Angular frontend and i'm trying to deploy both of them to a Google App Engine Standard app as two services within the same app.
The directory looks like this:
-backend
--cloudbuild.yaml
--app.yaml
-- ...
-frontend
--cloudbuild.yaml
--app.yaml
-- ...
-cloudbuild.yaml
The main cloudbuild.yaml goes through each folder and looks for a cloudbuild.yaml file and submits the build:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
for d in */; do
config="${d}cloudbuild.yaml"
if [[ ! -f "${config}" ]]; then
continue
fi
echo "Building $d ... "
(
gcloud builds submit $d --config=${config}
) &
done
wait
When i push the code to my Github repo, they get deployed successfully. However, both logs show something like below:
Step #2: Beginning deployment of service [default]...
Step #2: #============================================================#
Step #2: #= Uploading 0 files to Google Cloud Storage =#
Step #2: #============================================================#
Step #2: File upload done.
Step #2: Updating service [default]...
Step #2: ................................done.
Step #2: Setting traffic split for service [default]...
Step #2: .....done.
Step #2: Deployed service [default] to [https://thisapp.appspot.com]
So it seems like one is overriding the other as the default service. When I go to the services page, there is only one service which is the default one.
I know that we first have to deploy the default service and then the rest. However i thought since the main cloudbuild.yaml submits the build one by one, then there would be a default service and the second build would create a second service.
The other two cloudbuild.yaml files are as below:
steps:
# Install node packages
- name: 'gcr.io/cloud-builders/npm'
args: [ 'install' ]
# Build productive files
- name: 'gcr.io/cloud-builders/npm'
args: [ 'run', 'build', '--prod']
# Deploy to google cloud app egnine
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', '--version=prod']
steps:
- name: 'python:3.7'
entrypoint: python3
args: ['-m', 'pip', 'install', '-t', '.', '-r', 'requirements.txt']
- name: 'python:3.7'
entrypoint: python3
args: ['./manage.py', 'collectstatic', '--noinput']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', '--version=prod']
I'd be grateful if you can help me on this.
Thanks

It is happening because in your app.yaml file, you have not provided the service tag. If service tag is not provided, the service will be deployed as default service. For e.g. when you deploy your frontend, it is getting deployed as default service. And then you deploy your backend, it is also getting deployed as default service overriding the existing default service. or the vice versa.
Add the service tag in your frontend's app.yaml:
service: frontend
runtime: nodejs10
You may keep the backend as default service i.e. no service tag in backend's app.yaml

Related

deploying to app engine flex custom runtime with cloud build causes a large amount of builds to run

i am Kinda new to cloud build so i am kind of confused about what is happening.
first this is my file structure
cloudbuild.yaml
backend/
Dockerfile
app.yaml
I had an application which i dockerized and deployed to app engine felx in custom runtime
here's my Dockerfile
FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build
WORKDIR /app
# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish -c Release -o out
FROM mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim AS base
ENV ASPNETCORE_URLS=http://+:80;
WORKDIR /app
COPY --from=build /app/out .
EXPOSE 80
ENTRYPOINT ["dotnet", "myapp.dll"]
and this is my app engine flex file
runtime: custom
env: flex
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
service: backend
network:
name: my-network
subnetwork_name: my-network-subnet
instance_tag: "backend"
forwarded_ports:
I have successfully deployed this app on app engine flex using this command
gcloud app deploy --appyaml=app.yaml
Then i added a cloudbuild.yaml file following this google doc
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args: ['-c', 'gcloud config set app/cloud_build_timeout 2000 && gcloud app deploy --appyaml=backend/app.yaml']
as you can see in the cloudbuild.yaml i didnt add the timeout attribute because it gave me this error each time i tried to submit the build.
Error Response: [13] Error parsing cloudbuild.yaml for runtime custom: Argument is not an object: "2000s"
after removing the timeout attribute, cloud build started behaving in a weird way, it kept creating build jobs on its own until it reached over 20 builds.
i had to stop these builds manually because it exceeded the 120 minute free quota limit.
can some one tell me if my cloudbuild.yaml is the thing causing the issue or if its a problem with google cloud.
So the problem was writing the cloudbuild as a yaml file, instead i re-wrote it as a json file, i am not entirely sure why is the cloudbuild.yaml file was giving me errors but that was my solution.
{
"steps": [
{
"name": "gcr.io/google.com/cloudsdktool/cloud-sdk",
"entrypoint": "bash",
"args": [
"-c",
"gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy --appyaml=app.yaml"
]
}
],
"timeout": "1600s"
}
Also the cloudbuild and app.yaml must be in the root of the branch with the cloudbuild file and Dockerfile.

Google Cloud Build and App Engine enviroment variables

I have a secret token on my App Engine app.yaml
env_variables:
TOKEN: super-secret-token
And obviously this token is out of git. Using Google Cloud Build, how can I set this parameter TOKEN at build time or before?
You can use Secret Manager within Cloud Build to get the actual secret and replace the super-secret-token placeholder value in app.yaml prior to deploying your app to App Engine. That would look something like this:
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args: [ '-c', "gcloud secrets versions access latest --secret=secret-name --format='get(payload.data)' | tr '_-' '/+' | base64 -d > decrypted-data.txt" ]
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: /bin/sh
args:
- '-c'
- |
sed "s/super-secret-token/g" $(cat decrypted-data.txt)
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args: ['-c', 'gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy']
timeout: '1600s'
Having said that, your secret token will still be available unencrypted in your App Engine's environment variable which is not optimal security-wise. Instead you may want to query Secret Manager from within your App Engine code directly. You'll find code samples to do so here.

Permission error on Gitlab CI deployment with google app engine

I try to make a full process of integrated deployment with Gitlab and Google AppEngine.
But the deployment fail because it seems I miss rights. I really don't know which right is needed here.
I activated the App Engine API and the Cloud Build API.
The rights my service has:
my gitlab-ci.yml file:
Staging Deployment:
stage: Deploy
environment:
name: Staging
before_script:
- echo "deb http://packages.cloud.google.com/apt cloud-sdk-jessie main" | tee /etc/apt/sources.list.d/google-cloud-sdk.list
- curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
- apt-get update
- apt-get -qq -y install google-cloud-sdk
script:
- echo $DEPLOY_KEY_FILE_PRODUCTION > /tmp/$CI_PIPELINE_ID.json
- gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
- gcloud config list
- gcloud config set project $PROJECT_ID_PRODUCTION
- gcloud --quiet --project $PROJECT_ID_PRODUCTION app deploy gae/test.yml
only:
refs:
- master
And finally my test.yml:
# [START django_app]
runtime: python37
service: testing
entrypoint: gunicorn -b :$PORT manta.wsgi --timeout 120
default_expiration: "5m"
env_variables:
DJANGO_SETTINGS_MODULE: manta.settings.dev
handlers:
# This configures Google App Engine to serve the files in the app's
# static directory.
- url: /static
static_dir: static/
# This handler routes all requests not caught above to the main app.
# It is required when static routes are defined, but can be omitted
# (along with the entire handlers section) when there are no static
# files defined.
- url: /.*
script: auto
# [END django_app]
And the error I have during deployment:
ERROR: (gcloud.app.deploy) Permissions error fetching application [apps/testing]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.

Google Cloud Build - build.yaml & app.yaml in a sub directory not building correct source

I'm having trouble setting up my yaml files for google app engine. The configuration works correctly when my app.yaml file is in the root of the project but if it is within a subdirectory it does not build the correct source. I suspect I need to set the dir: option in the build config, but I have tried multiple variations and I can't get it to work.
Working file structure, deployed app is ~3mb in size.
src
deployment
└── staging
└── build.yaml
app.staging.yaml
# build.yaml
steps:
- name: node:12
entrypoint: yarn
- name: node:12
entrypoint: yarn
args: ['build']
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "app.staging.yaml"]
timeout: "1800s"
Not working file structure, deployed app is ~1kb in size.
src
deployment
└── staging
└── build.yaml
└── app.yaml
# build.yaml
steps:
- name: node:12
entrypoint: yarn
- name: node:12
entrypoint: yarn
args: ['build']
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "deployment/staging/app.yaml"]
timeout: "1800s"
In both scenarios I am kicking off the deployment with:
gcloud builds submit --config deployment/staging/build.yaml
What should my dir: be set to in the build.yaml steps so that the build step knows to build from root? Is there any way to debug this locally without having to upload the source every time?
Thanks!
A
you cannot have the app.yaml and the cloudbuild.yaml in the same directory if you are deploying in a non-custom runtime. Please see this comment
I believe by default cloud build expects the app.yaml file to be in the uppermost directory. I'm not sure if it's possible to change this in the cloud build settings.

Deploying automatically Node app to App Engine with Cloud Build and VPC connector

i have a Problem during deploying an App Engine app with Cloud Build and VPC connector to my MongodDB Atlas Database.
When i deploy it with gcloud, it works perfectly with this command:
gcloud beta app deploy
But i want CI (Continuous integration) to be implemented with Cloud Build.
During the Cloud build following error appears:
Step #3: #============================================================#
Step #3: #= Uploading 2 files to Google Cloud Storage =#
Step #3: #============================================================#
Step #3: File upload done.
Step #3: Updating service [nodeapi]...
Step #3: .......................................................................................................................................................................................failed.
Step #3: ERROR: (gcloud.beta.app.deploy) Error Response: [7] Error attaching GCE network to app.
Step #3:
Step #3: Details: [
Step #3: [
Step #3: {
Step #3: "#type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #3: "resourceName": "projects/visifingc/global/networks/default",
Step #3: "resourceType": "Network"
Step #3: }
Step #3: ]
Step #3: ]
Step #3:
Finished Step #3
ERROR
ERROR: build step 3 "gcr.io/cloud-builders/gcloud" failed: exit status 1
My app.yaml file:
runtime: nodejs10
service: nodeapi
vpc_access_connector:
name: "projects/visifingc/locations/europe-west1/connectors/app-engine"
network:
name: default
And cloudbuild.yaml:
steps:
- name: node:10.15.1
entrypoint: npm
args: ["install"]
- name: node:10.15.1
entrypoint: npm
args: ["run", "build"]
- name: "gcr.io/cloud-builders/gcloud"
args: ["beta","app","deploy"]
When a I look in App Engine to the configuration of Instance which was deployed (but with the error, therefore not really deployed)
following App Engine Instance Configuration can be seen:
runtime: nodejs10
env: standard
instance_class: F1
handlers:
- url: .*
script: auto
automatic_scaling:
min_idle_instances: automatic
max_idle_instances: automatic
min_pending_latency: automatic
max_pending_latency: automatic
network: {}
As you can see, something is wrong with the network because it is empty.
Could you help me please to find a solution? I thought it could be a rights problems, therefore i tried to give to all the accounts access rights to VPC, but id didn't helped.
Mostly i am following the defined setup with VPC network which is described here:
https://cloud.google.com/appengine/docs/standard/nodejs/connecting-vpc
I ran into the same problem, and it turns out it was a permission issue after all. For us, the fix was to give "Compute Network Admin" permissions to the CI/CD process.
I was having this issue earlier and struggled with it for a while. Solved it be changing from "gcloud beta app deploy" to simply "gcloud app deploy." Don't know why the beta version isn't working (especially if it was working from your CLI).
I know this isn't a satisfying answer, but hopefully it puts you on the right track!
BEFORE:
steps:
- name: "gcr.io/google-containers/busybox"
args: ["sed", "-i", "s/PROJECT_ID/${PROJECT_ID}/g", "app.yaml"]
- name: "gcr.io/cloud-builders/gcloud"
args: ["beta", "app", "deploy"]
AFTER:
steps:
- name: "gcr.io/google-containers/busybox"
args: ["sed", "-i", "s/PROJECT_ID/${PROJECT_ID}/g", "app.yaml"]
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]

Resources