I have configured a CI pipeline using a cloudbuild.yaml file. I'm trying to launch this pipeline with Pull Requests. It seems that the provided build triggers: https://cloud.google.com/cloud-build/docs/running-builds/automate-builds are not allowing this option. Is there a way to use webhooks to overcome this limitation? Like sending an HTTP request after a pull request event to cloud builds topic and configure a cloud function as a subscriber to launch the pipeline.
Thanks,
The Cloud Build Github App does builds on pull request: https://cloud.google.com/cloud-build/docs/run-builds-on-github
There are three ways to run builds
Manually: through API/gcloud
(Beta) Build Triggers: configurable through Google Cloud Console
(Alpha) Github App: builds automatically on changes to repo and pull
requests
These can all be used independently or in combination with each other.
A completed pull request is merged into an upstream branch (master, release, or another name). https://help.github.com/en/articles/merging-a-pull-request
You can set the Google Cloud Build trigger (in Google Cloud Console) type to "Branch" and enter the relevant branch. Choose Cloud Build configuration file as your build configuration and enter your cloudbuild.yaml file location.
Related
I'm using Bitbucket Pipelines to deploy my project to Google App Engine via gcloud app deploy which does a great job accomplishing what I want.
gcloud app deploy takes like 6-8 extra minutes for "Updating service..." which is the time taking for them to update their backend infrastructure primarily the Load Balancer.
This is not an issue by itself except that it eats up my monthly Build Time from Bitbucket.
I would rather have the pipeline return as soon it sent off the build commands, and I'll check them myself on Google Cloud Logs server.
The question is: do we have any flag for gcloud app deploy to tell him not to wait for the "Updating service..." ?
These are all the gcloud app deploy flags. There's no such flag.
To see if there's a possible 'hack' you could use, you could try manually deploying your app yourself using gcloud app deploy --log-http. The --log-http flag will produce an output of all http requests made during the deploy. You'll see the endpoints being called, the http method, the headers, payload and the duration for each call. Examining those (especially around the 'updating' bit) might show you something that could potentially be of help.
I'm a little bit confused between gcp components, here is my use case :
daily, I need to fetch data from an external API (the API return json data), store it in GCS then load it in Bigquery, I already created the python script fetching the data and store it in GCS and i'm confused which component to use for deployment :
Cloud run : from the doc it is used for deploying services, so I think its a bad choose
Cloud function: I think it works, but it is used for even based processing (through single purpose function...)
composer :(I'll use composer to orchestrate tasks, such as preprocessing of files in GCS, load them to BQ, transfert them to an archive Bucket) through kubernetesPodOperator, create a task that trigger the script to get the data
compute engine: I don't think that its the best chose since there are better ones
app engine: also I don't think it a good idea since it is used to deploy and scale web app ...
(correcte me if i'm wrong in what I said, ) so my question is : what is the GCP component used for this kind of task
Cloud run : from the doc it is used for deploying services
app engine: also I don't think it a good idea since it is used to deploy and scale web app ...
I think you've misunderstood. Both Cloud run and Google App Engine (GAE) are serverless offerings from Google Cloud. You deploy your code to any of them and you can invoke their urls which in turn will cause your code to execute and do stuff like go fetch data from somewhere and save it somewhere.
Google App Engine has a shorter timeout than Cloud Run (can't remember if Cloud Run has time out). So, if your code will take a long time to run, you don't want to use Google App Engine (unless you make it a background task) and if you don't need a UI, then you don't need GAE.
For your specific scenario, you can deploy your code to Cloud Run and use Cloud Scheduler to schedule it to be invoked at specific times. We have that architecture running in a similar scenario (we have a task that runs once daily; it's deployed to Cloud Run; Google Scheduler invokes the endpoint, it runs and saves data to datastore linked to an App Engine App). We wrote a blog article on deploying to Cloud Run and another on securing your cloud run (based off our experience in the earlier described scenario)
GAE Timeout:
Every request to a Google App Engine (Standard) must complete within 1 - 10 minutes for automatic scaling and up to 24 hours for basic scaling (see documentation). For Google App Engine Flexible, the timeout is 60 minutes (documentation).
I would like to know how to deploy the application from bitbucket using pipelines to multiple Google Cloud projects.
Here is our current set up and it is working fine.
On Bitbucket, the application repo with development/UAT test/production branches, once the pull request approved and merged into development/production, it shall deploy to the GCP app engine through the pipelines.
The problem now, we want to isolate each client in GCP which mean each client will have its own GCP project, cloud SQL, App engines, storage bucket, etc...
I need some advice on how to change the deployment workflow in bitbucket and pipelines, so will work for the new set up.
For the branches setup on bitbucket, I'm thinking like below, but if I go for option2, then it seems too much if got more clients.
Option 1 (repo branches)
development/
UAT test/
validation/
production
Option 2 (repo branches)
development/
UAT test client1/
UAT test client2/
validation_client1/
validation_client2/
production_client1/
production_client2/
The first step, I know I have to create different app.yaml for each app engine service for each client, so it can deploy the app engine service to different CGP projects/bucket/SQL instance.
Also just found out the bitbucket-pipelines.yml only support 10 steps, if I create so many branches then it will over the limits for sure.
Does anyone have any suggestions about how should be set up?
Thanks,
You could create Cloud build triggers to specific bitbucket branches or repos (whatever your branching model is defined) and deploy the app engine implementation to the App engine service on the same project, and if you need to customize other steps, you could use custom steps as described here. Finally you can take a look at how to create a basic configuration file for Cloud Build if you are not very familiar with this product
The Nginx load balancer of Google Cloud App Engine Flexible Environment (with custom runtime) logs remote IP addresses etc. to its access logs. What if I wanted to disable that behavior? Is it possible in any way? Docs don't say anything about configuring Nginx by yourself. Haven't find anything from the console either.
In order to disable any IP logging, you have to disable the Google Cloud Load Balancer logs and also any logging done by NGINX.
In order to disable the ones done by the Load Balancer of Google Cloud, this page of their official documentation explains how to do it. You would need select "Disable log source" from the cloud HTTP Load Balancer.
On the other hand, to configure NGINX, apparently the only way to do so is modifying the nginx.conf file.
Because this you are using App Engine Flex and it's not possible to access these virtual machines/containers once the App had been deployed, you would need make sure to change the configuration before executing the deployment.
I've found a Serverfault post which does explain how to disable NGINX logging, and another article which does explain the NGINX configurations with better depth in case they might be of use for you.
I believe this won't be possible. As per this Logging docs page:
The App Engine flexible environment produces the following logs:
Request logs record requests sent to all App Engine apps. The request
log is provided by default and you cannot opt out of receiving it.
App logs record activity by software within the App Engine app. The
log is provided by default and you cannot opt out of receiving it.
Runtime logs are provided from the flexible environment using a
preinstalled Logging agent.
I have a Google Cloud Platform application that runs on several Google App Engine Standard instances. One app has a cron.xml with some cron jobs defined pointing to REST endpoints on that app.
Now I want to create a cron job in another app, so I created another cron.xml, all the REST endpoints and deployed. After deployment I realized the deployment erased the cron jobs defined by the first app. I read somewhere that you can only have one cron.xml defined PER PROJECT and not PER APP. Is this correct? I have been reading the documentation and a book on Google App Engine and could not find an answer.
cron.yaml/cron.xml apply per application.
I just got an answer from Google Cloud support. Apparently there can be only one CRON file PER PROJECT. Each upload of a cron file will override the previous upload.
A workaround for this would be add a TARGET tag for the URLs that are not in the same App as the CRON file. Adding target: will re-route the request to the appName app.