Jenkins with Google Cloud Source Repository - google-app-engine

I am trying to use Jenkins for CI/CD.
I've developed a Python flask application. I am deploying this application into Google App Engine. So far I am using gcloud app deploy app.yaml command to deploy application to Google App Engine.
Code for this application is present in Google Cloud Source Repository.
Since the authentication to git(Google Cloud Source Repository) requires Google OAuth, I've installed Google OAuth Credentials Plugin
Now I am facing two issues
When I use "Google Service Account from metadata" Credential Kind, I am not seeing the credentials listed in "Source Code Management".
when I use "Google Service Account from private key", I am able to see credentials. But when I run my jenkins job I am getting below error
FATAL: Could not call
com.google.jenkins.plugins.source.GoogleRobotUsernamePassword.writeObject()
: Failed to serialize
com.google.jenkins.plugins.source.GoogleRobotUsernamePasswordModule$ForRemote#credentials
for class
com.google.jenkins.plugins.source.GoogleRobotUsernamePasswordModule$ForRemote
---- Debugging information ---- message : Could not call com.google.jenkins.plugins.source.GoogleRobotUsernamePassword.writeObject()
cause-exception : java.lang.RuntimeException cause-message :
Failed to serialize
com.google.jenkins.plugins.source.GoogleRobotUsernamePasswordModule$ForRemote#credentials
for class
com.google.jenkins.plugins.source.GoogleRobotUsernamePasswordModule$ForRemote
------------------------------- java.lang.UnsupportedOperationException: Refusing to marshal
org.joda.time.DateTime for security reasons; see
https://jenkins.io/redirect/class-filter/ at
hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:543)
at
com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at
com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at
com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
Question: How can I authenticate Google Cloud Source repository in Jenkins? What are the steps|plugins required to use Google Cloud Source repository in Jenkins?

Solution
You probably need to add a missing scope to the Compute Engine VM Instance running Jenkins that would give it access to Cloud Source Repository. You can follow the documentation or this steps, which ever you find convenient.
Go to Cloud Deployment Manager
Click on the name of the Jenkins deployment in question
Click on the instance name in the left info pane and it will redirect you to VM instance details
Stop the instance
Press Edit and set the right access scope for Cloud Source Repository
After your start the VM instance, try adding your git repository again and once you select the credentials, either "Google Service Account from metadata" or from service account, everything should work properly.
Elaboration
I stumbled upon the "Invalid authentication credentials." issue while trying to add a Cloud Source Repository after deploying Jenkins from Launcher.
In my case the reason why it happened was that during the process of deployment the Cloud API access scope for Cloud Source Repositories on the Compute Engine VM instance was set to Disabled, which prevented any interaction from that instance even if a service account had all the necessary roles/permissions.
Here are the scopes that are reconfigured by Launcher:
scopes:
- 'https://www.googleapis.com/auth/cloud.useraccounts.readonly'
- 'https://www.googleapis.com/auth/devstorage.read_only'
- 'https://www.googleapis.com/auth/logging.write'
- 'https://www.googleapis.com/auth/monitoring.write'
{% if enableComputeApi %}
- 'https://www.googleapis.com/auth/compute'
{% endif %}
- 'https://www.googleapis.com/auth/cloudruntimeconfig'
Adding the following scope to the VM instance running Jenkins was enough to fix the error:
https://www.googleapis.com/auth/source.read_only
Extra:
List of scopes for Google APIs.

I updated the plugin to 0.6 version.
In 0.6 version, I have the same problem in Jenkins "Free-style" project.
Bit I can success to checkout in Jenkins "pipeline" project.
I just copy the "checkout syntax" in others project and replace 2 parts of the syntax to the right:
credentialsid -> replace to your right credentials id that you created Google Service Account from private key in jenkins.
url ->replace to your google source repository url

Related

Google app engine deployment using wrong service account?

I have this app that's been running on Google App Engine for a good couple of years. Today I needed to modify its app.yaml file, but when I ran "gcloud app deploy" I got this error:
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [7] Failed to create cloud build: service account "redacted#cloudbuild.gserviceaccount.com" has insufficient permission to execute the build on project "redacted"
It appears that the default service account no longer has the necessary permissions. In fact, it may have been deleted completely, because I don't see it listed under gcloud projects get-iam-policy xxxx. What I see instead is this:
serviceAccount:service-redacted#gcp-sa-cloudbuild.iam.gserviceaccount.com
with a correct-looking role cloudbuild.serviceAgent.
I tried googling any changes related to GAE service accounts but didn't find any useful references. I followed the solution to this question, but it didn't help.
The question is, how do I change the permissions of the service account mentioned in the original error message, or, how do I tell GAE to use another (correct) service account when deploying my app?
According to [this][1] documentation
...To deploy new versions, a member must have the Service Account User (roles/iam.serviceAccountUser) role on the App Engine default service account, and the Cloud Build Editor (roles/cloudbuild.builds.editor) and Cloud Storage Object Admin (roles/storage.objectAdmin) roles on the project.....
Using the IAM page you can add the email address of the person deploying (the one you have authenticated via gcloud) and add the above roles to that email
[1]: https://cloud.google.com/iam/docs/understanding-roles?hl=en#app-engine-roles

Setting target url with gcloud

I have been using the Google App Engine Launcher's "Deploy" button. I realize that the Launcher is going away at the end of July 2020, so I'm trying to learn to use gcloud. My web site has a custom url, www.xxx.org, but "gcloud app deploy" uploads to xxx.appspot.com instead of to www.xxx.org, presumably because "target url" is set to xxx.appspot.com. Can someone please tell me what gcloud command I need to use to change "target url" to www.xxx.org, or some other scheme that would work? I've wandered through lots of gcloud and GAE documentation and web searches without being able to figure out what I need to do.
There is a very good and complete guide about mapping a custom domain to your application in the official documentation. As #DanCornilescu said,it needs to be configured, but it is not necessary to be done before deployment.
Basically, you need to follow the steps from the guide:
1) Verify that you are the owner of your domain through Webmaster
Central
2)Ensure that your domain has been verified, otherwise you will not be able to proceed with the following steps. Note that only verified domains will be displayed.
3)If you need to delegate the ownership of your domain to other users or service accounts, you can add permission through the Webmaster Central page
4)After you verify ownership of your domain, you can map that domain to your App Engine app
5)Add the resource records that you receive to the DNS configuration of your domain registrar
6)Test for success by browsing to your app via its new domain URL, for example www.example.com
In case you need to do more complex mappings for different urls and subdomains you can always write down the routing rules in the dispatch.yaml file and they will be mapped successfully.
Also, to add a bit more of information about the dipatch.yaml file, it will handle the routing in the App as opposed with the app.yaml which will take care of the App's settings. Both .yamls can exist in the same application and can be deployed at the same time by using the following gcloud command:
gcloud app deploy app.yaml dispatch.yaml
The deployment is always done to .appspot.com, the custom domain is automatically updated if it is already configured. If it is not then it needs to be configured (it can be done after the deployment as well).

Replacing the existing Project on Google Cloud Platform

I have deployed a Java Web application on Google Cloud platform and have purchased a domain and SSL Certificate from GoDaddy.com.
Its a simple application with a login page and a registration, where a specific(only one user) will have login credentials and post login will submit a form which will be saved in google DATASTORE in Google Cloud Platform(GCP).
I have two queries here :
1)
My application works perfectly,on my machine and even few machines around my home (as i have verified in few cyber centres around) but not at a client place located in different city(150 miles away). I'm not sure,as the client is not running any firewall or any browser plugin, as i have not visited the client place yet.
The call is as below.
Domain purchased from godaddy.com is : "ABC.in"
App engine project linked to this domain is : "web-abc"
If we hit "ABC.in" in browser it will in-turn call "web-abc.appspot.com".
in the background.Now though the login page loads,the client complains that the next subsequent pages does not load and page becomes blank.But for me in my system it works perfectly,as i can see the updated DATASTORE in GCP.
2)
Once i configure my DNS at GoDaddy.com, i'm unable to make any changes to the project hosted on GCP app engine.So i decided to create a new project and host it on the GCP. But GCP app engine page does not show any option to replace a project as it showed me during my first project setup.
I need to replace project "web-abc" linked to "ABC.in" with "web-pqr".
Please Please help...!
Thanks
To answer your second query you can just rest your CNAME record in the GODADDY DNS configuration and add the web-pqr reqq. That should work. Rest of the question< i really don't get it. Please share some error message or the screenshot when you try make change to your exiting project.

Insufficient authentication scopes from Jenkins build via Google Compute Engine

I searched the web for a while, but can't seem to find the right answer.
I have created a VM Instance on Google Compute Engine and I am running Jenkins on it. This instance checks a code repository I have and when a change occurs, I want to run the following command
gcloud --project=test preview app deploy -q app.yaml --version=dev
When I want to trigger a build, or it triggers it by himself, I get the following error:
Beginning deployment...
ERROR: Error Response: [403] Request had insufficient authentication scopes.
ERROR: (gcloud.preview.app.deploy) Could not retrieve the default Google Cloud Storage bucket for [test].
Please try again or use the [bucket] argument.
The VM instance does have acces to the storage as you can see in the following image, so I don't understand why this error pops up.
Go to the App Engine developer console https://appengine.google.com/ and in
Application Settings > Cloud integration > and turn ON the Google Cloud Storage
so this will set up the default bucket for you. This something that the new Google Developer Console will do automatically.If the project is created on the old console you have to do it manually.

Issue authorizing access with google analytics super proxy

I'm setting up google analytics super proxy to create some public queries on my google analytics data. I have followed the instructions given in the below link to setup a analytics super proxy in app engine:
https://developers.google.com/analytics/solutions/google-analytics-super-proxy
I have followed the steps specified, enabling access for analytics api and creating a client ID for the super proxy web application deployed in app engine. I updated the required properties in the app.yaml and config.py files in the super proxy and deployed it to app engine.
I'm able to open the admin page of the deployed super proxy, but when I try to authorize access, I get the an error saying 'invalid_client' (although the client id seems to match with the one I created in developer console):
I'm unable to figure out what has gone wrong, or if I've missed something? Can you please help?
Regards,
Anand
Go to the Dev Console -> Your Project -> APIs & auth -> Consent screen. Make sure all the non optional fields are filed in (specifically in your case PRODUCT NAME).

Resources