I have a maven project that updates appspot with my war, but when the build is executed by Atlassian Cloud Bamboo the OAuth authentication prompts for access tokens.
14-May-2015 00:17:18 [INFO] Retrieving Google App Engine Java SDK from Maven
14-May-2015 00:17:18 [INFO] Updating Google App Engine Application
14-May-2015 00:17:18 [INFO] Running -V 1.9.20 --oauth2 update /mnt/bamboo-ebs/bamboo-agent/build-dir/[project path]-1-SNAPSHOT
14-May-2015 00:17:21 Please open the following URL in your browser:
14-May-2015 00:17:21 https://accounts.google.com/o/oauth2/auth?access_type=offline&approval_prompt=force&client_id=XXXXXXXX.apps.googleusercontent.com&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=https://www.googleapis.com/auth/appengine.admin%20https://www.googleapis.com/auth/cloud-platform
Opening the url I'm able to get an access token, but I'm not able to enter it so the build hangs until cancelled.
How should I setup authentication?
Add the oath credentials to an EBS attached volume and copy them into place when the Agent is started.
Configure bamboo for EBS attached volumes
SSH into the instance, su to bamboo user
Download the Appengine SDK
Run appcfg to be prompted for authentication
appcfg -s my-app.appspot.com version
follow instructions for oauth which will create /home/bamboo/.appcfg_oauth2_tokens_java
mkdir /mnt/bamboo-ebs/appengine
cp /home/bamboo/.appcfg_oauth2_tokens_java /mnt/bamboo-ebs/appengine
edit /mnt/bamboo-ebs/bin/customize-extras.sh (see docs) adding the following
cp /mnt/bamboo-ebs/appengine/.appcfg_oauth2_tokens_java /home/bamboo/
chown bamboo:bamboo /home/bamboo/.appcfg_oauth2_tokens_java
Take a snapshot of the volume and assign the snapshot to your bamboo instance configuration
The next time bamboo starts a new instance the customize-extras.sh will be run and the necessary credentials will be in place for mvn appengine:update to find.
Related
Greetings stackoverflow community! First time asker, long time user.
I am testing out my cloudbuild.yaml file locally using Cloud Build Local component and Secret Manager and it is failing on "availableSecrets".
Error message: Error loading config file: unknown field "availableSecrets" in cloudbuild.Build
OS Platform: Windows 10/WSL2/Ubuntu 18.04
cloud-build-local: v0.5.2
Docker engine: v20.10.2
Nodejs version: v14.15.3
NPM version: 6.14.9
gcloud version: 326.0.0
Installed components: [BigQuery Command Line Tool, Cloud Datastore Emulator, Cloud SDK Core Libraries, Cloud Storage Command Line Tool, Google Cloud Build Local Builder, gcloud Beta Commands]
Documentation on Cloud Build build file: https://cloud.google.com/cloud-build/docs/build-config
Documentation to configure secrets with cloud build: https://cloud.google.com/cloud-build/docs/securing-builds/use-secrets
Documentation for cloud build local: https://cloud.google.com/cloud-build/docs/build-debug-locally
Steps performed:
Added secrets to Secret Manager
Enabled API between Cloud Build and Secrets Manager
Added cloudbuild service account as member of each secret password.
Added IAM permission Secret Manager Secrets Accessor to cloudbuild user. I don't know where I got this info from but it is residual at this point from other attempts to use Secret Manager with cloudbuild. I am not sure of the difference between applying access here vs applying to the Secret Manager secret.
Command: cloud-build-local --config=cloudbuild.staging.yaml --dryrun=false .
cloudbuild.staging.yaml:
- name: gcr.io/cloud-builders/npm
entrypoint: 'npm'
args: [ 'install' ]
- name: 'gcr.io/cloud-builders/gcloud'
args: ["app", "deploy"]
env:
- 'DAO_FACTORY=datastore'
- 'POLL_INTERVAL=15'
- 'PROMPT=staging>'
- 'ENVIRONMENT=staging'
- 'NAMESPACE=staging'
- 'RESET_DATASTORE=false'
secretEnv: ['ADMIN_USER', 'SUPER_ADMINS', 'BOT_TOKEN']
availableSecrets:
secretManager:
- versionName: projects/{project token}/secrets/SYSTEM_USER/versions/1
env: 'ADMIN_USER'
- versionName: projects/{project token}/secrets/SUPER_ADMINS/versions/1
env: 'SUPER_ADMINS'
- versionName: projects/{project token}/secrets/BOT_TOKEN/versions/2
env: 'BOT_TOKEN'```
Tag: cloud-build-local. I guess without reputation a meaningful tag cannot be created. Maybe an esteemed community member will create this as this may be specific to cloud-build-local only.
Support for Google Secret Manager in Google Cloud Build descriptor file is apparently very new and does not appear to be supported by cloud-build-local component at this time; please see comment from Guillaume about feature being a week old. When cloud build descriptor is ran in Cloud Build, it works fine.
I fixed a similar issue by upgrading the gcloud tool.
Is it possible to deploy Elasticsearch on App engine flex environment using a docker image.
I have tried the following
My files on the local machine
Folder : elasticsearch
app.yaml
Dockerfile
docker-entrypoint.sh
config folder(containing elasticsearch.yml)file
Contents of app.yaml
runtime: custom
env: flex
Dockerfile and docker-entrypoint.sh copied from https://github.com/GoogleCloudPlatform/elasticsearch-docker/tree/master/5/5.2.0
Modifications to the Dockerfile
replaced EXPOSE 9200 9300 to EXPOSE 8080
Modification to the elasticsearch.yml
cluster.name: "beaconinside-docker-cluster"
path.data: /usr/share/elasticsearch/data
http.host: 0.0.0.0
http.port: 8080
discovery.zen.minimum_master_nodes: 1
I build a container using the docker file on my local machine
docker build -t elasticdemo .
Then, I run the container
docker run -p 8080:8080 elasticdemo
I am able to access elasticsearch on 0.0.0.0:8080
Problem:
I am trying to deploy elasticsearch as an app to Google app engine flex environment
gcloud app deploy app.yaml --version elasticdocker --project myproject
The deployment fails with the following error
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
I was expected elasticsearch to deploy as an app and be available on the deployed url.
Could you please provide pointers/help/suggestions with this approach?
While you can deploy ES to App Engine Flexible environment it's not particularly useful. The VMs hosting GAE Flexible containers are restarted regularly as part of maintenance and whatever data is stored on the local disk will be lost on restart. If you want to use local disk for long term storage, I'd suggest to deploy the GCE VM's (or alternatively use a solution from the GCP Marketplace) or deploy to GKE which supports persistent disks
As for the actual question: you probably don't have a health check handler and therefore App Engine Flexible environment doesn't consider your app healthy after deploying it. The error message is useless, I agree.
From the GAE Flexible docs for building custom images:
"A health check is an HTTP request to the URL /_ah/health. A healthy application should respond with status code 200."
Alternatively you can turn off health checks by adding into app.yaml
enable_health_check: False
I have a set of data that I would like to use for production but I can't seem to get appcfg.py to download the localhost datastore. I have localhost:8080/_ah/remote_api on in app.yaml then, in the SDK shell, i ran
appcfg.py download_data --filename=local.db --url=http://localhost:8080/_ah/remote_api -A [MYAPPID]
I get this error message:
You must be logged in as an administrator to access this.
I can't find a way to authenticate my google account as the admin. any advice?
This is a general issue of remote_api admin authentication to the local development server, you may want to star it: https://code.google.com/p/googleappengine/issues/detail?id=12445. Several workaround suggestions in there (YMMV according to other posts)
The most recent one (2 days old) looks promising:
Workaround for appcfg.py upload_data to local dev server,
"Refreshing due to a 401" issue. Tested using Java dev server 1.9.42:
Log into the local dev server Development Console in a browser.
Open your browser's dev tools and pull the value for the dev_appserver_login cookie.
Edit lib/oauth2client/oauth2client/client.py in the app.
Find "def new_request" function. Before line self.apply(headers) in this function, add:
headers['Cookie'] = 'dev_appserver_login=your-cookie-value-here';
Run the upload_data command.
There's also this recipe, successful at the time, but it might be a bit old now:
appcfg.py shows You must be logged in as an administrator
The CircleCI appengine documentation suggests using a password to do deployment. How can I use the oauth2 flow instead of using passwords? I don't want to share my Google password.
Do I generate a ~/.appcfg_oauth2_tokens_java file, from token data stored as environment variables in CircleCI? Is there a simpler way?
I solved the issue this way:
deployment:
appengine:
branch: master
commands:
- erb .appcfg_oauth2_tokens_java.json > ~/.appcfg_oauth2_tokens_java # requires ENV in circle ci
- mvn -DskipTests=true appengine:update # tests have already been run
.appcfg_oauth2_tokens_java.json:
{
"credentials": {
"ubuntu": {
"access_token": "<%= ENV["GOOGLE_ACCESS_TOKEN"] %>",
"expiration_time_millis": 1431552739090,
"refresh_token": "<%= ENV["GOOGLE_REFRESH_TOKEN"] %>"
}
}
}
Then in CircleCI, configure the ENV variables for the two tokens. I got the tokens by locally running mvn appengine:update and going through the oAuth2 dance. Note: You may have to remove your existing ~/.appcfg_oauth2_tokens_java file first.
Reading the AppEngine SDK docs, it sounds like that would be a good approach. There is not a built-in way to do that on CircleCI.
If you don't want to use any user-related credential, you can leverage service accounts, like mentioned in this blog post:
Continuous Deployment with Google App Engine and CircleCI
I solved it this way on the latest GAE SDK 1.9.34 for Java.
Assuming you have a Base64 encoded ENV Variable with your JSON key for a service account you've created on the GCloud project:
dependencies:
pre:
- echo $GOOGLE_CLIENT_SECRET | base64 --decode > ${HOME}/client-secret.json
And then in the deployment section:
- $HOME/appengine-java-sdk-$APP_ENGINE_VERSION/bin/appcfg.sh -A $GCLOUD_PROJECT -M $GCLOUD_MODULE -V $BUILD_VERSION --service_account_json_key_file=$HOME/client-secret.json update $WAR_FOLDER
The --service_account_json_key_file doesn't seem to appear as an option when you use appcfg.sh help but it is there, and does work.
I am in the process of setting up a "QA environment" for my GAE app. This QA environment will simply be a small server on my home network with a dedicated IP address. I'm writing an Ant script to check the project out of my SVN repo, build it on my build server, and then deploy it "remotely" (across my home LAN) to the QA app server.
With Tomcat, I would just scp the web archive to the machine's webapps/ directory, and since it can be configured to hot-deploy, that is all I usually need for a QA deploy.
But I'm new to GAE, and so I'm not seeing how I can achieve such a remote deployment via Ant. The best I can think of (although somewhat convoluted) would be:
Checkout and build the WAR on the buildserver, like I normally would
scp the WAR to a staging directory, somewhere on the QA machine; say 192.168.1.55:/opt/gae/staging
Have a lightweight RESTful web service running on that machine (maybe hosted by Tomcat or Jetty) listening for a client to hit a certain API, say http://192.168.1.55:8080/GaeRemoteApi/deploy; when the request handler gets a request for this URL, it kicks off a shell command to copy the WAR into the correct directory and then execute appcfg.sh -upload to actually deploy the WAR to my QA app server
I'm pretty sure I could get this working within a day or two, but was wondering if the GAE ships with an easier (baked in) solution; or if a fresh set of eyes can think of something even simpler. Thanks in advance!
I think you should just keep it simple:
Since you are on Ubuntu, you can write a shell script that will:
ssh to the remote server
stop the current gae dev appserver
rename the existing war directory
scp the new deployment to the QA server war directory
ssh to the QA server and start the gae dev appserver
You can call a shell script from ant using: http://sumedha.blogspot.com.au/2008/06/how-to-call-shell-script-from-ant.html
To stop the dev appserver:
killall -e ./appengine-java-sdk/bin/dev_appserver.sh
To run the dev appserver:
nohup ./appengine-java-sdk/bin/dev_appserver.sh you/war/directory &
Run the development server?
https://developers.google.com/appengine/docs/java/tools/devserver