I am having an issue deploying a Node project to Google App Engine when using a service account. I am getting a 403 error.
{
"error": {
"code": 403,
"message": "Operation not allowed",
"status": "PERMISSION_DENIED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.ResourceInfo",
"resourceType": "gae.api",
"description": "The \"appengine.applications.get\" permission is required."
}
]
}
}
I have given my service account the App Engine Admin and Storage Object Admin roles. I am using the downloaded JSON key file to deploy.
I am then running these two terminal commands:
gcloud auth activate-service-account --key-file appEngineAuth.json
gcloud --quiet --verbosity=debug app deploy app.yaml --promote --log-http
This is my app.yaml file:
runtime: nodejs
env: flex
If I run gcloud auth list I see my service account user selected. I am able to deploy if I do gcloud init and then go through the process of using my Google account but I can't do that from my CI server.
I have deleted and recreated my service account a couple of times and made absolutely certain I am using the correct key file. I even tried giving my service account the Owner role but that didn't work.
This turned out to be an app engine bug. Support took about a week to fix the issue. As a temporary work around I had to create a new application and use it. This bug only affected one of my six applications.
Google case #: Case 15238823
Related
I want to deploy from bitbucket to google app engine using Open ID Connect (OIDC) authentication method (not key file).
I've setup Open ID Connect in my bitbucket repository. It shows
Identity provider URL
Audience
Example paylode
Unique identifiers
I've setup a Service Account in GCP that has App Engine Deployer role.
I've setup Workload Identity Federation with a pool and a provider. The Issuer(URL) of provider is set to Identity provider URL of bitbucket OIDC.
I've granted the service account access to the pool.
Here is my deployment step in bitbucket-pipelines.yml:
- step:
name: Deploy to Production
image: google/cloud-sdk:latest
oidc: true
script:
- echo $BITBUCKET_STEP_OIDC_TOKEN > identity-token
- gcloud auth login --cred-file clientLibraryConfig.json --update-adc
- gcloud app deploy
The clientLibraryConfig.json is downloaded from GCP:
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/837282586571/locations/global/workloadIdentityPools/fakfake-com/providers/bitbucket-fakfake-com",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/bitbucket#fakfake-com.iam.gserviceaccount.com:generateAccessToken",
"credential_source": {
"file": "identity-token",
"format": {
"type": "text"
}
}
}
Here is the output from pipeline:
echo $BITBUCKET_STEP_OIDC_TOKEN > identity-token
gcloud auth login --cred-file clientLibraryConfig.json --update-adc
+ gcloud auth login --cred-file ../../clientLibraryConfig.json --update-adc
Application default credentials (ADC) were updated.
Authenticated with external account credentials for: [bitbucket#fakfake-com.iam.gserviceaccount.com].
Your current project is [fakfake-com]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
gcloud app deploy
+ gcloud app deploy ../../app.yaml
ERROR: gcloud crashed (OAuthError): ('Error code invalid_grant: The audience in JWT does not match the expected values.', '{"error":"invalid_grant","error_description":"The audience in JWT does not match the expected values."}')
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
Build teardown
How can I fix the error of gcloud app deploy?
Created a new project, on first deploy I get this:
Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [13] Flex operation projects/primalexchange/regions/us-east1/operations/b5a5813a-14bf-43b6-af7f-e17b15866c4f error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2020-05-16T23:23:08.373Z4721.jc.14: Deployment Manager operation primalexchange/operation-1589671388926-5a5cc367252d6-eae4c5cb-bc5f1a6e errors: [code: "RESOURCE_ERROR"
location: "/deployments/aef-default-20200516t192110/resources/aef-default-20200516t192110"
message: "{\"ResourceType\":\"compute.beta.regionAutoscaler\",\"ResourceErrorCode\":\"403\",\"ResourceErrorMessage\":{\"code\":403,\"errors\":[{\"domain\":\"usageLimits\",\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'aef-default-20200516t192110\'. Limit: 8.0\",\"reason\":\"limitExceeded\"}],\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'aef-default-20200516t192110\'. Limit: 8.0\",\"statusMessage\":\"Forbidden\",\"requestPath\":\"https://compute.googleapis.com/compute/beta/projects/primalexchange/regions/us-east1/autoscalers\",\"httpMethod\":\"POST\"}}"
The build in the builder shows as a completed success.
It shows under versions that I have one version, with 0 instances, 100% traffic allocation, and when you click on the version it takes you to my site showing a 404 error.
Probably because it says that service is stopped, and that's probably because it didn't 'update'.
I don't think this is a 'my code' thing, 'cause I did a git checkout of a version that I once deployed under a different project successfully that basically amounts to a hello world app and while it once succeeded, it is now failing. Furthermore, I had another working deployment under the same project just a couple of days ago, but when I went to deploy with updated code, the deployment failed, with this very error, 'causing me to delete the entire project and start over, and now I'm getting the same error on this new project. How do I know it wasn't my code that caused the issue in the first place? I've had a build that failed because of my code and the logs of the build give that away. Here the build succeeds. there are no failures in the build.
So I've gone to this url https://compute.googleapis.com/compute/beta/projects/primalexchange/regions/us-east1/autoscalers
which states this:
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}
and I don't know if that's my problem or something else. I don't even know what that means really. so I made sure I've got creds at 'projectid.json'
and did
export GOOGLE_APPLICATION_CREDENTIALS='projectid-2c8anumber842b4.json'
and i disconnected code related to my firebase login stuff.
I did these commands
gcloud services enable servicemanagement.googleapis.com
gcloud services enable servicecontrol.googleapis.com
gcloud services enable endpoints.googleapis.com
I don't know how to fix this issue. I've been trying to fix it all weekend.
I when to Quota's under IAM, and looked at the quota for autoscalers in us-east1 and I found this which is not over anything:
and the only thing with a limit of 8 is cpu's and it says I'm at 2.
I don't know what I'm messing up..
yes, billing is enabled.
I just ran across this, which maybe this will get me some place...
What's an authorization header? And why didn't I need that the first time.. or what have I messed up this time.
And I changed my code so that all you get is a misspelling of hello world and it's still failing on deploy:
import logging
from flask import Flask, jsonify, redirect, url_for
from flask import request, make_response
from flask import render_template
from image_upload import upload_blob
from google.cloud import storage
from google.cloud import datastore
from datetime import datetime
from cryptography.fernet import Fernet
from models.media import Media
from models.user import User
from google.cloud import ndb
import json
import random
from uuid import uuid4
from flask import Flask, make_response, request
from google.auth.transport import requests
import google.oauth2.id_token
# from google.cloud import firestore
from flask_wtf.csrf import CSRFProtect, generate_csrf
from werkzeug.utils import secure_filename
from werkzeug.exceptions import RequestEntityTooLarge
#app.route('/', methods=['GET'])
def hi():
return "hellow world"
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
# This is used when running locally. Gunicorn is used to run the
# application on Google App Engine. See entrypoint in app.yaml.
app.run(host='127.0.0.1', port=8080, debug=True)
and now that I look at this more closely, how is this even running locally when I clearly have a double import of flask?
I will try to deploy after removing that, but i doubt that's the issue I'll be back to publish another essay about this issue shortly.
yeah that didn't matter. still failed.
This image is from the third project I started. And I got it to deploy up until now.. where it's failing with the same error:
Here's the App.yaml
runtime: python
api_version: '1'
env: flexible
threadsafe: true
handlers:
- url: /static/(.*)
application_readable: true
static_files: __static__/static/\1
require_matching_file: false
upload: __static__/static/.*
- url: /.*
script: main.app
secure: never
env_variables:
SECRET_KEY: ____________
automatic_scaling:
cool_down_period: 120s
min_num_instances: 2
max_num_instances: 20
cpu_utilization:
target_utilization: 0.5
network: {}
liveness_check:
initial_delay_sec: 300
check_interval_sec: 30
timeout_sec: 4
failure_threshold: 4
success_threshold: 2
readiness_check:
check_interval_sec: 5
timeout_sec: 4
failure_threshold: 2
success_threshold: 2
app_start_timeout_sec: 300
And it looks when I deploy I'm hitting an IP address in use limit of 4 when I checked the quota, and then it goes back down to 2 when I'm not. So maybe someone can advice me of how to set my app.yaml on how to avoid this? I don't know anything about scaling stuff so I wanted to do auto scaling and this app.yaml is basically default settings.
Never mind. I had my quota increased to 6 and it still errors out.
I'm not sure what's causing this, but I was able to resolve this same issue by setting scaling to manual or by setting max_num_instances below 8 in my app.yaml:
automatic_scaling:
min_num_instances: 1
max_num_instances: 7
Google Cloud restricts resources by Quota, see:
https://cloud.google.com/docs/quota
For many billing accounts, you are limited to 8 Compute Engine instances (VMs) per project. You can request quota increases through the console:
https://console.cloud.google.com/iam-admin/quotas
App Engine Flex uses Compute Engine VMs.. There are 2 ways you may be consuming these:
Multiple deployed versions running concurrently
Multiple instances per deployed version.
You should be able to determine this via the App Engine dashboard and|or by listing the Compute Engine instances.
https://console.cloud.google.com/compute/instances
Please add your app.yaml to your question as this would be insightful.
You can make authenticated requests to Google Cloud services by acquiring an access token using gcloud and including this in a request header e.g.
curl \
--request GET \
--header "Authorization: Bearer $(gcloud auth print-access-token)" \
https://compute.googleapis.com/compute/beta/projects/primalexchange/regions/us-east1/autoscalers
I got ahold of google tech support. They checked my quota. They checked my code. He ended up telling me to switch regions after I did a verbrosity=debug of deploy.
He said the problems was a lack of vm's available for google flex engine in the region I was in, and that I should either switch regions or switch to the standard google app engine environment.
And yes. I have to delete my project for the 4th time and start over to do either.
LOL
At least I know it's not something I'm doing.
UPdate:
4th project also failed. This time tech support said he'll get back to me once the engineers look at it.
note: deploys fine under standard app engine env
Update1:
"Hi,
This issue has been reported to our App Engine product team and is currently being investigated.
As of the moment , there are no ETA yet for the resolution but we will provide an update as soon as we heard any progress from the team.
Thank you."
and someone else posted a temp soln https://issuetracker.google.com/issues/157161938#comment2
I'm just leaving it under app engine standard for the time being
Could someone help me access Big Query from an App Engine application ?
I have completed the following steps -
Created an App Engine project.
Installed google-api-client, oauth2client dependencies (etc) into /lib.
Enabled the Big Query API for the App Engine project via the cloud console.
Created some 'Application Default Credentials' (a 'Service Account Key') [JSON] and saved it/them to the root of the App Engine application.
Created a 'Big Query Service Resource' as per the following -
def get_bigquery_service():
from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials
credentials=GoogleCredentials.get_application_default()
bigquery_service=build('bigquery', 'v2', credentials=credentials)
return bigquery_service
Verified that the resource exists -
<googleapiclient.discovery.Resource object at 0x7fe758496090>
Tried to query the resource with the following (ProjectId is the short name of the App Engine application) -
bigquery=get_bigquery_service()
bigquery.tables().list(projectId=#{ProjectId},
datasetId=#{DatasetId}).execute()
Returns the following -
<HttpError 401 when requesting https://www.googleapis.com/bigquery/v2/projects/#{ProjectId}/datasets/#{DatasetId}/tables?alt=json returned "Invalid Credentials">
Any ideas as to steps I might have wrong or be missing here ? The whole auth process seems a nightmare, quite at odds with the App Engine/PaaS ease-of-use ethos :-(
Thank you.
OK so despite being a Google Cloud fan in general, this is definitely the worst thing I have been unfortunate enough to have to work on in a while. Poor/inconsistent/nonexistent documentation, complexity, bugs etc. Avoid if you can!
1) Ensure your App Engine 'Default Service Account' exists
https://console.cloud.google.com/apis/dashboard?project=XXX&duration=PTH1
You get the option to create the Default Service Account only if it doesn't already exist. If you've deleted it by accident you will need a new project; you can't recreate it.
How to recover Google App Engine's "default service account"
You should probably create the default set of JSON credentials, but you won't need to include them as part of your project.
You shouldn't need to create any other Service Accounts, for Big Query or otherwise.
2) Install google-api-python-client and apply fix
pip install -t lib google-api-python-client
Assuming this installs oath2client 3.0.x, then on testing you'll get the following complaint:
File "~/oauth2client/client.py", line 1392, in _get_well_known_file
default_config_dir = os.path.join(os.path.expanduser('~'),
File "/usr/lib/python2.7/posixpath.py", line 268, in expanduser
import pwd
File "~/google_appengine-1.9.40/google/appengine/tools/devappserver2/python/sandbox.py", line 963, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named pwd
which you can fix by changing ~/oauth2client/client.py [line 1392] from:
os.path.expanduser('~')
to:
os.env("HOME")
and adding the following to app.yaml:
env_variables:
HOME: '/tmp'
Ugly but works.
3) Download GCloud SDK and login from console
https://cloud.google.com/sdk/
gcloud auth login
The issue here is that App Engine's dev_appserver.py doesn't include any Big Query replication (natch); so when you're interacting with Big Query tables it's the production data you're playing with; you need to login to get access.
Obvious in retrospect, but poorly documented.
4) Enable Big Query API in App Engine console; create a Big Query ProjectID
https://console.cloud.google.com/apis/dashboard?project=XXX&duration=PTH1
https://bigquery.cloud.google.com/welcome/XXX
5) Test
from oauth2client.client import GoogleCredentials
credentials=GoogleCredentials.get_application_default()
from googleapiclient.discovery import build
bigquery=build('bigquery', 'v2', credentials=credentials)
print bigquery.datasets().list(projectId=#{ProjectId}).execute()
[or similar]
Good luck!
I am making my first Web App using Generator Angular Fullstack. I went through the project initialization here: https://github.com/DaftMonk/generator-angular-fullstack
During initialization I set up oAuth for the following: Facebook, Google+, Twitter
I am using Openshift as well, and after initializing the project... I used the steps to add it to openshift. This included setting up environment variables for RHC for Facebook, Google+ and Twitter authentication. I added these as well.
However, with my new app... I cannot create a new account with Facebook, Google+ or Twitter.
When I create new account these are the errors I get:
Facebook:
Invalid App ID: id
Google+:
401. That’s an error.
Error: invalid_client
The OAuth client was not found.
Request Details
scope=https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
response_type=code
redirect_uri=http://site.rhcloud.com/auth/google/callback
client_id=id
That’s all we know.
Twitter:
Internal Server Error
I haven't done much besides go through the tutorial so far. But I feel I missed something. Any help on this topic would be grea.t Thanks :)
Well, first of all you have to configure your clientIDs, clientSecrets and callBackURLs. These you will need to find on each oAuth provider's developer platform page. i.e. for Facebook this would be: https://developers.facebook.com/apps/
These can be placed in the local.env.js file (a sample is included, like this:
module.exports = {
DOMAIN: 'http://localhost:9000',
SESSION_SECRET: "xxxxxxxxxxxxx",
FACEBOOK_ID: 'xxxxxxx',
FACEBOOK_SECRET: 'xxxxxxx',
TWITTER_ID: 'xxxxxxx',
TWITTER_SECRET: 'xxxxxx',
GOOGLE_ID: 'xxxxxxx',
GOOGLE_SECRET: 'xxxxxxx',
.....
};
When deploying to Heroku don't forget to set the DOMAIN config variable with http/https prefix when using Google+ sign-in.
heroku config:set DOMAIN=http://<your app name>.herokuapp.com
Otherwise you will get a redirect_uri_mismatch. If this ends with an internal server error you probably need to enable the Google+ API in the Google developer console.
If you have obtained your provided ID and SECRET keys and you are still having difficulty with OpenShift as I did ... I found setting environment variables for the application did the trick for me.
Google for example:
rhc set-env -a GOOGLE_ID=
rhc set-env -a GOODLE_SECRET=
The module exports in local.env.js worked fine for me when I was developing locally but didn't when I pushed my app to OpenShift via grunt buildcontrol
I'm creating a GAE application.
When I set my GAE PHP application as an authorized application to access my
Cloud SQL instance, I get the following warning:
App Engine regions must be the same as Cloud SQL instance regions!
How can I verify or change the region of my GAE application?
Thx
You can use gcloud app describe --project <projectId> command to get the location.
You cannot change an app's region after you set it.
Refer here.
For example:
$ gcloud app describe --project myapp-1337
authDomain: gmail.com
codeBucket: staging.myapp-1337.appspot.com
defaultBucket: myapp-1337.appspot.com
defaultHostname: myapp-1337.appspot.com
featureSettings:
splitHealthChecks: true
gcrDomain: us.gcr.io
id: myapp-1337
locationId: us-central
name: apps/myapp-1337
servingStatus: SERVING
You can see the location of your application at [1].
Regarding changing the region, please see [2] for more information.
[1] - https://appengine.google.com/
[2] - Change GAE application location
Update: EU app creation is now possible from the new Developers console and doesn't require whitelist / premier status. Looks like Location tab will only show in the GAE console if account was whitelisted / Premier. A way to find app location is still in the old GAE console -> from the list click on the app to go to the dashboard -> if you see e~ in the link after app_id=, your app is in EU, else if you see s~ your app is in the US.
Another alternative is to use gcloud command suggested by Ilya Zakreuski below.
You can get the AppId from the runtime environment:
Java:
ApiProxy.getCurrentEnvironment().getAppId()
or Python:
os.environ['APPLICATION_ID']
where the prefixes as mentioned by #Ilya and #Nikita still apply:
prefixed with e~ means EU and s~ means US.
P.S. It looks like both the dev consoles have been updated to get the App ID from elsewhere, so they don't have this prefix.