Google App Engine App failed to access Google Cloud Storage bucket - google-app-engine

Unable to access default Google Cloud Storage bucket from Appengine Project. This project was created with App engine SDK version prior to 1.9.0. I've created the bucket manually, as per GCS Documentation it was said by default the bucket is accessible to Appengine Projects, but its not accessible in my case. This is the code snippet that tries to create a file..
...
GcsService gcsService = GcsServiceFactory.createGcsService();
GcsFilename file = new GcsFilename(getGcsDefaultBucketName(), fileName);
GcsFileOptions.Builder builder = new GcsFileOptions.Builder();
GcsFileOptions options = builder.mimeType(mimeType).build();
GcsOutputChannel channel = gcsService.createOrReplace(file, options); //erroring in this line
...
Error found in Logs:
: com.google.appengine.tools.cloudstorage.NonRetriableException: java.lang.RuntimeException: Server replied with 403, verify ACLs are set correctly on the object and bucket: Request: POST https://storage.googleapis.com/1-ebilly.appspot.com/SERVICESTAGEREPORT-DEVICENAME-LYF2-CREATEDDATE-01012017-CREATEDDATE-19022017-.ZIP
: User-Agent: AppEngine-Java-GCS
: Content-Length: 0
: x-goog-resumable: start
: Content-Type: application/zip
:
: no content: Response: 403 with 212 bytes of content
: X-GUploader-UploadID: AEnB2Upq0Lhtfy5pbt06pVib8J0-L0XiGqW4JpB0G9PL87keY3WV7RCMVLCPeclD-D4UATEddvvwpAG2qeeIxUJx--brKxdQFw
: Content-Type: application/xml; charset=UTF-8
: Content-Length: 212
: Vary: Origin
: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Caller does not have storage.objects.create access to bucket myprojectID.appspot.com.</Details></Error>
:
: at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:120)
: at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:166)
: at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:156)
: at com.google.appengine.tools.cloudstorage.GcsServiceImpl.createOrReplace(GcsServiceImpl.java:70)
PS: I've tried to create a new Google Appengine Project and deployed the app init. This project is automatically created with a default GCS bucket and the same code is working fine without any error. My old project has lots of DB data which I want to retain and continue to use the same project without disposing it.
Please help with your thoughts to make the GCS bucket accessible in old project.

Resolved the issue by adding IAM permissin for appengine project. After reading the IAM "Access Control at the Project Level" document and comparing old project and new project permissions came to know that Appengine project level permission is not found in old project. After adding the permission the same code started to access the default bucket.
IAM Permissions before fix screenshot
IAM Permissions after fix screenshot

Try this:-
GcsService gcsService = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
GcsFileOptions options = new GcsFileOptions.Builder().mimeType(mime).build();
GcsFilename gcsfilename = new GcsFilename(BUCKET_NAME, fileName);
GcsOutputChannel outputChannel = gcsService.createOrReplace(gcsfilename, options);

Related

Permission denied during gcloud app deploy using Google Cloud SDK

It is insanely hard, to deploy an app to Google App Engine, using Google Cloud SDK.
I had tried the below 2 commands
C:\Users\yccheok\Desktop\jstock-android-appengine>gcloud config set project jstock-android
Updated property [core/project].
C:\Users\yccheok\Desktop\jstock-android-appengine>gcloud app deploy app.yaml --log-http --verbosity=debug
DEBUG: Running [gcloud.app.deploy] with arguments: [--log-http: "true", --verbosity: "debug", DEPLOYABLES:1: "['app.yaml']"]
DEBUG: No staging command found for runtime [python27] and environment [STANDARD].
DEBUG: API endpoint: [https://appengine.googleapis.com/], API version: [v1]
=======================
==== request start ====
uri: https://appengine.googleapis.com/v1/apps/jstock-android?alt=json
method: GET
== headers start ==
Authorization: Bearer ya29.GlxEBb1XVP1JK93-ARiaN_ZgiMbvZmw5KWfvJVfibDJ4FK_ZaMRoU1jVDTiWzsY606GSduJKJd9Nm8zA-_Iql5mGn4AMk4QVl8mPRycfekeZnOOHtbUvpkBMgOLOQA
accept: application/json
accept-encoding: gzip, deflate
content-length: 0
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/184.0.0 command/gcloud.app.deploy invocation-id/c9ae232d33b346d787b95a36e28c38c0 environment/None environment-version/None interactive/True python/2.7.13 (Windows NT 10.0.16299)
== headers end ==
== body start ==
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"
cache-control: private
content-length: 335
content-type: application/json; charset=UTF-8
date: Tue, 16 Jan 2018 19:16:21 GMT
server: ESF
status: 403
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
"error": {
"code": 403,
"message": "Operation not allowed",
"status": "PERMISSION_DENIED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.ResourceInfo",
"resourceType": "gae.api",
"description": "The \"appengine.applications.get\" permission is required."
}
]
}
}
-- body end --
total round trip time (request+response): 1.796 secs
---- response end ----
----------------------
DEBUG: (gcloud.app.deploy) Permissions error fetching application [apps/jstock-android]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 797, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 757, in Run
resources = command_instance.Run(args)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\surface\app\deploy.py", line 65, in Run
parallel_build=False)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 543, in RunDeploy
app = _PossiblyCreateApp(api_client, project)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 703, in _PossiblyCreateApp
api_client._FormatApp()))) # pylint: disable=protected-access
HttpException: Permissions error fetching application [apps/jstock-android]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
ERROR: (gcloud.app.deploy) Permissions error fetching application [apps/jstock-android]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
C:\Users\yccheok\Desktop\jstock-android-appengine>
Then, I went through https://cloud.google.com/appengine/docs/admin-api/accessing-the-api , it mentioned I need to use Admin API. So, I do it step by step carefully.
Step 1
Step 2
It mentions Admin API is enabled. Now I need credential.
Step 3
Step 4
OK. Now they mention I don't need create new credential. I can use Application Default Credentials ?!
Step 5
So, I went to https://developers.google.com/identity/protocols/application-default-credentials?hl=en_GB . I learn that I need to run
C:\Users\yccheok\Desktop\jstock-android-appengine>gcloud auth application-default login
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/auth?redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&prompt=select_account&response_type=code&client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform&access_type=offline
Credentials saved to file: [C:\Users\yccheok\AppData\Roaming\gcloud\application_default_credentials.json]
These credentials will be used by any library that requests
Application Default Credentials.
Step 6
Step 7
Still, after completing the above 7 steps, I still get the exact same error message, when trying to run
gcloud app deploy app.yaml --log-http --verbosity=debug
Can anyone let me know, what step I'm still require, in order to deploy my Python app to Google App Engine, using Google Cloud SDK?
The Admin API is for programmatically deploying the app, not for deploying using gcloud app deploy, for which you don't even need the Admin API enabled for your app.
From Deploying a Python App:
To programmatically deploy your apps, use the Admin API.
Before you begin
Before you can deploy your app:
The Owner of the GCP project must create the App Engine application.
Ensure that your user account includes the required privileges.
(but I can see how the above could be mis-intrepreted as an invitation to use the Admin API)
Most likely the account actually used by gcloud app deploy is missing or doesn't have the required permissions.
You can check the acount used with gcloud auth list. In my case the account is my email address, not a service account (I'm not sure if a service account can be used).
Use gcloud auth login (and maybe gcloud auth revoke) if you need a different account.
And you can check the account's privileges (if any) on the project/app on the IAM Page.
One additional note on this,
When you enable the App Engine API and the cloud builder api, make sure the Cloud Build Service Account also has access to the project.
I ran into that problem after enabling the correct apis.
This was using a build trigger. I could deploy locally from a command line because I was authenticated as myself. However, if you are using a build trigger, it will use the build service account, which needs access.
Hope this helps.
I had this issue. In my case it was solved by setting the project using the project ID, rather than the project name. See this answer
gcloud app deploy ERROR: Permissions error fetching application [apps/<PROJECT_NAME>]

gcloud app deploy give 400 / forbidden error / cannot push img to google container registry

I am trying to deploy .net core application from google compute vm to google app engine using gcloud app deploy. I get the following error
> WARNING: We couldn't validate that your project is ready to deploy to App Engine Flexible Environment. If deployment fails, please check the following mess
age and try again:
Server responded with code [400]:
Bad Request Unexpected HTTP status 400.
Failed Project Preparation (app_id='s~project-id'). Out of retries. Last error: Temporary error occurred while verifying project: TEMPORARY_ERROR: Unabl
e to check API status
Beginning deployment of service [default]...
WARNING: Deployment of App Engine Flexible Environment apps is currently in Beta
Building and pushing image for service [default]
Some files were skipped. Pass `--verbosity=info` to see which ones.
ERROR: (gcloud.app.deploy) Could not copy [/tmp/tmpLwvVOb/src.tgz] to [us.gcr.io/project-id/appengine/default.20170118t043919:latest]: HttpError accessing
<https://www.googleapis.com/resumable/upload/storage/v1/b/staging.project-id.appspot.com/o?uploadType=resumable&alt=json&name=us.gcr.io%2Fcasepro-v3%2Fappe
ngine%2Fdefault.20170118t043919%3Alatest>: response: <{'status': '403', 'content-length': '166', 'vary': 'Origin, X-Origin', 'server': 'UploadServer', 'x-g
uploader-uploadid': 'AEnB2UqprxH-2tIhsSZdGxDOtS8UnWSI29YTo4kaptNK67SWJpLVqR0zEtCAHgFyE64wj1HfCyUL5sy9z4AZkTRFYuxXfdw5TA', 'date': 'Wed, 18 Jan 2017 04:40:0
0 GMT', 'alt-svc': 'quic=":443"; ma=2592000; v="35,34"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"errors": [
{
"domain": "global",
"reason": "forbidden",
"message": "Forbidden"
}
],
"code": 403,
"message": "Forbidden"
}
}
>. Please retry.
I have already enabled billing api, app engine admin api and storage api. Service a/c that is being used has editor rights. VM instance has been created using cloud launcher for Jenkins Bitnami package. I am trying to deploy app from command line from the vm before I configure Jenkins to do the same.
What to do to resolve this?
The problem is that gcloud app deploy is trying to deploy to the project id 'project-id', which cannot be your project id.
Try setting the project like this:
gcloud config set project MY-PROJECT-ID
Then, retry the gcloud app deploy command.
If this fails, please reply with your full gcloud command line, and the results of these two commands:
gcloud config list
gcloud version

Cronjob on App Engine results in a Method Not Allowed error (405)

I am attempting to run a cronjob on App Engine with Endpoints.
I am receiving a Method Not Allowed (HTTP 405) error when I use the cronjob. How can I get around this error?
Here is my current cron.xml:
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
<cron>
<url>/_ah/spi/messaging/v1/checkUpdates</url>
<description>Check for news every 5 minutes</description>
<schedule>every 5 minutes</schedule>
</cron>
<cron>
<url>/_ah/spi/package.backend.MessagingEndpoint.checkUpdates</url>
<description>Check for news every 5 minutes</description>
<schedule>every 5 minutes</schedule>
</cron>
</cronentries>
My Endpoint without code body:
#ApiMethod(
name = "checkUpdatesPost",
httpMethod = ApiMethod.HttpMethod.POST
)
public void checkUpdatesPost() {
checkUpdates();
}
#ApiMethod(
name = "checkUpdates",
httpMethod = ApiMethod.HttpMethod.GET
)
public void checkUpdates() {
// ... Stuff
}
Running either of these functions through Google APIs Explorer runs correctly:
Testing with GET
GET https://myapp.appspot.com/_ah/api/messaging/v1/checkUpdates
Testing with POST
POST https://myapp.appspot.com/_ah/api/messaging/v1/checkUpdatesPost
Here is the log, as seen on the Logs Viewer:
You can see here AppEngine-. refers to an attempt by Cron. Chrome 48 is an attempt by my browser / Google APIs Explorer.
As I understand it, Cron does not work with POST. For this reason, I created another function for GET that mirrors the function of POST.
In my cron.xml I supplied two variations.
/_ah/spi/messaging/v1/checkUpdates - This is what I would use in a browser to access the resource.
/_ah/spi/package.backend.MessagingEndpoint.checkUpdates - This is what I see in the Logs Viewer once I have run #1 in the browser.
It appears to be clearly stated in the documentation.
Calling Google Cloud Endpoints
You cannot call a Google Cloud Endpoint from a cron job. Instead, you
should issue a request to a target that is served by a handler that's
specified in your app's configuration file or in a dispatch file. That
handler then calls the appropriate endpoint class and method.

Permission denied writing from App Engine to associated Cloud Storage bucket

We have a project that runs on App Engine and creates files on Cloud Storage. The two are connected as being part of the same cloud platform project.
In App Engine we have a "Google APIs Console Project Number", and in Cloud Console -> Credentials we have that project number listed under "Client ID" (1[..........].apps.googleusercontent.com) and "Email Address" (1[..........]#developer.gserviceaccount.com).
Every morning, we have some cron jobs that upload files to our Cloud Storage bucket. This has worked flawlessly since September 2013 but as of this morning (Oct 16, 2014) we're getting "permission denied" errors from Cloud Storage.
We're using the cloudstorage client library, which raises cloudstorage.ForbiddenError. Here's the log & exception output:
Expect status [201] from Google Storage. But got status 403.
Path: u'/bucketname/icon_20141016.png'.
Request headers: {'x-goog-resumable': 'start', 'x-goog-api-version': '2', 'content-type': 'image/png', 'accept-encoding': 'gzip, *'}.
Response headers: {'alternate-protocol': '443:quic,p=0.01', 'content-length': '151', 'via': 'HTTP/1.1 GWA', 'x-google-cache-control': 'remote-fetch', 'vary': 'Origin', 'server': 'UploadServer ("Built on Oct 9 2014 15:35:27 (1412894127)")', 'date': 'Thu, 16 Oct 2014 11:56:10 GMT', 'content-type': 'application/xml; charset=UTF-8'}.
Extra info: None.
Since we're using the Cloud platform connection between the two services, I feel like I can only diagnose the problem on my production App Engine instance. I would prefer not to deploy new versions and risk breaking a production server. This also appears to be a Cloud Storage issue this morning, but the only status page I could find says everything is working fine.
As #tx802 suggested, I checked the bucket ACLs carefully.
$ gsutil getacl gs://bucket
<Entry>
<Scope type="UserByEmail">
<EmailAddress>1[..........]#developer.gserviceaccount.com</EmailAddress>
</Scope>
<Permission>FULL_CONTROL</Permission>
</Entry>
I looked at the App Engine application settings and saw the service account is actually appname#appspot.gserviceaccount.com, so I gave that account full control:
$ gsutil chacl -u appname#appspot.gserviceaccount.com:FC gs://bucket
I'm not sure what changed since yesterday's cron run, but now it succeeds.

How does AppEngine(or its embedded-server) processes different urls

I created a simple appengine project that accesses a restful webservice hosted in http://commerce.qa.mycomp.com (mycomp should be replaced with my actual company name).
I am using the Jersey Client to make the client request. I do a POST request to the above url.
When I run the application locally, it always returns 404 Not Found response. For experimenting, I did a POST request to http://www.bbc.co.uk/news/ and this works fine, and returns 200 as the status.
I just decoupled my app from appengine and ran it in the separately-configured tomcat server, and there it works fine and returns 200 status code. I think appengine uses Jetty as the server. Does Jetty have any bugs processing urls like commerce.qa.mycomp.com. Why I am asking this is urls that start with www.any.com seems to be working fine.
The two code snippets shown below is not working when run within appengine locally(not running even if I host it to appspot).
Client client = Client.create();
WebResource service = client.resource("http://commerce.qa.mycomp.com/rest");
ClientResponse response = service
.header("Content-Type", "text/xml; Charset=utf-8")
.header("Authorization", "Basic dwt3hkl553lsfsfssf3")
.post(ClientResponse.class, "does not need to be actual xml");
URL url = new URL("http://commerce.qa.mycomp.com/rest");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "text/xml; Charset=utf-8");
conn.addRequestProperty("Authorization", "Basic dwt3hkl553lsfsfssf3");
OutputStream os = conn.getOutputStream();
os.write("no need to be actual xml".getBytes());
os.flush();
System.out.println("Response Code: " + conn.getResponseCode());
But when run with tomcat, it just works.
My installations are:
Google App Engine Java SDK 1.6.1
Google Plugin for Eclipse 3.7
jersey-client-1.12, jersey-core-1.12, jersey-json-1.8
Please share thoughts.

Resources