Google Cloud Storage access denied and MapReduce. Cannot add Service Account into developers list - google-app-engine

I'm running a MapReduce job on Google App Engine with a configuration similar to this:
MapReduceSettings.Builder()
.setBucketName("my-bucket")
.setWorkerQueueName(QUEUE_NAME)
.setModule(MODULE)
.build();
The bucket is used for temporary data by App Engine itself.
The problem is that when I run the job, it fails with the following stacktrace:
com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.RuntimeException: Writeable Bucket 'my-bucket' test failed. See http://developers.google.com/appengine/docs/java/googlecloudstorageclient/activate for more information on how to setup Google Cloude storage.
at com.google.appengine.tools.mapreduce.MapReduceSettings.verifyAndSetBucketName(MapReduceSettings.java:134)
at com.google.appengine.tools.mapreduce.MapReduceSettings.<init>(MapReduceSettings.java:89)
at com.google.appengine.tools.mapreduce.MapReduceSettings.<init>(MapReduceSettings.java:31)
at com.google.appengine.tools.mapreduce.MapReduceSettings$Builder.build(MapReduceSettings.java:83)
at
...
...
Caused by: com.google.appengine.tools.cloudstorage.NonRetriableException: java.lang.RuntimeException: Server replied with 403, verify ACLs are set correctly on the object and bucket: Request: DELETE https://storage.googleapis.com/my-bucket/2f249469-c77a-4540-bbbd-45fcd27d7600.tmp
User-Agent: App Engine GCS Client
no content
Response: 403 with 111 bytes of content
Content-Type: application/xml; charset=UTF-8
Content-Length: 111
Vary: Origin
Date: Tue, 12 Aug 2014 18:20:20 GMT
Expires: Tue, 12 Aug 2014 18:20:20 GMT
Cache-Control: private, max-age=0
Server: UploadServer ("Built on Jul 31 2014 18:25:34 (1406856334)")
Alternate-Protocol: 443:quic
X-Google-Cache-Control: remote-fetch
Via: HTTP/1.1 GWA
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message></Error>
I already tried to add the Service Account Name (name-of-my-app#appspot.gserviceaccount.com) as a team member but beside the user it keeps saying
Invitation sent. Waiting for response.
How can I add the Service Account into the developers list so I can give it access to my bucket?
Thanks

I solved it using the gsutils command line tool:
gsutil acl ch -u name-of-my-app#appspot.gserviceaccount.com:WRITE gs://my-bucket
According to Google Storage documentation, sometimes it's not possible to add the service account into the developers list, even if they don't say why
Note: In some circumstances, you might not be able to add the service account as a team member. If you cannot add the service account, use the alternative method, bucket ACLs, as described next.

Related

401 Unauthorized on Dataverse Web API post method

I am using Dataverse via the Web API to access data from a server. This works very well in itself. All unlocked tables/entities can be accessed via the corresponding URL using GET method and also return valid results. That means the access and the authentication via security token works.
However, when I call the POST-Method to exactly the same table/entity, I get an error message back:
__checkpoint ⇢ 401 UNAUTHORIZED from POST https://ANONYM.crm4.dynamics.com/api/data/v9.2/
The role assigned to the application user actually has explicit write permissions to the table.
The post looks like this:
POST /api/data/v9.2/ANONYMtime-acquisitions HTTP/1.1
user-agent: ReactorNetty/1.1.2
host: ANONYM.crm4.dynamics.com
ACCEPT: */*
content-type: application/json
content-length: 282
{"cr2a0_id":0,"cr2a0_Abweichung":7.5,"cr2a0_Datum":"Sat Feb 01 00:00:00 CET 2020","cr2a0_Ist-
Arbeitszeit":7.5,"cr2a0_Mitarbeiter":"ANONYM","cr2a0_Pause":0. 0,"cr2a0_Soll-
Arbeitszeit":0.0,"cr2a0_Wochentag":"ANONYM","cr2a0_Beginn":"09:00","cr2a0_Ende":"16:30",
"cr2a0_Info":""}
I would have expected that the writing access also works. What else could be the reason for this?
I tried to reproduce the same in my environment and got the same error like below:
I created an Azure AD Application and added API permission:
I generated the auth-code by using below endpoint:
https://login.microsoftonline.com/TenantID/oauth2/v2.0/authorize?
&client_id=ClientID
&response_type=code
&redirect_uri=redirectUri
&response_mode=query
&scope=https://admin.services.crm.dynamics.com/user_impersonation
&state=12345
I generated the access token by using below parameters:
GET https://login.microsoftonline.com/TenantID/oauth2/v2.0/token
client_id:ClientID
client_secret:ClientSecret
scope:https://admin.services.crm.dynamics.com/user_impersonation
grant_type:authorization_code
redirect_uri:redirectURi
code:code
By using the above access token I am able to make the GET request successfully as below:
If still the issue occurs, make sure to assign Security role like below:
And check the version of the CRM like below:
To make a POST query, refer the below blog:
Dynamics CRM 365 WEB API Common Errors and Resolution – xrm CRM Dynamics by Bipin Kumar

Azure AD app with Application.ReadWrite.OwnedBy cannot add more owners to apps it created

I have an AAD app that creates other AAD apps. The first app (creator) has right Application.ReadWrite.OwnedBy and is able to update display names of the apps it created but it is not able to add more owners on those apps. Does it need more privileges to do that?
HTTP request updating the created-app display name (succeeds):
PATCH https://graph.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/applications/a1236923-6de6-4e78-87dd-494d621fc20c?api-version=1.6 HTTP/1.1
Authorization: Bearer eyJ0eXAi...
Content-Type: application/json; charset=utf-8
Host: graph.windows.net
Content-Length: 45
Expect: 100-continue
{
"displayName": "test"
}
HTTP request adding an owner to the created-app (fails):
POST https://graph.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/applications/a1236923-6de6-4e78-87dd-494d621fc20c/$links/owners?api-version=1.6 HTTP/1.1
Authorization: Bearer eyJ0eXAi...
Content-Type: application/json; charset=utf-8
Host: graph.windows.net
Content-Length: 122
Expect: 100-continue
{
"url": "https://graph.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/users/ac990eb3-b25a-4e06-ade5-41c7613693ff"
}
HTTP/1.1 403 Forbidden
request-id: 9be47e62-abcd-4768-926f-ffd62544e696
client-request-id: 149115f6-c9b7-4dd4-a267-711a40c51f23
...
{"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."}}}
I verified that the creator app does indeed have Application.ReadWrite.OwnedBy in the roles array of the JWT Bearer token.
I also verified that the creator app is present in the owner list of the created app by calling GET https://graph.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/applications/a1236923-6de6-4e78-87dd-494d621fc20c/owners/64898a3a-2fb1-45f0-a514-e83fabbc01f2.
In general, in order to add an owner, the calling app (and the calling user, in the case of delegated permissions, though in this case it's an app only) needs to have the permission to read the object of the owner being added, as well as the permission to read and write to the object which is being given a new owner.
Thus, in order for an app to add a User object as an owner to an Application object that the calling app is itself an owner of, the app needs (at least) Directory.Read.All (to read the new owner's User object), and Application.ReadWrite.OwnedBy (to be able to write to the Application object's list of owners).
According to your error message, you do not have enough privileges. As far as I known, if you want to assign user to AAD Application with Application privileges, you need to have these privileges : Application.ReadWrite.OwnedBy and Directory.Read.All, Application.ReadWrite.All and Directory.Read.All.
Besides, Microsoft strongly recommends that you use Microsoft Graph instead of Azure AD Graph API to access Azure Active Directory resources. For more details, please refer to the article. If you want to use Microsoft graph api to assign user to AAD Application, please refer to the document.

Issues authenticating OAuth with Azure bot Service

I recently wrote a c# bot using botframework (v4) that connects to users' account through oAuth (aadv2). I followed this article for the azure setup.
For most users, it's working perfectly fine but a small part of them can't get the magic code.
Instead they got a 404 after login in AD portal)
Here's the result of a curl against the auth url :
curl https://token.botframework.com/api/oauth/signin?signin=921d46120f2743bfe0e82b4c859898cda9bec041f8 -v -L
< Location: https://token.botframework.com/.auth/web/login/6cea70f5-7239-3769-57ce-9e90e9003dcc_aaaf9313-e069-86b0-6a34?redirect_uri=https%3a%2f%2ftoken.botframework.com%2fapi%2foauth%2fPostSignInCallback%3fsignin%3d921d46120f2743bfe0e82b4c859898cda9bec041f8
> GET /.auth/web/login/6cea70f5-7239-3769-57ce-9e90e9003dcc_aaaf9313-e069-86b0-6a34?redirect_uri=https%3a%2f%2ftoken.botframework.com%2fapi%2foauth%2fPostSignInCallback%3fsignin%3d921d46120f2743bfe0e82b4c859898cda9bec041f8 HTTP/1.1
> Host: token.botframework.com
< HTTP/1.1 404 Not Found
< Content-Length: 103
< Content-Type: text/html
< Server: Microsoft-IIS/10.0
< X-Content-Type-Options: nosniff
< Date: Thu, 13 Dec 2018 16:28:12 GMT
<
The resource you are looking for has been removed, had its name changed, or is temporarily unavailable.* Connection #0 to host token.botframework.com left intact
And as I said, it's working for most of the users perfectly fine so I don't think it's a configuration issue in the portal or in the bot...
I have this issue also. It started about one weeks ago.
No solution as yet - have raised it with MS Bot Framework Team.

Old QBOE SDK customer

We have a merchant who has been using our old SDK integration of QBOE for a number of years. Starting on 09-22-2014 he started receiving errors when doing a SignonAppCertRq query. It looks like it is failing when it tries to generate a response. Is there anyone at Intuit who can look into this? I've sanitized the merchants login, appid and ticket; I can email it to Intuit support if needed.
The QBOE Query
<?xml version="1.0"?><!DOCTYPE QBXML PUBLIC '-//INTUIT//DTD QBXML QBO 6.0//EN' 'http://apps.quickbooks.com/dtds/qbxmlops60.dtd'><QBXML><SignonMsgsRq><SignonAppCertRq><ClientDateTime>2014-09-22T15:22:06</ClientDateTime><ApplicationLogin>qboe.merchants-website.com</ApplicationLogin><ConnectionTicket>TGT-63-LbBOZNF...</ConnectionTicket><Language>English</Language><AppID>654479...</AppID><AppVer>1</AppVer></SignonAppCertRq></SignonMsgsRq></QBXML>
The Response from the Intuit Server
HTTP/1.1 500 Internal Server Error
Date: Mon, 22 Sep 2014 15:22:07 GMT
Server: Apache/1.3.41 (Unix) mod_ssl/2.8.31 OpenSSL/0.9.8r ApacheJServ/1.1.2
Content-Type: text/plain
Connection: close
General error building XML response.
Exception from other package:
org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified.
The qbXML SDK for QBOE was deprecated about a year ago, and has now been discontinued.
You need to port to the new v3 REST APIs.

Cannot GET Google Realtime API resource

I'm playing with the Google Drive API. I've managed to authenticate successfully and retrieve file listings and other things, but my goal is to write an alternative (if limited) editor for documents held in Google Drive. Unfortunately the downloadUrl referred to in the docs (https://developers.google.com/drive/v2/reference/files) does not exist, only exportLinks. It seems in order to deal with the native formatting of documents in Google Drive, it's necessary to use the realtime API.
Very well, I tried to retrieve the undocumented (but hinted at) realtime resource hinted at by the documentation, found at: https://www.googleapis.com/drive/v2/files/FILEID/realtime -
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=UTF-8
Date: Tue, 05 Aug 2014 03:02:51 GMT
Expires: Tue, 05 Aug 2014 03:02:51 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alternate-Protocol: 443:quic
Transfer-Encoding: chunked
{
"error": {
"errors": [
{
"domain": "global",
"reason": "lockedDomainCreationFailure",
"message": "The OAuth token was received in the query string, which this API forbids for response formats other than JSON or XML. If possible, try sending the OAuth token in the Authorization header instead."
}
],
"code": 400,
"message": "The OAuth token was received in the query string, which this API forbids for response formats other than JSON or XML. If possible, try sending the OAuth token in the Authorization header instead."
}
}
An odd error message. Authenticating with a header instead of an access_token parameter doesn't make it work. I used the API explorer found for the realtime resource (https://developers.google.com/drive/v2/reference/realtime/get) and it didn't work either. It gives the following message when used with the same file:
400 Bad Request
- Hide headers -
cache-control: private, max-age=0
content-encoding: gzip
content-length: 123
content-type: application/json; charset=UTF-8
date: Tue, 05 Aug 2014 03:14:42 GMT
expires: Tue, 05 Aug 2014 03:14:42 GMT
server: GSE
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "Invalid Value"
}
],
"code": 400,
"message": "Invalid Value"
}
}
I know the file is fine because the rest of the API calls I've made against it have worked (exploring the comments functionality and other things). It's frustrating issue I've run against, but hopefully someone can help. I would like to be able to edit collaborative documents outside of the canonical browser based editor in a non-destructive way!
Small update:
GET https://www.googleapis.com/drive/v2/files/1s8NArXPG0CWRHaA9HQ-zND086Uh5CoUFC2p3b3NI3Ek/realtime?key={YOUR_API_KEY}
Is the URL the API explorer (found here https://developers.google.com/drive/v2/reference/realtime/get) shows itself using to get the 400 error message above. As it turns out, I'd only set up an OAuth2 client ID - a "Simple API Key" is needed for the realtime API. A server side API key will not work either - it must be an API key for an Android application, iOS application or a browser based application. This is horrendously inconvenient because I'm not interested in browser based anything, but maybe there's some way to write a small hosted shim to make my dreams come true. Why must you do this, Google??!?
You cannot use the realtime API to access existing documents. It is for creating your own custom data, collaborative data models.

Resources