I want to access datastore (and storage) data of an AppEngine project via google-cloud-datastore and google-cloud-storage with an Python program on my own server.
This works with my AppEngine staging server, by creating a service account and giving it owner access (to the project).
Doing the same thing with the production AppEngine instance fails with
google.api_core.exceptions.PermissionDenied: 403 Missing or insufficient permissions.
Part of the problem might be, that I might be using the wrong project to create the service account with. There are more than one project with the same name in my cloud console. How do I identify the correct one?
How do I get more details about the problem?
First, note that the Datastore and the Cloud Storage are 2 different products with 2 different accessing methods.
The Datastore is closely tied to the GAE project - each project has its own datastore. The external access procedure in general is captured in How do I use Google datastore for my web app which is NOT hosted in google app engine?.
When switching the project (staging to production in your case) there are 2 things to keep in mind:
as you observed, you need to change the project you're accessing.
you also need to change the credentials you load and use for access to match the project you select, as each project has it own service account key configured in the above-mentioned procedure
For the google-cloud-datastore library both of these are simultaneously configured via the datastore.Client() call parameters (emphasis mine):
class google.cloud.datastore.client.Client(project=None,
namespace=None, credentials=None, _http=None, _use_grpc=None)
project (str) – (Optional) The project to pass to proxied API methods.
credentials (Credentials) – (Optional) The OAuth2 Credentials to use for this client. If not passed (and if no _http object is passed),
falls back to the default inferred from the environment.
The Cloud Storage is completely independent from GAE, the GAE project/credentials you use (if any) have no bearing on bucket/object access restrictions whatsoever. There's nothing you need to do from the google-cloud-storage library perspective when switching from one GAE project to another
To eliminate the confusion created by multiple projects having the same name just go to the IAM & admin Settings page, select the respective projects from the drop-down list on the top blue bar and rename them using meaningful names (click in the Project name box to edit the name, then click SAVE). Then re-check if you're using the right keys for the desired project.
Related
What is the best way of sharing a bucket between a multi-region App Engine app's projects?
I am enhancing an App Engine project to be multi-regional which means having many projects each set to one region. The existing app has per-user Buckets for uploading files which are created by the back-end as needed, then using resumable downloads / signed URLs etc for the uploads from the user. The permissions are all left as defaults, using the API storage.createBucket(bucketName);. The "App engine Flexible service account" has full access by default, and I've also granted read-access to another service account I made for a cloud function which processes the uploaded files.
Now I've come to need multiple projects accessing the same user-buckets (depending on how I balance the regions), so I need to add permissions for the additional region-projects' two service accounts to create and access these buckets.
I know I could grant access specifically (like this answer https://stackoverflow.com/a/20731155/209288) but as I will have N users, and R regions I would need to add N x R permissions - and need to back-fill in the new permissions each time a new region is added (which would not be often but will happen). I want new user buckets to instantly be able to be used from all regions, and new region-apps to be able to access all previously existing user-buckets.
So I was hoping there's some kind of group functionality I can use to grant group access to all the buckets on creation, then assign the region-app's services accounts into that group. Therefore the matrix of access is combined instantly using this group.
I found there is something for human users: https://cloud.google.com/storage/docs/collaboration#group but I'm not sure if this is appropriate for service accounts.
I looked at access conditions, but it seems to only affect the resources, not the "calling" users.
Another worry I have is that the existing buckets are all in one region/project (just the region I was using when the app was single region), and future buckets will also be created from which ever region/project the user happens to be being served from. If I shut down a region/project I don't want to lose those buckets.
I wondered if I should have a "host" project for these shared buckets, and perhaps a project-level setting in that project could unify the access from the other regional projects? (This is similar to Guillome's idea from this related question of mine)
I have tried one solution which seems fairly simple - using project-level permissions instead of bucket-level.
i.e. to add the "App Engine Default Service Account" from each regional project to the IAM permissions of the cloud storage project (be it the original region or if I did the host project idea).
In my region-project build script I add
gcloud projects add-iam-policy-binding $GCS_IMPORT_PROJECT_ID --member=\'serviceAccount:$PROJECT_ID#appspot.gserviceaccount.com\' --role='roles/storage.admin'
Where GCS_IMPORT_PROJECT_ID is the original or host project, and PROJECT_ID is the new region project being created.
The drawback of this is that it allows the other project to access all buckets, not just the user-upload buckets but infrastructure buckets like cloud-build artefacts, and my app's default and config bucket. This isn't a security issue as it's one app, but could possibly result in a mixup down the line.
I am hosting some Web Applications in Google Cloud Platform using App Engine and those are for internal purpose only. One month ago I got a mail from Google Cloud Team, saying one of my apps needs verification. By based on their response I did some research and finally migrated all apps to the Organisation level as they mentioned in documentation (below link for reference). https://support.google.com/googleapi/answer/7394288#gsuite-app
But, yesterday also I got another notification regarding the same.
May I ignore this notification, or are there any further steps I need to complete?
As stated in this other documentation page:
If you're creating an internal web app for which [...] your project is
associated with a Cloud Organization that your users belong to, you
don't need to go through verification. Internal users of your
application won't see the unverified app screen.
If your application will only be used by internal users belonging to the same organization as where your project is located, you can ignore this message. It was probably triggered by the fact that your application is indeed not verified (although you do not need to do so).
So if that is the case, you will only need, as stated in the link you shared, to create an Organization and then migrate your existing project to that organization (then make sure that the users who will be accessing the app belong to the same organization).
I am using a Custom Domain for a Google Application project. I have multiple projects and I use the same domain for each, although only one mapping is active at any time. Historically this has been as simple as Verifying the domain on the latest project and then Adding the domain. The domain has then automatically switched to the new project.
I have not used this approach for some months and when I tried it recently I got the messages
www.xxxxx.com is already mapped to a project.
xxxxx.com is already mapped to a project.
Research on StackOverflow suggests the use of the following command
gcloud beta app domain-mappings create xxxxx.com
This does look the right thing to do, unfortunately the response to the command is:
ERROR: (gcloud.beta.app.domain-mappings.create) App [aaaaaa] is the subject of a conflict: Domain 'xxxxx.com' is already mapped to another application. You must delete the existing domain mapping before you can re-map the domain, or you may specify 'DomainOverrideStrategy.OVERRIDE' on the request to force overwrite the existing mapping. Domain 'xxxxx.com' is currently mapped to an application on which you do not have permissions.
I do not want to use the delete approach since there will be a gap in service before the re-map. I would like to use the OVERRIDE option but I cannot work out how to add it to the gcloud command and I cannot locate any documentation.
Update 6Nov17.
In the absence of an answer I have used the delete approach and it worked as expected. The re-map was possible immediately after the mapping was deleted for the current app. Unfortunately for some users access to the web page was not possible, or error messages were returned, for a while. After about 10-15 minutes normal service was resumed. For my web site a gap of 10-15 minutes is manageable. This will not be true for many sites and I anticipate that Google Cloud will tidy up this procedure before it exits Beta.
If for some reason it is not possible to access the current app to delete the mapping then I guess deleting the app's subdomain information at the domain registrar will have the same effect, although it may be difficult to predict when the delete of the associated mapping will happen.
On the plus site the new automatic SSL provision worked flawlessly.
I believe there is an undocumented Google API available to create and manage Google Cloud Console (and App Engine) projects on behalf of third party users.
Does anyone know how to use it?
I think older versions of the Google Eclipse Plugin obtained an OAuth2 token in the (undocumented) scope https://www.googleapis.com/auth/appengine.admin, and this allowed it to generate a Cloud Console project on your behalf. The latest version doesn't seem to do this. App Engine's own appcfg.py also uses this scope, but doesn't seem to do much more than deploy the code - I'm looking to change core settings for the project, such as Name, Redirect URLs, and Web Origins.
Any information would be appreciated.
I maintain a WordPress plugin providing secure Google Apps Login for end users, and currently have to give detailed instructions to admins for creating a new Cloud Console project manually, and entering settings such as Redirect URL. Ideally, I would create a simple on-line service to do all of this for them.
Thank you!
It is possible to programmatically create a new Developer Console project on behalf of a Google Account (yes, you read that right). You do so in a very roundabout way:
Request the https://www.googleapis.com/auth/drive.scripts scope from the user (standard OAuth 2.0 flow).
Use the Drive API's drive.insert method to create a new file with a mimetype of application/vnd.google-apps.script.
Somehow try to get the project ID, maybe by uploading some Apps Script code? This is the part that I was never able to figure out.
A little known fact is that every Google Apps Script project has a hidden Developer Console project associated with it. This project is not shown in the list of projects, but it does exist. It is created automatically when the user starts a new Apps Script project, and the drive.insert method is enough to cause this to happen.
How do you get to the hidden project? Well, the only way I know of is to open the Apps Script project from the Drive website, open the "Resources > Advanced Google Services" dialog, and click the link to the Developer Console. You'll find the project ID in the URL.
Aside from not being shown in your list of projects and not being able to use App Engine, this is a normal Developer Console project. You can add additional OAuth client credentials, service accounts, Compute Engine instances, etc. And of course once you have a project ID, all of the various management APIs will work: creating new virtual machines, making use of a service account's impersonation ability, etc.
I want to create application with consists of Desktop application and google cloud storage. So, each my client should have separate cloud storage. Does google provide such thing?
More info.
Because I do not know what can offer google app engine i wrote this question.
I need some database hosting for my desktop application. In future I think I will switch to GWT and app engine. I want to sell my application so each my client can't access my other client databases. I was thinking that would be safer if each client will have data in a separate database so I can't do some mistakes in code.
You can separate data in the datastore using namespaces on google app engine:
https://developers.google.com/appengine/docs/java/multitenancy/multitenancy
It's up to you to decide how to implement the namespaces. You can separate them out by your user authentication system.
You can create a folder per client and restrict the folder access to the user (works only with Google Accounts) or your can do the same with buckets, create a bucket per user (which might be an overhead if you have a lot of users).
For database AppEngine datastore has the ability to separate the data by namespaces. this doen't require any user account and its your responsibility to select with which namespace to work with per request.
You can use GAE namespace capability as pointed above by #dragonx without Google authentication.
Use a client name as a namespace identifier (needs to be unique) . How you fetch this client name is upto you. It can be stored in GAE itself if you wish or can be deciphered from the url used specific to a client.
Do have a look at the GAE multitenancy link https://developers.google.com/appengine/docs/java/multitenancy/multitenancy
The example here can be easily adapted to use any string identifier per client.