I am following the steps here: https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-gcs.html and having trouble on step 6
I ran:
create storage integration my_integration
type = external_stage
storage_provider = gcs
enabled = TRUE
STORAGE_ALLOWED_LOCATIONS = ('gcs://<my-bucket>')
;
which completed successfully. Then DESC STORAGE INTEGRATION MY_INTEGRATION; successfully describes the storage integration and lists a STORAGE_GCP_SERVICE_ACCOUNT
However I cannot find this service account in the google cloud platform console that owns that bucket.
My snowflake account is on AWS though according to that tutorial page, I am allowed to use AWS or CGP for this integration.
Is there somewhere I should indicate in the create integration command which google cloud project I am referring to so that it knows where to create the service account?
And advice on performing this step?
This issue was because my bucket was not publicly accessible. Once I turned it to publicly accessible I was able to find that service account when adding roles.
It appears that the service account is not a service account "On" the google cloud platform account that hosts the bucket but rather one setup by snowflake on their own managed services.
So its like granting an external storage account permissions rather than an internal one.
Related
For using the google pub sub we need key.json file which serves as service account.
For deployed code we can pass the service_account: name#service-name.iam.gserviceaccount.com in app.yaml which works if everything is in google cloud.
As it is not recommend to download the service account file.
Is there a way where we can just pass the key name in local environment file along with individual credential who has access to service account in run time or any other way where we can run google cloud services in local enviornment without downloading the service key file?
Google makes this really hard to find, but gcloud auth application-default login might do what you need.
From that page:
Obtains user access credentials via a web flow and puts them in the
well-known location for Application Default Credentials (ADC).
This command is useful when you are developing code that would
normally use a service account but need to run the code in a local
development environment where it's easier to provide user credentials.
The credentials will apply to all API calls that make use of the
Application Default Credentials client library. Do not set the
GOOGLE_APPLICATION_CREDENTIALS environment variable if you want to use
the credentials generated by this command in your local development.
This command tries to find a quota project from gcloud's context and
write it to ADC so that Google client libraries can use it for billing
and quota. Alternatively, you can use the --client-id-file flag. In
this case, the project owning the client ID will be used for billing
and quota. You can create the client ID file at
https://console.cloud.google.com/apis/credentials.
Pub/Sub is really easy to use from my local work station. I set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path to my .json authentication object.
But what if you need to interact with multiple different Pub/Sub projects? This seems like an odd way to do authentication. Shouldn't there be a way to pass the .json object in from the Java code?
How can I use the client libraries without setting the system's environment variable?
Thanks
You can grant access for PubSub in different project using single Service Account and set it as env variable. See PubSub Access Control for more details.
A service account JSON key file allows to identify a service account on GCP. This service account is the equivalent of a user account but for the app (without user, only machine).
Thereby, if your app need to interact with several topic from different projects, you simply have to grant the service account email with the correct role on each topics/projects.
Another important thing. The service account key file are useful for app outside GCP. Else it's bad. Let me explain this. The service account key file is a secret file that you have to keep and store securely. And a file, it's easy to copy it, to send it by email and even to commit it into public git repository... In addition, it's recommended to rotate this key at least every 90 days for security reasons.
So, for all these reasons and the difficulty in security that represent the service account key files, I don't recommend you to use them.
With your local environment, use your user account (simply use the gcloud SDK and perform a gcloud auth application-default auth). Your aren't a machine (I hope!!)
With GCP component, use the component "identity" (load a service account when you deploy a service, create a VM,..., and grant the correct role on this service account without generating JSON key
With external app (other cloud provider, on premise, Apigee, CI/CD pipeline,...), generate a JSON file on your service account, you can't avoid them in this case.
I want to access datastore (and storage) data of an AppEngine project via google-cloud-datastore and google-cloud-storage with an Python program on my own server.
This works with my AppEngine staging server, by creating a service account and giving it owner access (to the project).
Doing the same thing with the production AppEngine instance fails with
google.api_core.exceptions.PermissionDenied: 403 Missing or insufficient permissions.
Part of the problem might be, that I might be using the wrong project to create the service account with. There are more than one project with the same name in my cloud console. How do I identify the correct one?
How do I get more details about the problem?
First, note that the Datastore and the Cloud Storage are 2 different products with 2 different accessing methods.
The Datastore is closely tied to the GAE project - each project has its own datastore. The external access procedure in general is captured in How do I use Google datastore for my web app which is NOT hosted in google app engine?.
When switching the project (staging to production in your case) there are 2 things to keep in mind:
as you observed, you need to change the project you're accessing.
you also need to change the credentials you load and use for access to match the project you select, as each project has it own service account key configured in the above-mentioned procedure
For the google-cloud-datastore library both of these are simultaneously configured via the datastore.Client() call parameters (emphasis mine):
class google.cloud.datastore.client.Client(project=None,
namespace=None, credentials=None, _http=None, _use_grpc=None)
project (str) – (Optional) The project to pass to proxied API methods.
credentials (Credentials) – (Optional) The OAuth2 Credentials to use for this client. If not passed (and if no _http object is passed),
falls back to the default inferred from the environment.
The Cloud Storage is completely independent from GAE, the GAE project/credentials you use (if any) have no bearing on bucket/object access restrictions whatsoever. There's nothing you need to do from the google-cloud-storage library perspective when switching from one GAE project to another
To eliminate the confusion created by multiple projects having the same name just go to the IAM & admin Settings page, select the respective projects from the drop-down list on the top blue bar and rename them using meaningful names (click in the Project name box to edit the name, then click SAVE). Then re-check if you're using the right keys for the desired project.
I've followed this tutorial: https://cloud.google.com/kms/docs/store-secrets
My next step was to have my apps read my secrets file from the storage bucket and decrypt safely at runtime. These apps run within various projects (staging, dev, production etc).
I've read the service account documentation over and over but can't quite understand the correct way forward.
What I found that worked was to simply add service accounts to MY_KMS_PROJECT and MY_STORAGE_PROJECT from the tutorial. I set them up to have access to read storage buckets and to decrypt KMS keys. Just by creating those service accounts, suddenly apps in other projects could read and decrypt. Is that how it's supposed to work?
I thought that I would have had to create a service account for each project that I want to have accessing the KMS projects from the tutorial? Or use IAM somehow to grant access? For example, how would I grant access to some apps within some projects and not others?
I'm attempting to now give access to apps when they are running in my local dev environment, which usually requires downloading a service account and pointing GOOGLE_APPLICATION_CREDENTIALS to the file. But it seems strange to me to download the service accounts from the MY_KMS_PROJECT or MY_STORAGE_PROJECT, especially since I already have a service account for accessing firebase. Are service accounts somehow global to all projects? Can they be combined? GOOGLE_APPLICATION_CREDENTIALS seems only good for pointing at a single service account.
Note: most of my apps are running on google app engine standard or flexible.
Here is the code from the apps within my projects that "just work" as described above:
client, err := google.DefaultClient(ctx, cloudkms.CloudPlatformScope)
if err != nil {
log.Fatal(err)
}
// Create the KMS client.
kmsService, err := cloudkms.New(client)
if err != nil {
log.Fatal(err)
}
....
And for accessing the bucket:
// Create the storage clientstorage
Client, err := storage.NewClient(ctx)
if err != nil {
log.Fatal(err)
}
....
For your first question :
This could be happening due to the fact that those applications use the same services accounts that you created for the respective projects. In this case you would need to ensure that those applications do not use the aforementioned service accounts.
For your second question:
You might try to use one service account at organization level . These service accounts would be inherited by all the projects in the organization. You can grant the service account those roles or permissions that meet your requirements
For your third question:
They are “Global” to all projects in case you use organization-level service accounts : you can take a look at this documentation where you might find valuable information about service accounts and organization
Please let me know your thoughts and how it goes.
This is what I discovered works. Until I hear that it's not correct or ideal, it's how I'm moving forward:
Follow the "storing secrets" tutorial (https://cloud.google.com/kms/docs/store-secrets).
Note that when creating the two accounts from the tutorial, there will be no service accounts automatically created since there is no compute or app engine instances by default.
Go to IAM for the kms storage project and add any other projects with apps that you want to have access to download the encrypted secrets. Add them by their service account emails, with the role for viewing buckets.
Go to IAM for the kms management project and do the same for any projects with apps that you want to be decrypting the shared secrets, via their service account emails.
For Localhost development, use "gcloud auth application-default login" before starting your services.
I have a GAE application with a database of users.
When one of the user tries to download, say, file myapplication.appspot.com/somefile.jpg, I would:
check on the GAE database whether he is allowed to
if he is allowed, redirect him to a cloud storage bucket of mine from where he can download somefile.jpg
if he is not allowed, return him a 404 error code, and do some magic so that directly trying to download somefile.jpg from the cloud storage bucket does not complete.
Now what’s unclear to me is how to control access to somefile.jpg. How can I restrict the download to this scope of users?
PS: using something else than Google Storage is not an option (for those of you guys who thought about blobstore).
You don't need to restrict access on a per user basic you can restrict access on a per application (Google App Engine App) basis.
Every application has a service account, what you can do is set an ACL on the bucket to allow access to the application service account.
Now all you need to write an handler that would access Google Storage and return the data to the user.
As Shay noted, every App Engine application automatically has associated with it an internal account, called the “service account”. Normally, the service account name follows the pattern “your-app-id#appspot.gserviceaccount.com”, however, you can confirm the exact name by visiting the App Engine Administration Console, then clicking on your app name, followed by the “Application Settings” link, at which point you should see your service account name.
Once you find your service account name, add it to the “Team” subpage on the APIs console with “Can edit” permissions. This is even easier than updating the bucket ACL because you don't have to change any ACLs, however, bear in mind this applies to all buckets in your project. If you'd like to restrict your app to only have access to a subset of the buckets owned by your project then you'll want to update the per-bucket ACL(s), as Shay proposed.