My App engine is running in project-1. I want to access the BQ present in project-2. How can i make app engine in project-1 access the BQ present in project-2?
You should request "can view" (or "can edit") permissions on the dataset of your interest. Owner of project-2 (or respective dataset) will be able to do so.
You don't need to be present on project level and in some cases it is not even appropriate - but you must have appropriate permissions on dataset level
If, by chance, you are the owner of project-2 or respective dataset - you can easily do this by following below instructions
https://cloud.google.com/bigquery/bigquery-web-ui#sharedataset
The easiest way to accomplish this is to add the default service account of "project-1" to the permissions list of "project-2":
within the cloud console go to the permissions section of project-1
select the service accounts sub tab
look for the default service account (or create a new one)
add the service account to the permissions of project-2
EDIT
You need to create your client in a fashion that uses the applications default service account. For example if you're using python it would look something like:
# Grab the application's default credentials from the environment.
credentials = GoogleCredentials.get_application_default()
# Construct the service object for interacting with the BigQuery API.
bigquery_service = build('bigquery', 'v2', credentials=credentials)
Now with your PK file you can launch the dev appserver in a fashion that the same client client will work correctly: Unable to access BigQuery from local App Engine development server
Related
What is the best way to upload from AngularJS to S3 without passing the access key or secret down to the client? I have a friend who is trying to accomplish this by creating a pre-signed URL on the server (NodeJS & Express) and then sending the URL to the client for upload. He's claiming it's failing due to CORS but he CAN upload using the sdk from the same AngularJS app & browser.
Update for Clarity
The tricky part is they cannot store anything in an environment variable AND each bucket has different credentials. They're storing info on 10-20 buckets / regions / IAM users and need the end user to select the bucket BY NAME on the client. This means they cannot store anything globally. The server must be able to generate something for the client to use per each request.
His original question:
Generate S3 Put URL Per Request
You can do that by using an IAM Instance Profile for your ec2 server. That way you don't have to provide your access & secret key, because any call done via the AWS SDK will be authenticated via the instance profile permissions. Either of this, in order! authenticate aws api calls:
In system environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
In the Java system properties: aws.accessKeyId and aws.secretKey.
In the default credentials file (the location of this file
varies by platform).
In the instance profile credentials, which exist within the instance metadata associated with the IAM role for the EC2 instance.
Note the instance profile creds are checked last!
More info here
I've created several GAE applications but failed to retrieve "Service Account Name" - there is no gserviceaccount mail on a 'Application Settings' page. My goal is to add service accounts of those applications to a list of members of the main application with edit permissions.I've tried to add account to a list of members of main project just by following pattern <appId>#appspot.gserviceaccount.com but it failed with following error "The email account you invited is not a valid Google account".My second idea was to create service account member on non-main project first following same pattern it didn't lead to an error but it has a following status "Invitation sent. Waiting for response".Have no idea how to fix it. I would greatly appreciate any help.Just for your notice: main application has billing enabled and has generated service account. Non-main applications have default version deployed and marked as "running".
Services account are always created by default when you create a new project in the Google Cloud Platform.
So there's no need for you to add them or re-create them in your project.
You may find them in the Permissions page under the main project section in the Developers' Console.
Or by using the URL https://console.developers.google.com/project/YOUR_PROJECT_NAME/permissions and using your real project name.
Older apps/projects didn't create service accounts.
Add the service account by
Going to https://console.developers.google.com/project/YOUR_PROJECT_NAME/permissions
Press Add Member
Add this email YOUR_PROJECT_NAME#appspot.gserviceaccount.com
What is the recommended way to provide Bigquery access to end users?
We have a multitenant application for appengine where each company has its own namespace in Datastore. We can ingest data from these individual namespaces into BigQuery and run various reports.
We would like to let our users to be able to run queries in their respective BigQuery projects. What is the best way to achieve this?
Possible options:
user sends query to appengine, which passes it to his BigQuery project and returns reply
user sends query to BigQuery directly. The question here, how is the best to faciliate this?
some other way?
My favorite is "user sends query to BigQuery directly. The question here, how is the best to faciliate this?":
BigQuery supports sharing datasets with specific individuals. So you could load each table into its own dataset, and then share that dataset with the individual customers (or even with groups).
You can do this manually on the web UI or, via the REST API:
access[].domain A domain to grant access to. Any users signed in with the domain specified will be granted the specified access. Example: "example.com".
access[].groupByEmail A fully-qualified email address of a mailing list to grant access to. This must be either a Google Groups mailing list (ends in #googlegroups.com) or a group managed by an enterprise version of Google Groups.
access[].userByEmail A fully qualified email address of a user to grant access to. For example: fred#example.com.
access[].role READER, WRITER, OWNER
https://developers.google.com/bigquery/docs/reference/v2/datasets/update
I am trying to use the Google Prediction API for the first time.
I am just following the steps given in the article https://developers.google.com/appengine/articles/prediction_service_accounts.
I am getting a strange problem while executing step 2.4 in the above mentioned article.
I have followed the steps as below.
1) I have an application created in say xyz.com domain, and I have service account name of my application as "myapp#appspot.gserviceaccount.com".
2) Then I went to "Team" tab on the Google API Console, and tried to add the service account name of my application, to the project in which I have activated Prediction API and Google Cloud Storage.
While adding the serivce account to the project it gives me an error saying that
"Only users in domain xyz.com may be added to the project".
The same kind of message is also displayed on the bottom of the "Team" tab.
xyz.com is the domain in wich my application is deployed.
Could any one please help me understand why this kind of message is comming?
Are there any domain level admin settings required to add the service account to the Google Console API project?
Regards,
Nirzari
Currently, if you created a project with your Apps account, you can only add members of that same domain.
What you'll have to do is create a new project from something like xxx#gmail.com account (NOT your Apps domain account). You can then add both #appspot.gserviceaccount.com and yourself#xyz.com.
I think you can even remove xxx#gmail.com later on, once you've added yourself#xyz.com. Even activate billing for yourself#xyz.com, not xxx#gmail.com, if you need to.
Take from https://developers.google.com/appengine/docs/python/googlestorage/overview
You can modify the ACL of the bucket manually:
An alternate way to grant app access to a bucket is manually edit and set the bucket ACL and the default object ACL, using the gsutil utility:
Get the ACL for the bucket and save it to a file for editing: gsutil getacl gs://mybucket > myAcl.txt
Add the following Entry to the ACL file you just retrieved:
<Entry>
<Scope type="UserByEmail">
<EmailAddress>
your-application-id#appspot.gserviceaccount.com
</EmailAddress>
</Scope>
<Permission>
WRITE
</Permission>
</Entry>
If you are adding multiple apps to the ACL, repeat the above entry
for each app, changing only the email address to reflect each app's
service name.
Set the modified ACL on your bucket: gsutil setacl myAcl.txt gs://mybucket
I try to build an application in Google App Engine, but i have some question and don't know how to solve it.
I try to build two application, one is get a String from user, and other is process the String like divide or encrypt.
My question is how to transmit the String between two application in Google App Engine? And can I build an application just process the String, don't present the WEB page.
Any one can give me some tips? thanks a lot.
The only way for two AppEngine applications to communicate with one another is through the normal HTTP request/response model. For your case, we'll have App A, which answers requests from a User, who provides a string to be processed. App B will receive requests from App A, which passes along the string to be processed.
App A handles the URL /providestring?string=... where ... is some arbitrary value
App B handles the URL /processstring?string=... where ... is some arbitrary value
User uses a browse or writes an application that makes a call to /providestring
App A's URL-handling code runs and extracts the value of the string parameter
App A uses URLFetch to call App B's /processstring
App B's URL-handling code runs and extracts the value of the string parameter and does whatever kind of processing it does and
sends some sort of response to the caller.
One way to do this would be to have a shared database. App A would intake user data and save it to the database. App B would pull the user data and process it.
An Accepted feature request is for multiple App Engine applications to share datastore access [1].
Multiple App Engine applications can share access to Google Cloud SQL instances [2].
To grant access to your App Engine application:
From the Google Cloud SQL pane of the APIs Console, find the instance that you
want to grant access to and click the on the name of your instance.
Click on the Instance settings button on the instance's dashboard.
On the Instance settings window that appears, enter your Google App Engine
application ID under the Authorized applications section. You can grant access to
multiple applications, by entering them one at a time.
Note: you can achieve this with one app using the datastore. Here are a couple of accepted patterns:
A RequestHandler takes in user data, queues a task to process the data and save it.
A RequestHandler takes in user data & saves it. A cron job pulls all recent models of that type, processes them, and saves the processed data.
[1] https://code.google.com/p/googleappengine/issues/detail?id=1300
[2] https://developers.google.com/cloud-sql/docs/before_you_begin#configure_access_control