How to set Cloud Tasks dispatch when load balancer and GAE are combined - google-app-engine

I'm using Cloud Tasks from GAE now.
Also, by setting GAE as the backend of the load balancer, the following processing is tested.
batch-service is a service I created.
Request to /job/test_cron from local machine
go to Load balancer
go to GAE's service(batch-servise) from Load balancer
Create Cloud Task and request /job/test_task from GAE
go to GAE's service(batch-servise)
process and complete
I made each setting assuming the above flow, but the request when creating a task in GAE does not go to batch-servise, but goes to default service.
Therefore, the actual processing is as follows.
Request to /job/test_cron from local machine
go to Load balancer
go to GAE's servise(batch-servise) from Load balancer
Create Cloud Task and request /job/test_task from GAE
go to GAE's servise(default servise)
process and complete
GAE uses dispatch.yaml to direct all requests like /job/~ to batch-servise.
Therefore, Requesting /job/test_cron directly to GAE works as expected.
When using a load balancer, I think that dispatch.yaml cannot be used because the IP of GAE is not used. Is this correct?
Also, if anyone else knows how to configure GAE dispatch, it would be very helpful if you could tell me.

To override default service you can define AppEngineRouting which defines routing characteristics specific to App Engine - service, version, and instance.
You can refer this sample which routes to the default service's /log_payload endpoint. And update to this:
const task = {
appEngineHttpRequest: {
httpMethod: 'POST',
relativeUri: '/log_payload',
appEngineRouting: {
service: 'batch-servise'
}
},
};
When using a load balancer, I think that dispatch.yaml cannot be used because the IP of GAE is not used. Is this correct?
The load balancer does not interfere or interact with routing rules in your dispatch.yaml file. The dispatch.yaml rules are not evaluated until a serverless NEG directs traffic to App Engine.
Configuring dispatch.yaml:
The root element in the dispatch.yaml file is dispatch: and contains a list of routing definitions that are specified by the following subelements.
Dispatch rules are order dependent, and only the first rule that matches a URL will be applied.
You may have a look at these Examples
For more information, see How Requests are Routed.

Related

Google Cloud App Engine Standard - Custom Domain Name with Subdomains

I am struggling with the problem now for a couple of days. I hope someone can point out where I made the mistake.
I have a domain mydomain.com. (not purchased with Google.)
I have one AppEngine Standard Service (default) with my website.
Then I have 30 additional AppEngine Standard Services for my different api's (service1-service30).
When I browse to mydomain.com or www.mydomain.com I should be redirected to the "default" service.
When I browse to e.g., service25.mydomain.com I wanted to be redirected to the corresponding service. In this example service "service25"
What I did:
AppEngine Settings
DomainRegistrar
Dispatch.yaml:
dispatch:
- url: "mydomain.com/*"
service: default
- url: "www.mydomain.com/*"
service: default
Calling mydomain.com and www.mydomain.com are working as expected with an valid SSL certificate.
Calling service25.mydomain.com redirects to the "default" service
Calling service26.mydomain.com returns ERR_CONNECTION_CLOSED
Calling service27.mydomain.com returns DNS_PROBE_FINISHED_NXDOMAIN
Regarding Mapping Custom Domains and How Requests are Routed at least one of my previous methods should work and the default mapping/routing to the corresponding service name should be done.
If I add
- url: "service25.mydomain.com/*"
service: service25
it works, but due to the limitation of max 20 routes in the Dispatch.yaml file this is not working for me.
What did I do wrong?
Thank you very much in advance.

Meteor multiple instances on App Engine / meteor-files

I deployed a meteor app on App Engine on one instance, it works well.
However, when I want to scale on two instances, I sometimes got 401 on HTTP GET requests. Every call through websocket is successful with two instances.
More details:
I use meteor-files to handle upload and download.
When I download a file, the client makes an HTTP request (GET) to download the file from the server. In a method, I check this.userId (from the meteor) to compare it with the owner of the file (on mongoDb)
with one instance
when the user is authenticated, it always works: this.userId is always set
with two instances
when the client is authenticated with instance 1 AND the request is directed to instance 1 => OK
when the client is authenticated with instance 1 AND the request is directed to instance 2 => this.userId is null.
What I tried
In app.yaml:
network:
session_affinity: true
However when I check the config in the google app engine service, I got:
network:{}
It seems to be related to this bug: https://issuetracker.google.com/issues/154647126
My questions
How do you handle mutli instances with Meteor ?
How do you handle multi instances with Meteor in Google App Engine?
Thanks,

Service to service requests on App Engine with IAP

I'm using Google App Engine to host a couple of services (a NextJS SSR service and a backend API built on Express). I've setup my dispatch.yaml file to route /api/* requests to my API service and all other requests get routed to the default (NextJS) service.
dispatch:
- url: '*/api/*'
service: api
The problem: I've also turned on Identity-Aware Proxy for App Engine. When I try to make a GET request from my NextJS service to my API (server-side, via getServerSideProps) it triggers the IAP sign-in page again instead of hitting my API. I've tried out a few ideas to resolve this:
Forwarding all cookies in the API request
Setting the X-Requested-With header as mentioned here
Giving IAP-secured Web App User permissions to my App Engine default service account
But nothing seems to work. I've confirmed that turning off IAP for App Engine allows everything to function as expected. Any requests to the API from the frontend also work as expected. Is there a solution I'm missing or a workaround for this?
You need to perform a service to service call. That's no so simple and you have not really example for that. Anyway I tested (in Go) and it worked.
Firstly, based your development on the Cloud Run Service to Service documentation page.
You will have this piece of code in NodeJS sorry, I'm not a NodeJS developer and far least a NexJS developer, you will have to adapt
// Make sure to `npm install --save request-promise` or add the dependency to your package.json
const request = require('request-promise');
const receivingServiceURL = ...
// Set up metadata server request
// See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
const metadataServerTokenURL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
const tokenRequestOptions = {
uri: metadataServerTokenURL + receivingServiceURL,
headers: {
'Metadata-Flavor': 'Google'
}
};
// Fetch the token, then provide the token in the request to the receiving service
request(tokenRequestOptions)
.then((token) => {
return request(receivingServiceURL).auth(null, null, true, token)
})
.then((response) => {
res.status(200).send(response);
})
.catch((error) => {
res.status(400).send(error);
});
This example won't work because you need the correct audience. Here, the variable is receivingServiceURL. It's correct for Cloud Run (and Cloud Functions) but not for App Engine behind IAP. You need to use the Client ID of the OAuth2 credential named IAP-App-Engine-app
Ok, hard to understand what I'm talking about. So, go to the console, API & Services -> Creentials. From there, you have a OAuth2 Client ID section. copy the Client ID column of the line IAP-App-Engine-app, like that
Final point, be sure that your App Engine default service account has the authorization to access to IAP. And add it as IAP-secured Web App User. The service account has this format <PROJECT_ID>#appspot.gserviceaccount.com
Not really clear also. So, go to the IAP page (Security -> Identity Aware Proxy), click on the check box in front of App Engine and go the right side of the page, in the permission panel
In the same time, I can explain how to deactivate IAP on a specific service (as proposed by NoCommandLine). Just a remark: deactivate security when you have trouble with it is never a good idea!!
Technically, you can't deactive IAP on a service. But you can grant allUsers as IAP-secured Web App User on a specific service (instead of clicking on the checkbox of App Engine, click on the checkbox of a specific service). And like that, even with IAP you authorized all users to access to your service. it's an activation without checks in fact.

How can I call a Google Cloud Function from Google App Engine?

I have an App Engine project.
I also have a Google Cloud Function.
And I want to call that Google Cloud Function from the App Engine project. I just can't seem to get that to work.
Yes, if I make the function full public (i.e. set the Cloud Function to 'allow all traffic' and create a rule for 'allUsers' to allow calling the function) it works. But if I limit either of the two settings, it stops working immediately and I get 403's.
The App and Function are in the same project, so I would at least assume that setting the Function to 'allow internal traffic only' should work just fine, provided that I have a rule for 'allUsers' to allow calling the function.
How does that work? How does one generally call a (non-public) Google Cloud Function from Google App Engine?
You need an auth header for the ping to the function url. It should look like:
headers = {
....
'Authorization': 'Bearer some-long-hash-token'
}
Here is how to get the token:
import requests
token_response = requests.get(
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=' +
'https://[your zone]-[your app name].cloudfunctions.net/[your function name]',
headers={'Metadata-Flavor': 'Google'})
return token_response.content.decode("utf-8")
'Allow internal traffic only' does not work as expected. My App Engine app is in the same project as the Functions, and it does not work. I had to turn on 'Allow all traffic', and use the header method.
Example:
def get_access_token():
import requests
token_response = requests.get(
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=' +
'https://us-central1-my_app.cloudfunctions.net/my_function',
headers={'Metadata-Flavor': 'Google'})
return token_response.content.decode("utf-8")
def test():
url_string = f"https://us-central1-my_app.cloudfunctions.net/my_function?message=it%20worked"
access_token = get_access_token()
print(
requests.get(url_string, headers={'Authorization': f"Bearer {access_token}"}
)
As mentioned in the docs, Allow internal traffic only mentions the following:
Only requests from VPC networks in the same project or VPC Service Controls perimeter are allowed. All other requests are rejected.
Please note that since App Engine Standard is a serverless product, it is not part of the VPC and then the requests made from this product are not considered "Internal" calls, actually the calls are made from the Public IPs of the instances and for this reason you get an HTTP 403 error message.
Also using a VPC Serverless Connector won't work since this more a bridge to reach resources in the VPC (like VMs or Memorystore instances) but not a Cloud Function because this is also a Serverless product and it does not have an IP in the VPC.
I think here are three options:
Using App Engine Flex:
Since App Engine Flex uses VM instances, these instances will be part of the VPC and you'll reach the Function even when setting the "Allow internal traffic only" option.
Use a VM as a proxy:
You can create a VPC Serverless Connector and assign it to the app in App Engine. Then you can create a VM and reach the function using the VM as a proxy. This is not the best option because of the costs but at the end is an option.
The last option considers that the function can use the Allow All Traffic option:
You can set some security on the Cloud Function to only allow a particular Service Account and you can use this sample code to authenticate.
EDITED:
A good sample of the code for this option was shared by #gaefan in the other answer.
#GAEfan is correct.
As an addition: I used the official Google Auth library to give me the necessary headers.
const {GoogleAuth} = require('google-auth-library');
// Instead of specifying the type of client you'd like to use (JWT, OAuth2, etc)
// this library will automatically choose the right client based on the environment.
const googleCloudFunctionURL = 'https://europe-west1-project.cloudfunctions.net/function';
(async function() {
const auth = new GoogleAuth();
let googleCloudFunctionClient = await auth.getIdTokenClient(googleCloudFunctionURL);
console.log(await googleCloudFunctionClient.getRequestHeaders(googleCloudFunctionURL));
})();

How to secure connection between different GAEs?

For some reason I need to create two GAEs with project A and B:
A(flex env) is a proxy server bounded with Endpoint and restrict access with API key.
B(standard env) is real server which does real jobs.(B cannot apply Endpoint framework)
Client only awares proxy server address and send all requests to A
Now I would like to secure connection between A and B. In other words, B is only accessible from A. Is there any way to achieve it? (Firewall not work here because GAE has not static IP range.)
If you want to determine the identity of the App Engine app that is
making a request to your App Engine app, you can use the request
header X-Appengine-Inbound-Appid. This header is added to the request
by the URLFetch service and is not user modifiable, so it safely
indicates the requesting application's ID, if present.
In your application handler, you can check the incoming ID by reading
the X-Appengine-Inbound-Appid header and comparing it to a list of IDs
allowed to make requests.
Note: The X-Appengine-Inbound-Appid header is only set if the call is
made to the appspot.com domain. If the app has a custom domain, this
header will not be set.
https://cloud.google.com/appengine/docs/standard/go/appidentity/#asserting_identity_to_other_app_engine_apps
This should work the same for all App Engine standard environments.

Resources