I want to deploy some modules on App Engine. For now I let them "talk" to each other by a simple REST api.
I wonder if is there any kind of "local address" to use instead of *.appspot.com public domain?
If nothing available, what is the fasted protocal/method to communicate between two modules not including sharing the same database and memcache?
The only way to communicate between modules is via HTTP requests, synchronously via URL Fetch API or async via Push Queue API, which can only be done via *.appspot.com URLs. But this are always resolved to local IP address so inter-module communication always goes through internal AppEngine network.
Also, the official docs about module communication uses ModuleService API which resolves module addresses to *.appspot.com addresses, so this is an official google way of addressing modules.
You can share data between modules via datasore/memcache but I don't consider this communication as it does not actively notify receiving party about the data.
Related
I am having trouble understanding how to use a microservices model. The idea of a microservice is that I have multiple local servers, each serving a different port. Connecting to these local servers can be easily done locally (e.g., using an Express hosted website). But if I am using a frontend application, such as React, how am I supposed to call the different APIs.
The only solution I can seem to think is to create a subdomain per API, but this seems far-fetched and impractical since I would need to create a lot of entries inside the Names Server (e.g., Cloudflare).
If I am using an application like Apache or Nginx, is there a way to publicly access the APIs using a single domain? Or using subsubdomains such as api1.subdomain.domain.com, api2.subdomain.domain.com ... but without adding each of these subdomains to the name server?
An alternative I can think of is creating a public API whose job is to connect to local services, but this seems to defeat the purpose of microservices.
I can't find anything online and all tutorials always use localhost which does not work in production code.
Thanks in advance!
You should research API Gateways / Edge-Services.
Personally, I like hosting the containers for microservices in Kubernetes and forwarding all traffic to *.mydomain.tld to the kubernetes cluster and configuring the load balancing (in this case: which subdomain should be routed to which service) there.
I have an App Engine Standard (1st Gen) Python app, set up with serverless VPC access. I use requests 2.3 (I have issues with later versions on GAE) to do HTTP requests.
When I try doing an HTTP request to an internal IP address (10.x), it refuses to connect. However, serverless VPC access is working properly: I tested connecting to the same IP address on the same port using a non-HTTP client library (e.g. redis) from App Engine, and this works fine. It's just HTTP requests that fail.
I suspect the URL Fetch service is trying to do the HTTP requests, and fails on internal addresses. If so, is there a way to use requests without triggering the URL Fetch service? Or do you have any other clue what might be going wrong?
I think I found the answer by looking at the SDK source code:
from python_std_lib import httplib
gets you an HTTPLib that uses sockets, without having to override the global httplib (causing sockets to be used everywhere).
I have jobs and APIs hosted on cloud composer and App Engine that works fine. However for one of my job I would need to call an API that is IP restricted.
As far as I understand, I see that there's no way to have a fixed IP for app engine and cloud composer workers and I don't know what is the best solution then.
I thought about creating a GCE with a fixed IP that would be switched on/off by the cloud composer or app engine and then the API call would be executed by the startup-script. However, it restrains this to only asynchronous tasks and it seems to add a non desired step.
I have been told that it is possible to set up a proxy but I don't know how to do it and I did not find comprehensive docs about it.
Would you have advice for this use-case ?
Thanks a lot for your help
It's probably out of scope to you, but you could whitelist the whole range of app engine ip by performing a lookup on _cloud-netblocks.googleusercontent.com
In this case you are whitelisting any app engine applications, so be sure this api has another kind of authorization and good security. More info on the App Engine KB.
What I would do is install or implement some kind of API proxy on GCE. It's a bummer to have a VM on 24/7 for this kind of task so you could also use an autoscaler to scale to 0 (not sure about this one).
As you have mentioned: you can set up a TCP or UDP proxy in GCE as a relay, and then send requests to the relay (which then forwards those requests to the IP-restricted host).
However, that might be somewhat brittle in some cases (and introduces a single point of failure). Therefore, another option you could consider is creating a private IP Cloud Composer environment, and then using Cloud NAT for public IP connectivity. That way, all requests from Airflow within Composer will look like they are originating from the IP address of the NAT gateway.
I have multiple apps which I would like to speak with each other. Every time I deploy to one, the new version has different IPs. How can I give them proper access to each other via firewall rules?
Check this documentation explaining how to let different services communicate with each other. Basically, since the deployed services run on its own domain, the idea is to issue HTTP requests to a handler in the other service. The service domains have this format:
http://[VERSION_ID].[SERVICE_ID].[MY_PROJECT_ID].appspot.com
Or:
https://[VERSION_ID]-dot-[SERVICE_ID]-dot-[MY_PROJECT_ID].appspot.com
For example, if I want to communicate to my service "website", to the handler "welcome", which is located in "my-project" I would send a request to:
http://website.my-project.appspot.com/welcome
To do so, you can use the request package in Node.js, for example.
I have some appengine modules in my project.
I am building a "Cloud Endpoints" that will works like a API Gateway. Both in them same project.
The endpoints will receive a request and forward to another appengine module, so, when the module process the request, the endpoints will return the response to frontend.
The main reponsibility this API Gateway will be validate permissions and log informations.
The frontend sends: GET,PUT and POST methods.
I read about URLFetch to do it.
I would like to know, Is it fast to use URLFetch to to do it?
Should I use other framework to to id?
If you're on App Engine, external requests should use URL fetch regardless of if you use it directly or use your language level networking primitives. It should be relatively fast, though you should benchmark this for yourself to see if it's an acceptable latency.