I have multiple apps which I would like to speak with each other. Every time I deploy to one, the new version has different IPs. How can I give them proper access to each other via firewall rules?
Check this documentation explaining how to let different services communicate with each other. Basically, since the deployed services run on its own domain, the idea is to issue HTTP requests to a handler in the other service. The service domains have this format:
http://[VERSION_ID].[SERVICE_ID].[MY_PROJECT_ID].appspot.com
Or:
https://[VERSION_ID]-dot-[SERVICE_ID]-dot-[MY_PROJECT_ID].appspot.com
For example, if I want to communicate to my service "website", to the handler "welcome", which is located in "my-project" I would send a request to:
http://website.my-project.appspot.com/welcome
To do so, you can use the request package in Node.js, for example.
Related
Our AppEngine app is connecting to a remote service which requires a VPN and also required me to add entries to the hosts file on my local machine in order to connect to their endpoints.
e.g.
10.200.30.150 foo.bar.com
This is working fine when running the app locally, but I can't figure out how to set this up on Google Cloud to work once deployed.
I can't use the IP addresses directly because it errors that the IP is not on the cert's list.
How do I map the host names to the IPs in Google Cloud so that AppEngine can use them?
From the error mentioned in the comment I suspect connecting directly through the IP fails because the certificate doesn't recognize the IP to DNS mapping as valid and therefore the secure connection setup breaks. Based on the requirements of connecting to the API by VPN and tweaking the hosts mapping there are few things you may try.
The simplest approach that may work would be using a Google Compute Engine VM instance, since there you would able to manipulate the etc/hosts file and replicate the local machine setup. This VM could be used either as the main app service or as a proxy from App Engine to the 3rd party API endpoint. To go that route I would suggest taking a look at these two posts which explain how to change the etc/hosts file on GCE (Changing the file once wouldn't work as the VM periodically overrides it, see the posts for cronjob like workaround).
Separately, as your app runs in App Engine flexible environment there is the chance to provide a docker container with the app packaged. It may be possible to set the workaround above in the docker file and have it working in App Engine too.
I have jobs and APIs hosted on cloud composer and App Engine that works fine. However for one of my job I would need to call an API that is IP restricted.
As far as I understand, I see that there's no way to have a fixed IP for app engine and cloud composer workers and I don't know what is the best solution then.
I thought about creating a GCE with a fixed IP that would be switched on/off by the cloud composer or app engine and then the API call would be executed by the startup-script. However, it restrains this to only asynchronous tasks and it seems to add a non desired step.
I have been told that it is possible to set up a proxy but I don't know how to do it and I did not find comprehensive docs about it.
Would you have advice for this use-case ?
Thanks a lot for your help
It's probably out of scope to you, but you could whitelist the whole range of app engine ip by performing a lookup on _cloud-netblocks.googleusercontent.com
In this case you are whitelisting any app engine applications, so be sure this api has another kind of authorization and good security. More info on the App Engine KB.
What I would do is install or implement some kind of API proxy on GCE. It's a bummer to have a VM on 24/7 for this kind of task so you could also use an autoscaler to scale to 0 (not sure about this one).
As you have mentioned: you can set up a TCP or UDP proxy in GCE as a relay, and then send requests to the relay (which then forwards those requests to the IP-restricted host).
However, that might be somewhat brittle in some cases (and introduces a single point of failure). Therefore, another option you could consider is creating a private IP Cloud Composer environment, and then using Cloud NAT for public IP connectivity. That way, all requests from Airflow within Composer will look like they are originating from the IP address of the NAT gateway.
In Google App Engine, I have 3 services, 1 for front end, 2 for back end.
Is there a way to block http calls to my backend services for accounts not from my company's domain (and the service account of the front end), but allow everyone http access to my front end service?
I know there is the firewall option, but this is restricted to IP addresses, I would prefer user based
If it matters all services are python3
There's currently no option to filter traffic to specific App Engine services within a single application/project:
App Engine Firewall filters by source IP ranges but can only be set for the whole app, not per service.
Identity-Aware Proxy can filter access by user account as you'd prefer but also applies to the whole app. Also, it only supports user account and can't be used with service accounts.
One option you may have would be to split your app in 2 different projects. Keep the front-end in one project open to the world and restrict access to the backend services in your other project via firewall rules.
I have seen the following being used in task queues in GAE. Maybe it would help.
If u were using python 2, in standard environment, i think u could have used login handler element in app.yaml file.
You could have added following lines to your app.yaml file:
handlers:
- url: /.*
script: worker.app
login: admin
This prevents other users from accessing this service.
But the same login handler is not available for python3, according to Google Docs.
Just found following in Google Docs:
If a task performs sensitive operations (such as modifying data), you might want to secure the handler URL to prevent a malicious external user from calling it directly. You can prevent users from accessing task URLs by restricting access to App Engine administrators. Task requests themselves are issued by App Engine and can always target a restricted URL.
You can restrict a URL by adding the login: admin element to the handler configuration in your app.yaml file.
You can also call your backend services through cloud tasks or task queues (both are almost the same i guess), in case this only work for cloud tasks.
Find the code usage here:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/6f5f3bcb81779679a24e0964a6c57c0c7deabfac/appengine/standard/taskqueue/counter
Find details about handler here.
https://cloud.google.com/appengine/docs/standard/python/config/appref#handlers_element
Find details about Cloud task and queue here:
https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-handlers
I was unable to access my Backend Instance at custom domain.
For example, I have an app and I access the Normal Instance sucessfully at:
http://www.[my_app_id].appspot.com or http://[my_app_id].appspot.com
And I have a backend config name=test and I accessed Backend Instance successfully at:
http://test.[my_app_id].appspot.com
In admin interface, the "Instances" link show the instances of Backend and Normal Instance separately. The content show is the same, but is easy to see when a request go to the Backend Instance and when go to Normal Instance.
Then I configured the wildcard "test" in Google Apps to access my Backend Instance at a custom URL:
I continue access the Normal Instance sucessfully at:
http://www.[my_domain].com or http://[my_domain].com
But request at
http://test.[my_domain].com
redicted to the Normal Instance instead of Backend Instance.
The doc's said it should work but I cann't at this moment and I need uses custom domain because my app is multitenancy.
What I do wrong?
Your backed is really supposed to be accessed by the front end, as I understand it.
So when your application front end makes a request to it's back end (e.g. via a URL), it'll work as it's all done internally.
Have you set your back end to be publicly accessible?
https://developers.google.com/appengine/docs/python/backends/overview#Public_and_Private_Backends
Backends are private by default, since they typically function as a component inside an application, rather than acting as its public face. Private backends can be accessed by application administrators, instances of the application, and by App Engine APIs and services (such as Task Queue tasks and Cron jobs) without any special configuration. Backends are not primarily intended for user-facing traffic, but you can make a backend public for testing or for interacting with an external system.
I don't know why the redirection is not working, but perhaps you should modify your question to show what problem it is you are trying to solve here and get an answer to that instead?
Google App Engine currently limits you to 2,000 emails per day (for free) via their API.
I am trying to find a definitive answer if it is possible to use a third-party system if you need to send more. I know that they disallow raw sockets, so I would assume that there might be trouble with this approach... but surely I'm not the first to see it.
Worst case, I can build a simple offsite web service that my GAE can call... but I'd much rather just be able to send directly through an SMTP server.
Thanks!
Nope.
You're correct: you cannot make raw socket requests, nor any other direct outbound requests except through the urlfetch API. To talk to an external SMTP server, you would need to use a webservice as a proxy.
We use the Postmark mail outsourcing service via the hutools.postmark API. Since the communication is HTTP based, it works like a charm on Google AppEngine. This might be an option for you, although it is also a for-pay service. We use it to get arround GAEs sender restrictions.
I've successfully used third party providers for email services with Google App Engine. I've used both SendGrid and MailGun using their HTTP-API.