I have a Google pub/sub subscription that pushes messages for a topic to an AppEngine standard service endpoint.
I want to restrict access to the AppEngine standard service to user IPs and still allow for messages coming from Google Pub/sub.
In the AppEngine firewall, the only option is to allow certain IP ranges.
What is the IP range(s) of Google pub/sub?
I've noticed that all the IP requests from Pub/Sub push subscriptions are coming from 2002:axx:xxxx::. As per IETF RFC 3056, 2002::: is an 6to4 range. 2002:axx:xxxx::, then, is translated to 10.XXX.XXX.XXX, which is an range reserved for the internal networking in this case used by Google.
Note: that other services apart from Pub/Sub might [at some point] use range 2002:a00::/24, 10.XXX.XXX.XXX. For example App Engine Flexible. If requests from Flexible are not describable in this particular project then you will have block it and give it a higher priority.
In the end your Firewall entries:
10 Allow 2002:a00::/24 Pub/Sub
default Deny * The default action.
Also, there is an issue created about it in Google's issue-tracker.
Related
I have a python (flask) application running on Google App Engine (flex); the application is protected by the GAE firewall where:
Default rule is 'Deny' all ingress
There is a whitelist of IP addresses from which traffic is allowed.
I have some microservices deployed on Cloud Run (fully managed) which:
Receive requests from the GAE app (e.g. for heavy duty tasks)
Send the results of whatever they process as http requests back to handlers/endpoints in the GAE app
Thus the GAE app is the main point of interaction with clients and a dispatcher of heavy tasks, while the processing of those tasks is carried out by the microservices. I have set up a static outbound IP address of the Cloud Run hosted service which verfiedly works and traffic is routed through the NAT gateway as required in the documentation. The respective NAT IP address is on the firewall whitelist.
The problem is that the firewall still does not let in the Cloud Run >>> GAE app requests which bounce back with 403 statuses (of course, if I change the default firewall rule to 'Allow', traffic goes through). If I host the same microservice in a docker container on a GCE VM with a static IP address like this everything works flawlessly. This makes me hypothesize that albeit Cloud Run outbound traffic is indeed routed through the static IP address when traffic is towards addressees outside GCP, when I try to ping an internal (project-wise) asset it still goes though some dynamically selected IP (i.e. the static IP solution simply does not work). Unfortunately the logs don't show the 403-ed attempt so I can't see from what IP addresses those request seem to come (from a GAE standpoint).
I would be very grateful for ideas how this can be fixed as it greatly diminishes the value of the otherwise wonderful idea to have static outbound IP addresses for Cloud Run.
First, thank you both for your help and suggestions, they are very helpful. I found the solution with some kind help from Google:
When the Cloud Run microservice and the GAE app are hosted in the same project traffic is still routed through internal channels and appears to come from IP address 0.0.0.0 which can be whitelisted (so it would work) as long as one considers this address encompasses GCP assets which are parts of other projects too (to the best of my understanding)
A more robust solution seems to be setting up an externally facing load balancer as described here and putting it in front of the GAE app; in such a case, Cloud Run will indeed consistently use its static outbound IP address as described in the documentation
You are correct saying that the static IP is not honoured when packets are routed internally to GCP.
I think this is what you want. You have to allow in the firewall one of the IPs mentioned there (not sure which one right now).
Just as you and #Ema mentioned, this is an expected behavior having in mind that the traffic from Cloud Run to App Engine is intern.
When you use Cloud Nat to send all traffic there, it does happen. If you create a container and ping, let's say to www.github.com. You will find that the traffic goes through the IP you set. On the other hand, if you ping to www.google.com, given that the traffic is intern, and the site to reach out is in the same infrastructure, the request doesn't even goes through public internet.
Additionally, just to keep in mind Static outbound IP address is still in Beta and it is not recommended to use Beta features/products in production environments.
As you mentioned and as it is stated in Allowing requests from your services:
Creating a rule for IP 0.0.0.0 will apply to all Compute Engine instances with Private Google Access enabled, not only the ones you own. Similarly, allowing requests from 0.1.0.40 or 10.0.0.1 will allow any App Engine app to make URL Fetch requests to your app.
This questions might be of your interest:
What are the outbound IP ranges for GCP managed Cloud Run?
Possible to get static IP address for Google Cloud Functions?
I'm fairly new to things that aren't strictly front end, so after reading the Google pub/sub docs and doing a few searches its not clear to me whether using it with react is possible.
My use case is I (hypothetically) have tens of thousands of people on my webpage at a time that all need to be told at the same time that some external event occurred (the message would be very small).
I know Google Firestore has a listener feature but based on this specification it would not be within the free tier usage anymore. I've seen libraries that allow Google Pub/Sub to be used with IOT devices so I'm confused on why I can't find any resources on using it in the browser.
Creating a Cloud Pub/Sub subscriber in the frontend would be an anti-pattern for several reasons. First of all, the quota limits only allow 10,000 subscriptions per topic. Since you say you have tens of thousands of people on the web page at a time, you would not be able to create enough subscriptions for this case. Additionally, subscriptions created when users come to the website would not be able to get any notifications from before the time the subscription was created; Cloud Pub/Sub only guarantees delivery of messages published after the subscription was successfully created. Finally, you'd have the issue of security and authentication. In order to start a subscriber from the client, you'd need to pass it credentials that it could use. If you use separate credentials for each webpage viewer, then you'd have to create these credentials on the fly and revoke them when the user disappears. If you use the same credentials across all of the subscribers, then one subscriber could intercept the feed of another subscriber.
Overall, Cloud Pub/Sub is designed for the torrents use case: fewer feeds with a lot of data that has to be processed by fewer subscribers. What you are talking about is the trickles use case: a small number of messages that need to be distributed among a large number of subscribers with individual ACLs. Firebase Cloud Messaging is the product designed for this latter case.
While it is true that Cloud Pub/Sub is on the path for Google Cloud IoT, it is used on the publish side: many devices send their events to a topic that can be processed by subscribers. Note that these messages from devices don't come directly into Cloud Pub/Sub; they go through a Cloud IoT server and that server is what publishes the messages to Cloud Pub/Sub. Device authentication is done via Cloud IoT and not via permissions on Cloud Pub/Sub topics. The delivery of messages to IoT devices is not done with Cloud Pub/Sub.
I am hitting this quota and I was wondering wether this quota for External IP?
INVALID_ARGUMENT: The following quotas were exceeded: IN_USE_ADDRESSES (quota: 8, used: 7 + needed: 2).
I have a few services that work via pub/sub and making request outside. Do I still need External IP? Or somehow I can set and use Internal IP?
Meantime I made a request to increase but want to understand this.
This quota is related to the ephemeral IPs used by App Engine. Actually App Engine will always use External IPs, you cannot avoid that, you may want to take a look on how App Engine manages IPs in this documentation.
Everytime you deploy a new version of your app, App Engine by default will retain the old versions with its IPs. You can avoid this situation by stopping the previous versions on deployment with the flag --stop-previous-version.
This is also already answered here.
I have my own domain with G-Suite Business and application in GAE that sending emails to our clients. But limit 100 sendings per day is too low.
How to increase mail send quota above 100/day? That's possible?
Thanks for any ideas.
The build in GAE mail is not really intended for production use, on top of the 100 mail limit there is also a limit on how many unique email addresses can be sent to in 1 day if im not mistaken.
For production I would recommend SendGrid, documentation can be found here : https://cloud.google.com/appengine/docs/standard/python/mail/sendgrid
Good service with a lot of needed mail features, like custom domain etc.
Google Cloud Platform support no longer provides quota increases for the Mail API.
Besides that, Richard's reply is correct so you can use Sendgrid or other available mail providers mentioned in the documentation. Additionally, as you mentioned GSuite, you may consider using Sockets API with your GSuite accounts (subject to GSuite quota).
Our site is hosted on Google App Engine which means we do not have a server farm with dedicated IP addresses that can be contacted “directly”. It is an elastic farm that spins up with load.
But our client is looking for static ips to route it through firewall and provide suitable access for this site:
"Proxy does not support any voice and video, hence I have to know all the Ip used for this portals so that I can route this via Firewall."
Is there anyway to list all the IP used for a site hosted on Google App engine?
regards,
Kanchan
The best you can achieve is this:
App Engine's current range of outgoing IP addresses are encoded in the
sender policy framework (SPF) record of
_cloud-netblocks.googleusercontent.com. You may need to recursively perform DNS SPF lookups to resolve the entire list of IP ranges. Start
by resolving _cloud-netblocks.googleusercontent.com
https://cloud.google.com/appengine/kb/general#static-ip
You could of course do it the other way round and only allow access to your application from a specific IP range.
Trying to keep track of GAE ip addresses is not a very good idea. They will most likely change over time without any announcements or heads ups.
You will be better of with a proxy. This project https://github.com/TellusTalk/Node_Proxy is one way to accomplish this.