I have a few services deployed in one App Engine flexible environment project. The service instances are running in the same VPC network (not default) which is specified in their app.yaml files and assigned IP addresses from the same subnetwork. By default both services are accessible from outside and I want to configure firewall rules to leave only the default service allowed for incoming traffic.
So as the first step I'm configuring a rule to deny all ingress for all instances in this VPC, with the idea of creating another rule with a higher priority to allow traffic to only one instance which is running the default service.
The problem is that when I create a firewall rule with priority 1001 to block all incoming traffic my default service still get requests. But if I specify the priority 1000 or bellow then the traffic is blocked and I get 502 Server Error.
The question is of course WHY? The VPC documentation states that there are only two default firewall rules used for all manually created VPC networks:
A default "allow egress" rule.
Allows all egress connections. Rule has a priority of 65535.
A default "deny ingress" rule.
Deny all ingress connection. Rule has a priority of 65535
Then how come that my 1001 priority rule fails to block the ingress and the exactly the same rule with priority 1000 (or below) works as expected? Are there any other default firewall rules for Flex instances or am I missing something else here?
From what I understand, both firewalls must allow the traffic in order for a request to reach your App Engine. If any one has a good reason to block the request, then the request is blocked.
You can know which firewall is blocking by looking at the response code :
A 502 -> VPC firewall
A 403 -> App Engine Firewall
If logs are enabled, you can know which rule on the VPC firewall is causing the deny.
Regarding the "1000 against 1001 priority" rule, that's a very strange (and undocumented) behavior. So let's document it here for the folks who might stumble on it :
Rules on the VPC network for flexible App Engine instances seems to
work only if priority is below 1000.
Related
I have a python (flask) application running on Google App Engine (flex); the application is protected by the GAE firewall where:
Default rule is 'Deny' all ingress
There is a whitelist of IP addresses from which traffic is allowed.
I have some microservices deployed on Cloud Run (fully managed) which:
Receive requests from the GAE app (e.g. for heavy duty tasks)
Send the results of whatever they process as http requests back to handlers/endpoints in the GAE app
Thus the GAE app is the main point of interaction with clients and a dispatcher of heavy tasks, while the processing of those tasks is carried out by the microservices. I have set up a static outbound IP address of the Cloud Run hosted service which verfiedly works and traffic is routed through the NAT gateway as required in the documentation. The respective NAT IP address is on the firewall whitelist.
The problem is that the firewall still does not let in the Cloud Run >>> GAE app requests which bounce back with 403 statuses (of course, if I change the default firewall rule to 'Allow', traffic goes through). If I host the same microservice in a docker container on a GCE VM with a static IP address like this everything works flawlessly. This makes me hypothesize that albeit Cloud Run outbound traffic is indeed routed through the static IP address when traffic is towards addressees outside GCP, when I try to ping an internal (project-wise) asset it still goes though some dynamically selected IP (i.e. the static IP solution simply does not work). Unfortunately the logs don't show the 403-ed attempt so I can't see from what IP addresses those request seem to come (from a GAE standpoint).
I would be very grateful for ideas how this can be fixed as it greatly diminishes the value of the otherwise wonderful idea to have static outbound IP addresses for Cloud Run.
First, thank you both for your help and suggestions, they are very helpful. I found the solution with some kind help from Google:
When the Cloud Run microservice and the GAE app are hosted in the same project traffic is still routed through internal channels and appears to come from IP address 0.0.0.0 which can be whitelisted (so it would work) as long as one considers this address encompasses GCP assets which are parts of other projects too (to the best of my understanding)
A more robust solution seems to be setting up an externally facing load balancer as described here and putting it in front of the GAE app; in such a case, Cloud Run will indeed consistently use its static outbound IP address as described in the documentation
You are correct saying that the static IP is not honoured when packets are routed internally to GCP.
I think this is what you want. You have to allow in the firewall one of the IPs mentioned there (not sure which one right now).
Just as you and #Ema mentioned, this is an expected behavior having in mind that the traffic from Cloud Run to App Engine is intern.
When you use Cloud Nat to send all traffic there, it does happen. If you create a container and ping, let's say to www.github.com. You will find that the traffic goes through the IP you set. On the other hand, if you ping to www.google.com, given that the traffic is intern, and the site to reach out is in the same infrastructure, the request doesn't even goes through public internet.
Additionally, just to keep in mind Static outbound IP address is still in Beta and it is not recommended to use Beta features/products in production environments.
As you mentioned and as it is stated in Allowing requests from your services:
Creating a rule for IP 0.0.0.0 will apply to all Compute Engine instances with Private Google Access enabled, not only the ones you own. Similarly, allowing requests from 0.1.0.40 or 10.0.0.1 will allow any App Engine app to make URL Fetch requests to your app.
This questions might be of your interest:
What are the outbound IP ranges for GCP managed Cloud Run?
Possible to get static IP address for Google Cloud Functions?
I'm using AWS EC2 to run a database that supports search capabilities - similar to Elasticsearch. The database is only running in a single AWS region due to budgetary constraints.
The database is also running inside of a private subnet in a VPC. Currently there are no inbound or outbound connections that it can make.
I need to allow access to the database so that only my serverless functions can connect to it via HTTP. Users should not be allowed to access it directly from the client-side. Using Lambda is possible but is far from ideal due to long cold start times. Users will expect search results to appear very quickly, especially when the page first loads. So something else is required.
The plan is to replace Lambda with Cloudflare Workers. With faster start times and closer distance to end users all over the world, connecting to the database this way would probably give me the speed I need while still offering all the benefits of a serverless approach. However, I'm not sure how I can configure my VPC security group to allow connections only from a specific worker.
I know that my workers all have unique domains such as https://unique-example.myworkerdomain.com and they remain the same over time. So is there a way I can securly allow inbound connections from this domain while blocking everything else? Can/should this be done through configuring security groups, internet gateway, IAM role, something else entirely?
Thank you for any help and advice
There are a couple of options.
ECS
You can run an ECS cluster in the same VPC as your database, and run Fargate tasks, which have sub-second start times (maybe 100ms or less?). And you can run ECS tasks on hot cluster instances (but you then pay for them all the time), but perhaps a scale to/from zero approach with ECS would allow you to manage cost without compromising on most of user requests (the first request after a scale-to-zero event would get 100ms+ latency, but subsequent requests would get similar). Lambda actually does something similar to this under the hood, but with much more aggressive scale-down timelines. This doesn't restrict from a specific domain, but may solve your issue.
Self-Managed Proxy
Depending on how your database is accessed, you might be able to have a reverse proxy such as Nginx in a public subnet doing request validation to limit access to the database. This could control access by any request headers, but I'd recommend doing TLS client validation to ensure that only your functions can access the database through the proxy, and it might be possible to validate the domain this way (by limiting the trusted CA to an intermediate CA that only signs for that domain, alternatively, I think Nginx can allow a connection depending on traits of the client cert matches regexes such as domain name).
Route Through Your Corporate Network
Using a VPN, you can originate the function from within your network or somehow filter the request, then the database could still be in a private subnet with connectivity allowed from the corporate network through the VPN.
Use AWS WAF
You make a public ALB pointing at your database, and set up AWS WAF to block all requests that don't contain a specific header (such as an API key). Note: you may have to also set up Cloudfront, I forget off the top of my head whether you can apply WAF directly to an ELB or not. Also note: I don't particularly advise this, as I don't think WAF was designed with sensitive strings in the rules, so you may have to think about who has describerule / describewebacl permissions on WAF, also these rules may end up in logs because AWS doesn't expect the rules to be sensitive. But it might be possible for WAF to filter on something you find viable. I'm pretty sure you can filter on HTTP headers, but unless those headers are secret, anyone can connect by submitting a request with those headers. I don't think WAF can do client domain validation.
I have a GAE project (flexible) consisting of 1 default and 2 subservices:
foo.appspot.com
service1.foo.appspot.com
service2.foo.appspot.com
Now I want to use foo.appspot.com as API proxy & auth gateway to the internal services service1 and service2. The proxy itself I wrote and it is working fine.
I am struggling with adjusting the GAE Firewall to forbid incoming world traffic to service1 and service2 because I would like force an API user to send requests to foo.appspot.com. Traffic to the default service foo should be allowed.
It seems I can just enter IPs in the Firewall settings but not service names. The docs says that it should work but does not show how.
Thanks for the help!
App engine Flex environment is built on the Google Compute Engine and consequently, it supports the Virtual Private Cloud networking system.
With the VPC networks, you can configure firewall rules that would use Instance Tags to determine the target or source component in a firewall rule. Hence, you simply have to configure the app.yaml files of the target service/version to use the appropriate instance tags.
Google has recently added Firewall (beta) support for Google App Engine.
Is there a way to deny all external access but allow all internal GCP access, including GCP cloud functions running in the same project?
Whereas the Firewall allows you to allow or deny specific IP ranges, there doesn't seem to be a way to ascertain which IP ranges a function might be running from. And using the typical internal IP range and mask, e.g. 10.0.0.0/8 does not seem to allow access from GCP cloud functions.
The default rule is Allow from *. You can edit that rule and change it to Deny from * to close down all external access via the firewall.
Next, you're going to have to look up all of GCP's IP address blocks and add those into your Allow rules. The instructions for looking them all up are here.
There is an open issue logged for accessing via internal APIs that you can subscribe to follow.
I have many microservices in app engine only for internal use. But, by default, app engine opens service-project.appspot.com domain to public, and anyone can access them via http or https.
Is there a way to restrict access only for certain IP address?
The trivial way i can think of is checking source IP address in application code.
Or, I can create custom docker image with nginx configuration which checks source ip address. But, these are not quite clean solutions because access control is actually independent from application, and I don't want to hard code static IP address inside the container.
I assumed there is a way to setup firewall rule for app engine, but I could not find it. Identity-Aware Proxy seems like another option, but it is not available for app engine flex.
I know this is cold comfort, but we're working on re-enabling App Engine flex support for IAP. It's going to be more than just a few days, though.
https://cloud.google.com/appengine/docs/flexible/java/migrating#users has some options that might be more palatable than hardcoding IPs. You won't be able to use GCE firewall rules because the appspot.com traffic is coming through Cloud HTTP Load Balancer, so the GCE instance firewall only sees the IP of the load balancer. If you do want to verify IPs within your app, use X-Forwarded-For as described at https://cloud.google.com/compute/docs/load-balancing/http/#components .
Hope this helps! --Matthew, Cloud IAP engineer