I find it odd that it's common to host your DB miles away from your webserver, which makes little sense to me.
Also, the cost of serving data from your DBaaS to your PaaS web service and back, then from the web service to the end user, means you're paying more than double as much.
Ideally, I would want a PaaS that scales really well and can host my database and web in the same region at least, while also minimizing costs.
I see many articles on how to do this the AWS way, with VPC Peering. While GCP has VPC also, I have not figured out how I can enable Google App Engine to exist in the same private cloud as a GCP hosted MongoDB Atlas cluster.
Is it possible? If so, how?
If the answer is "just use AWS then", unfortunately AWS does not have the desired PaaS per say, other than another 3rd party like Heroku, that then won't do VPC Peering.
Edit:
VPC peering is now available for GCP and Azure as well.
Original post:
It is not possible to have Google App Engine and a GCP hosted MongoDB Atlas cluster in the same Virtual Private Cloud.
Since they are different projects each one is in a different Virtual Private Cloud. The only way for the two VPCs to communicate internally would be VPC Peering, which is not currently supported. So the only way for them to communicate is publicly, by adding the IP addresses of the GCP services in the whitelist of the Atlas project.
Atlas VPC Peering Documentation:
Atlas does not support VPC Peering for clusters deployed on Google
Cloud Platform. For Atlas clusters deployed on GCP add the IP addresses of your GCP services to Atlas project IP whitelist to grant those services access to the cluster.
Choose the region of your MongoDB Atlas cluster to be the same as your Google App Engine application. Having the services in the same region minimizes latency and network costs.
Related
We want to access some services on Google Kubernetes Engine from a Google App Engine standard via http(s). We don't want to expose the services to external networks because of security.
app engine flexible can use private network with an internal loadbalancer or vpc(post). How about app engine standard?
2 years ago, we could not do it. (post)
We can set rules about IP addresses, but an IP address for an app engine standard is not static.
What is the best solution?
Thanks for your help.
app engine flexible can use private network with an internal loadbalancer or vpc(post). How about app engine standard?
Not really as per this:
"A VPC network, sometimes just called a “network,” is a virtual version of a physical network, like a data center network. It provides connectivity for your Compute Engine virtual machine (VM) instances, Kubernetes Engine clusters, App Engine Flex instances, and other resources in your project."
We can set rules about IP addresses, but an IP address for an app engine standard is not static.
Yes. They are always changing. The recommendation here is to their Static IP range combined with:
Instead, we suggest that you take a defense in depth approach using OAuth and Certs.
You could also look at things like Cloud Armor (in alpha as of this writing), or a Virtual WAF. Longer term it would make more sense to move your workloads from Google App engine into GKE.
On google cloud I have created an app engine on one project and have some auto scaled(server spinup/spindown according to traffic) compute engines on different project. I have a task for allowing traffic to app engine only from those auto scaled compute engines.
Problem with the auto scaled environment we can't guess the server public IP, where as we can get private subnet IP's.
Please let me know how can I solve this issue.
For connecting through a private IP, you need your app engine and compute engine instances to be in the same network. App Engine standard environment is on a different network than the Compute Engine instances and don't have private IP access to each other currently.
You can't achieve that with app engine standard, you will need to consider app engine flex for this (see here how to set the network of your app engine flex instances).
You will have to connect using the external IP address of the service running on your Compute Engine instance by setting up your firewalls.
I achieved this by doing as follow:
Let's say you have Project A and Project B, and a service running in Project A that wants to connect to a compute engine running in Project B.
You need two VPC in A and B with no IP collision.
Create a VPC Network peering From A to B and from B to A.
Create a VPC serverless access connector in project A.
Config your service to use the new connector.
vpc_access_connector:
name: "projects/PROJECT_ID/locations/REGION/connectors/CONNECTOR_NAME"
We want to use an app engine flexible process to update our ElasticSearch index, which is on Google Kubernetes Engine. We need to connect to ElasticSearch via a http(s) address. What's the recommended way to do this? We don't want to expose the cluster to the external networks since we don't have authentication in front of it.
I've seen this SO post but both k8s and AE have changed a lot in the 2 years since the question/answer.
Thanks for your help!
The post you linked to was about App Engine Standard. App Engine Flex is built on top of the same Google Cloud networking that is used by Google Compute Engine virtual machines and Google Kubernetes Engine clusters. As long as you put the App Engine flex application into the same VPC as the Google Kubernetes Engine cluster you should be able to communicate between them using internal networking.
On the other hand, to expose a Kubernetes service to anything running outside of the cluster will require you to modify the service for Elastic search because by default Kubernetes services are only reachable from inside of the cluster (due to the way that the service IPs are allocated and reached via IPTables magic). You need to "expose" the service, but rather than exposing it to the internet via an external load balancer, you expose it to the VPC using an internal load balancer. See https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing.
in addition to Robert's answer, make sure that app engine and GKE are in the same region,
because Internal load balancers are not usable from other region.
check this https://issuetracker.google.com/issues/111021512
Is there a way to deploy "internal facing" applications in Google App Engine. AWS offers this capability as explained here and so does Azure as explained here.
What is the GCP equivalent for this? It appears App Engine Flexible Environment could be the answer but I could not find a clear documentation on whether Flexible Environment is indeed the way to host intranet facing applications. Is there someone from GCP who can advise?
Update
I tested the solution recommended by Dan recently. Listed below are my observations:
App Engine Flex allows deploying to a VPC and this allows VPN scenarios. The VPN scenarios however is for connections (originating) from App Engine to GCP VPCs or to other networks outside GCP which can be on-prem or in another cloud.
Access (destined) to the app itself from a GCP or another network is always routed via the internet facing Public IPs. There is no option to access the app at a private IP at the moment.
If there's another update, I will update it here.
Update 28Oct2021
Google has now launched Serverless Network Endpoint Group(NEG)s. With this users can connect AppEngine, Cloud Run & Cloud Function endpoints to a LoadBalancer. However at the moment, you can only use Serverless NEGs with an external HTTP(S) load balancer. You cannot use serverless NEGs with regional external HTTP(S) load balancers or with any other load balancer types. Google documentation for Serverless NEGs is available here.
I'm not sure this meets your requirements, but it's possible to set up an App Engine Standard application (not certain about Flexible) such that it is only accessible to users logged into your G-Suite domain. This is the approach I've used for internal-facing applications in the past, but it only applies if your case involves an entity using G-Suite.
You can set this up under the App Engine application Settings, under Identity Aware Proxy.
In this scenario the application is still operating at a publicly accessible location, but only users logged into your G-Suite domain can access it.
It should be possible with the GAE flexible environment. From Advanced network configuration:
You can segment your Compute Engine network into subnetworks. This
allows you to enable VPN scenarios, such as accessing databases within
your corporate network.
To enable subnetworks for your App Engine application:
Create a custom subnet network.
Add the network name and subnetwork name to your app.yaml file, as specified above.
To establish a VPN, create a gateway and a tunnel for a custom subnet network.
The standard env GAE doesn't offer access to the networking layer to achieve such goal.
I created a VM instance in Googl Compute Engine and app in Google App Engine standard environment. I was planning to have my app in App Engine and my database server in Compute Engine. But, I can't connect between the both by internal IP. Is this possible? the app and db are in the same zone (us-east1) but the connection by IP don't work, only work with external IP. Apparently, the rules of firewall are correct.
As of April 9, 2019, you can use the serverless VPC
connector.
This will allow your App Engine application to connect to other internal resources in your VPC network on the Google Cloud Platform, such as Compute Engine VM instances, Cloud Memorystore instances, and any other resources with an internal IP address.
To create a connector:
$ gcloud services enable vpcaccess.googleapis.com
$ gcloud beta compute networks vpc-access connectors create CONNECTOR_NAME \
--network VPC_NETWORK --region REGION --range IP_RANGE
$ gcloud beta compute networks vpc-access connectors describe CONNECTOR_NAME --region REGION
Note: You can see which IP ranges are currently reserved in the Google Cloud Platform Console. You can choose any unused CIDR /28 IP range to use for your connector, for example, 10.8.0.0/28.
$ gcloud beta compute networks vpc-access connectors create my-vpc-connector \
--region=us-central1 --range=10.8.0.0/28
Create request issued for: [my-vpc-connector]
Waiting for operation [xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx] to complete...done.
Created connector [my-vpc-connector]
To connect your connector to a service add this to your service's app.yaml file:
vpc_access_connector:
name: "projects/PROJECT_ID/locations/REGION/connectors/CONNECTOR_NAME"
Deploy the service:
$ gcloud beta app deploy
Note: To use Serverless VPC Access, make sure you use gcloud beta to deploy your service. You can get access to beta commands by running gcloud components install beta.
After you deploy your service, it is able to send requests to Internal IP addresses or DNS Names in order to access resources in your VPC Networks. In case any trouble please allow about one hour or more to let the connectors are propagated completely in the GCP Global Networks.
For connecting through a private IP you need that the app and DB will be in the same network.
You can't achieve that with app engine standard, you need app engine flex for this (see here how to set the network of your app engine flex instances)
You will have to connect using the external IP address of the service running on your Compute Engine instance. App Engine standard environment is on a different network than the Compute Engine instances and don't have private IP access to each other currently. Switching to an application (or service of an application) on the flexible environment might work for you if the costs associated make sense.