We plan to use Appengine for our Node.JS REST-API which will serve content for our customers. The data will come from a MOngoDB Cluster (or Replicaset), which will also be hosted on Google Cloud Services. Some of the contents might be cached using the Memcache of AppEngine.
Now my question. Is it neccessary to setup a kind of VLAN or similar, to connect to the DB server? How can we assure that the DB server is connected with a good bandwidth / latency to the AppEngine instances? Or is this no matter of concern at all?
Another question, does Google Cloud have a kind of internal DNS, which can be used? So for example that we have a domainname like "ourmongodb.internal" which can be used in connection string of our REST API? Or how is the most recent approach to connect to our DB server instance?
If you use app engine flex you can setup your nodejs app engine server and the GCE VMs that run the mogoDB cluster in the same network and than connect them both in the internal IP space.
Related
Use case: I am managing both GCP infrastructure and local infrastructure and am looking for a way for an app on Google App Engine to send data to/from a MySQL database on the local infrastructure, which is behind a VPN. I've set up and tested a Cloud VPN Gateway and VPC Connector that allows a Google Compute Engine instance to connect to the MySQL database and send and retrieve data.
Per this thread, and my own experimentation, the Google App Engine standard environment cannot currently connect to a local network via Google Cloud VPN directly. I've also been testing Cloud Function and Cloud Run to see if they can connect with the Cloud VPN, and it seems that they also have this limitation.
What I'd like to confirm is that only the Google App Engine flex environment OR a Google Compute Engine instance can connect through Cloud VPN. Google's documentation across all these resources doesn't ever outright say whether any of them can connect to Cloud VPN through a VPC Connector (just that they can connect to GCP networks via VPC Connector), so I'm hoping someone here can corroborate my testing. Additionally, is there any other GCP resource that can make use of this functionality that I've missed?
I have a App Engine Application written in Flask Python 3.7
My usecase is to get information from Composer Metadata DB (dag runs, dag success, dag failures etc) from Composer metadata DB and show as a dashboard inside App Engine Application (few charts).
Homework Done so far -
I was able to run sql queries on Composer metadata after logging in to one of the worker nodes (as worker nodes already have Cloud SQL proxy running which connects to Cloud SQL running in other container). This was done after creating a Compute engine under same VPC as of Composer and then doing ssh from Compute engine to one of the worker nodes.
Now the question is how to connect to Composer metadata DB that is under VPC from App Engine application using Cloud SQL Proxy ?
I would look at Serverless VPC connector, although it designed mainly for App Engine and Cloud Functions, you may consider implement this connector on a Flask app side that gives you opportunity to unify network between App engine and Compute engine nodes parties, thus you would be able to reach Cloud SQL proxy as well.
The setup is fairly much simple, though you just have to attach connector to the specific VPC network and region in the particular GCP project. The IP addresses pool must be in CIDR /28 range, reserved for a connector usage.
We have this discussion in our office and can not come to a conclusion. So I am reaching out here for some advice.
We have a Google Cloud SQL running with no public IP. Google App engine from different App Engine project connect to this single cloud SQL by authorizing their service account.
There are no VPC setup between the projects. The apps are on google app engine standard environment. The instance's private IP is not used in the app projects.
The connections between the projects are made using the tutorial found here
https://cloud.google.com/sql/docs/mysql/connect-app-engine
creating an connection string as
mysql+pymysql://<db_user>:<db_pass>#/<db_name>?unix_socket=/cloudsql/<cloud_sql_instance_name>
The question is how does the traffic flow from other App Engine projects to this Cloud SQL instance?
Does the connect handshake go via the internet (ie outside Google's Network) or does google handles the traffic and routes it internally without the request ever going to the internet?
It would be a great help if any one can help answer these questions.
The answer to this actually varies depending on which version of App Engine you are using.
On older versions of App Engine Standard, the /cloudsql/ unix socket connected over an internal network directly to your instance.
On more recent versions of App Engine Standard, it uses a version of the Cloud SQL proxy to authenticate your connection to the instance via it's public IP. This is why the Connecting from App Engine page states your Cloud SQL instance must have a public IP.
If you have configured your Cloud SQL to use a Private IP address then connectivity occurs using VPC Network Peering and your communication from your Google App Engine (running inside Google and VPC connected) to your managed Cloud SQL (running on a separated VPC Network) is all internal within Google using VPC.
Details on this can be found in the article here:
Introducing private networking connection for Cloud SQL
Private IP (MySql)
Configuring private IP connectivity (PostgreSQL)
The connection from the App Engine Standard to the Cloud SQL instance it is made over the internet. There are more internal services between the App Engine and the Cloud SQL, but the calls are not made to the private IP of the instance, by default.
If you look in the readme of the connector's repository you can see that you can use almost the same method to connect to the Cloud SQL instance from your local env. That might be a clue that things happen on the internet.
When trying to connect my Google App Engine to my Google Cloud SQL Instance (Second Generation), I cannot find the "...Authorized App Engine applications section..." (https://cloud.google.com/appengine/docs/php/cloud-sql/#PHP_Build_a_starter_application_and_database).
Am I just blind, or does this not exist anymore?
If it doesn't exist, how does one connect a Google App Engine to a Google Cloud SQL (Second Generation)?
Please review the limitations of Google Cloud SQL Second Generation.
Because Cloud SQL Second Generation instances are in beta, the following features are not available:
Service Level Agreement (SLA)
MySQL 5.5
MySQL 5.6 is supported.
Google App Engine connectivity. Connectivity is supported for other clients, including Compute Engine, Managed VMs, Container Engine, and your workstation.
....
I'd like to mention that although Google App Engine connectivity is not yet supported for the Cloud SQL Second Generation like the way is supported for Cloud SQL 1st Gen, however this doesn't mean that you cannot use Cloud SQL 2nd Gen with your App Engine applications.
You can use access control model which is described in this article as used for other applications. Since IP address of your App Engine application will be not a static address, you will need to authorize 0.0.0.0/0 IP range as an allowed network and use Allow only SSL connections feature of the Cloud SQL to allow only SSL connections. Configure SSL and generate keys and client certificate for your application and establish a secure connections using SSL.
Right now, App Engine cannot be used with CloudSQL Gen2. It should be possible once the CloudSQL Gen2 graduate to General Availability but right now, if you need to use it with App Engine, you'll need to stick with CloudSQL v1
I am currently test-driving Google Container Engine (GKE) and Kubernetes as a possible replacement to AWS/ElasticBeanstalk deployment. It was my understanding that just by the virtue of my dynamic servers being in the same project as the cloud sql instance, that they'd naturally be included in the firewall rules of that project. However, this appears not to be the case. My app servers and SQL server are in the same availability zone, and I have both ipv4 and ipv6 enabled on the sql server.
I don't want to statically assign IP Addresses to cluster members that are themselves ephemeral, so I'm looking for guidance on how I can properly enable SQL access to my docker-based app hosted inside GKE? As a stopgap, I've added the ephemeral IPs of the container cluster nodes and that has enabled me to use CloudSQL but I'd really like to have a more seamless way of handling this if my nodes somehow get a new ip address.
The current recommendations (SSL or HAProxy) are discussed in [1]. We are working on a client proxy that will use service accounts to authenticate to Cloud SQL.
[1] Is it possible to connect to Google Cloud SQL from a Google Managed VM?
Sadly, this is currently the only way to do this. A better option would be to write a controller that dynamically examined the managed instance group created by GKE and automatically updated the IP addresses in the Cloud SQL API. But I agree the integration should be more seamless.