Does anybody knows how can we use more than 12 concurrent connections from App Engine to the Cloud SQL?
We found that the documentation says:
"Each App Engine instance running in a standard environment cannot have more than 12 concurrent connections to a Google Cloud SQL instance."
We're using Java for App Engine using Hibernate... The connection pooling states that it shouldn't be used in production environment but if we disable the pooling we reach the max number of connections.
From Google Cloud FAQ: https://cloud.google.com/sql/faq#sizeqps
Google App Engine Limits
Requests from Google App Engine applications to Google Cloud SQL are subject to the following time and connection limits
For apps running in the Google App Engine standard environment, all database requests must finish within the HTTP request timer, around 60 seconds. For apps running in the flexible environment, all database requests must finish within 24 hours.
Offline requests like cron tasks have a time limit of 10 minutes.
Requests to Google Cloud SQL have limitations based on the scaling type of the App Engine module and how long an instance can remain in memory (residence).
Each App Engine instance running in a standard environment cannot have more than 12 concurrent connections to a Google Cloud SQL instance.
Related
I have a App Engine Application written in Flask Python 3.7
My usecase is to get information from Composer Metadata DB (dag runs, dag success, dag failures etc) from Composer metadata DB and show as a dashboard inside App Engine Application (few charts).
Homework Done so far -
I was able to run sql queries on Composer metadata after logging in to one of the worker nodes (as worker nodes already have Cloud SQL proxy running which connects to Cloud SQL running in other container). This was done after creating a Compute engine under same VPC as of Composer and then doing ssh from Compute engine to one of the worker nodes.
Now the question is how to connect to Composer metadata DB that is under VPC from App Engine application using Cloud SQL Proxy ?
I would look at Serverless VPC connector, although it designed mainly for App Engine and Cloud Functions, you may consider implement this connector on a Flask app side that gives you opportunity to unify network between App engine and Compute engine nodes parties, thus you would be able to reach Cloud SQL proxy as well.
The setup is fairly much simple, though you just have to attach connector to the specific VPC network and region in the particular GCP project. The IP addresses pool must be in CIDR /28 range, reserved for a connector usage.
I'm currently developing a project with python 3.7, Django 2.1, Mysql as database.
I'm deploying it in google cloud app engine standard environment and for the database I'm using a cloud SQL - MySql 2nd gen instance.
The application works well, however when I analyze the logs I see these errors:
"aborted connection - Got an error reading communication packets"
In this case the connection is being closed by my app (django).If I configure my app to have persistent connections and I put wait_timeout (i.e 60) in the config of the cloud sql, the error is:
"aborted connection - Got timeout reading communication packets".
I just determined that it's not a problem with sql cloud, or with the configuration of my application, but that it's an app engine problem. I came to this conclusion in the following way:
if I connect to the sql cloud instance through Mysql workbench, no connection is aborted
Similarly if I run my application on a local server, but connecting to cloud sql (through the cloud_sql_proxy), no error is generated and everything works perfect.
So my conclusion is that it is a problem of how the app engine connects to the cloud sql instance.
Why does this happen? How could it be solved?
The "Aborted connection" messages you're seeing, are usually triggered when a connection is closed improperly or there is a networking anomaly between the server and the client.
Sometimes Cloud SQL instances and GAE have long-live idle connections. In order to address this issue, it is recommended to set "wait_timeout' flag below 600 seconds - as you've already attempted to do so.
Another possible solution, is to implement application-level keepalives. SQLAlchemy provides “pre-ping” for this. Otherwise, generate activity on all open connections by sending a simple SQL statement such as "SELECT 1;" regularly, at least once every 5 minutes. Also consider using statements in your code like “with db.connect() as conn:” to control the connection’s lifetime.
I believe this is because requests from App Engine applications to Cloud SQL are subject to the following time and connection limits:
For apps running in the App Engine standard environment, all database requests must finish within the HTTP request timer, around 60 seconds. For apps running in the flexible environment, all database requests must finish within 60 minutes.
Offline requests like cron tasks have a time limit of 10 minutes.
Requests to Cloud SQL have limitations based on the scaling type of the App Engine module and how long an instance can remain in memory (residence).
Each App Engine instance running in a standard environment cannot have more than 60 concurrent connections to a Cloud SQL instance. For applications written in Java 8 or Go 1.8, the limit is 100.
Connection Issues: If you see errors containing "Aborted connection nnnn to db:", it usually indicates that your application is not terminating connections properly. It could also be caused by network issues. This error does not mean that there are problems with your Cloud SQL instance.
Google App Engine seems to automatically tunnel its connections to Cloud SQL 2nd generation internally through Cloud SQL Proxy. This was discovered inadvertently while trying to sort out how to use TLS, unsuccessfully: "TLS requested but server does not support TLS" error with Google Cloud SQL (2nd generation) from Google App Engine?
I noticed that this works without allowing unsecured access globally to the Cloud SQL instance... which is nice. However, we can only filter the accepted hostname for connections to cloudsqlproxy~% and not to localhost, and this allows virtually any "cloudsqlproxy" host to connect with the right credentials.
Is this safe and correct to do, and better than using %... which would obviously bypass any sort of host filtering? Or, does this open any cloudsqlproxy's possible connection to our 2nd generation instance?
The goal is to restrict connections on a particular user account on the SQL instance to ONLY come from our App Engine project. Nothing else should be able to connect with these credentials.
Good question, you are right that using cloudsqlproxy-% is the strictest filtering you can apply for App Engine connections right now and unfortunately that means you cannot effectively say "allow connections from App Engine but not from Cloud SQL Proxy".
It's hard to come up with a solution that maintains the consistency between App Engine Standard and App Engine Flexible since App Engine Flex VMs live in the customer project. It could be somewhat confusing if the restriction only applied to App Engine Standard, but not App Engine flex.
You can somewhat limit the exposure by limiting who can use the Cloud SQL Proxy by limiting the Editors (and owners) of a project as the account connecting using Cloud SQL Proxy must have Editor access or above. In the future, this will become more fine grained with IAM support.
When trying to connect my Google App Engine to my Google Cloud SQL Instance (Second Generation), I cannot find the "...Authorized App Engine applications section..." (https://cloud.google.com/appengine/docs/php/cloud-sql/#PHP_Build_a_starter_application_and_database).
Am I just blind, or does this not exist anymore?
If it doesn't exist, how does one connect a Google App Engine to a Google Cloud SQL (Second Generation)?
Please review the limitations of Google Cloud SQL Second Generation.
Because Cloud SQL Second Generation instances are in beta, the following features are not available:
Service Level Agreement (SLA)
MySQL 5.5
MySQL 5.6 is supported.
Google App Engine connectivity. Connectivity is supported for other clients, including Compute Engine, Managed VMs, Container Engine, and your workstation.
....
I'd like to mention that although Google App Engine connectivity is not yet supported for the Cloud SQL Second Generation like the way is supported for Cloud SQL 1st Gen, however this doesn't mean that you cannot use Cloud SQL 2nd Gen with your App Engine applications.
You can use access control model which is described in this article as used for other applications. Since IP address of your App Engine application will be not a static address, you will need to authorize 0.0.0.0/0 IP range as an allowed network and use Allow only SSL connections feature of the Cloud SQL to allow only SSL connections. Configure SSL and generate keys and client certificate for your application and establish a secure connections using SSL.
Right now, App Engine cannot be used with CloudSQL Gen2. It should be possible once the CloudSQL Gen2 graduate to General Availability but right now, if you need to use it with App Engine, you'll need to stick with CloudSQL v1
Somebody knows if is it possible connect an application from Appengine to a mysql database hosted in compute engine?
I'm trying to do this with python and i have this error:
Can't create TCP/IP socket (-1)
I'm using SqlAlchemy ORM which use the next configuration:
create_engine('mysql+mysqldb://root#ip/database')
and locally works but when i deploy the application to appengine doesn't work.
Thanks
Google App Engine, by default, runs code in a sandboxed environment, meaning that certain aspects of the Python runtime environment are restricted, or respond differently than they would otherwise. One of these aspects is outbound network connectivity—while GAE supports sockets, there are certain restrictions, and sockets are only available for paid apps.
The recommended options for storing information in a GAE app include the App Engine Datastore, Google Cloud SQL, and Google Cloud Storage. Google Cloud SQL is MySQL, and works with SQLAlchemy, so that's probably your best option.
If you absolutely need to run your own MySQL server (rather than using Google Cloud SQL) and connect to it from a GAE app, the other option is to use the managed VM environment, which permits unrestricted network access (since it's essentially a Google Compute Engine VM with the App Engine runtime on top).