Google Cloud Memorystore (Redis) ETIMEDOUT in App Engine - google-app-engine

Im writing a NodeJS app and trying to connect to GCPs Redis MemoryStore, but I'm getting the ETIMEDOUT 10.47.29.131:6379 error message. The 10.47.29.131 corresponds to the REDISHOST. I'm trying to reach the server by it's internal private IP.
While the app works locally with a local Redis installed, it does not when deployed to the GCP AppEngine.
My GCP-Setup
Redis instance running at location europe-west3-a
Created a connector under "Serverless VPC access" which is in europe-west3
Redis and the VPC-connector are on the same network "default".
App Engine running in europe-west
Redis isntance:
VPC-connector:
The app.yml
runtime: nodejs
env: flex
automatic_scaling:
// or this but without env: flex (standard)
vpc_access_connector:
name: "projects/project-ID/locations/europe-west/connectors/connector-name"
beta_settings:
cloud_sql_instances: project-ID:europe-west3:name
env_variables:
REDISHOST: '10.47.29.131'
REDISPORT: '6379'
// removed this when trying without env: flex (standard)
network:
name: default
session_affinity: true
I followed these instructions to set everything up: https://cloud.google.com/memorystore/docs/redis/connect-redis-instance-standard
Digging deeper, I found: https://cloud.google.com/vpc/docs/configure-serverless-vpc-access where they mention something about permissions and serverless-vpc-access-images, and while trying to follow the instructions: https://cloud.google.com/compute/docs/images/restricting-image-access#trusted_images I couldn't find "Define trusted image projects." anywhere
What am I missing here?

Well, turns out, the problem was the region I've selected for the Redis instance.
From Documentation:
Important: In order to connect to a Memorystore for Redis instance, the connecting client must be located within the same region as the instance.
A region is a specific geographical location where you can run your resources. Each region is subdivided into several zones.
For example, the us-central1 region in the central United States has zones us-central1-a, us-central1-b, us-central1-c, and us-central1-f.
Althouh the documentation clearly says, that AppEngine and Memorystore have to be in the same region, my assumption on what regions actually are, was false.
When I created the AppEngine, I created it in europe-west, which is the same as europe-west1. On the other hand, when I created the redis instance, I used europe-west3, with the assumption that west3 is the same region as west, which is not.
Since the AppEngines region cannot be changed, I created another redis-instance in europe-west1 and now everything works.
So, the redis region must be exactly the same as the AppEngine region. region1 is the same as region, but region2 or region3 are not the same as region.

Related

GCP - what are mysql-access instances?

I have a Java application deployed to AppEngine Standard and connects to Cloud SQL via public IP. I was looking at VM instances dashboard and found a set of instances with the following naming pattern and these are alerting for high cpu utilization.
aet-uswest1-mysql--access-{abcd}
The description says "Anthos/GKE and Dataproc VMs are Google-managed and include built-in agents.". These are all e2-micro instances and I could not change the instance type. At first I thought these are the underlying AppEngine instances, but the AppEngine instances I use are F4 class, I think these instances are something else.
What are these instances and how are they used?
Here is the list I see under instances in VM instances dashboard. I can't ssh or manage these instances. (I have randomized the instance names and ip addresses)
Name,Agent,Active Alerts,System Events,Zone,Private IP,Size
aet-uswest1-mysql--access-abcd,Not applicable,0,0,us-west1-a,10.5.0.9,e2-micro
aet-uswest1-mysql--access-efgh,Not applicable,0,0,us-west1-a,10.5.0.12,e2-micro
aet-uswest1-mysql--access-ijkl,Not applicable,0,0,us-west1-b,10.5.0.3,e2-micro
aet-uswest1-mysql--access-mnop,Not applicable,0,0,us-west1-c,10.5.0.7,e2-micro
aet-uswest1-mysql--access-qrst,Not applicable,0,0,us-west1-b,10.5.0.5,e2-micro
aet-uswest1-mysql--access-uvwx,Not applicable,0,0,us-west1-b,10.5.0.10,e2-micro
aet-uswest1-mysql--access-yz01,Not applicable,0,0,us-west1-c,10.5.0.2,e2-micro
aet-uswest1-mysql--access-23df,Not applicable,0,0,us-west1-c,10.5.0.6,e2-micro
aet-uswest1-mysql--access-efef,Not applicable,0,0,us-west1-a,10.5.0.11,e2-micro
aet-uswest1-mysql--access-57sf,Not applicable,0,0,us-west1-b,10.5.0.13,e2-micro
My understanding that AppEngine service was connected to Mysql via public network is incorrect. I found that these instances are related to VPC connector used by AppEngine standard to access MySQL service.

gcloud.app.deploy - The caller does not have permission [duplicate]

I have followed through the steps in google cloud platform guide, but still getting permission error. which says the caller does not have the permission. pls what am I doing wrong.
this is the out of command gcloud config list
region = us-central1
zone = us-central1-f
[core]
account = <gmail-account>
disable_usage_reporting = True
project = <project-id>
Your active configuration is: [default]
this is the error it raised
ERROR: (gcloud.app.deploy) Error Response: [13] Flex operation projects/<project-id>/regions/europe-west1/operations/error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2020-07-28T15:45:31.962Z49210.jv.11: Deployment Manager operation <project-id>/operation-... errors: [code: "RESOURCE_ERROR"
location: "/deployments/aef-default-..../resources/aef-default-...."
message: "{\"ResourceType\":\"compute.beta.regionAutoscaler\",
\"ResourceErrorCode\":\"403\",
\"ResourceErrorMessage\":{\"code\":403,
\"message\":\"The caller does not have permission\",
\"status\":\"PERMISSION_DENIED\",
\"statusMessage\":\"Forbidden\",
\"requestPath\":\"https://compute.googleapis.com/compute/beta/projects/<project-id>/regions/europe-west1/autoscalers\",
\"httpMethod\":\"POST\"}}"
Please check your project quotas usually this error is raised when your project doesn't have enough IPs or VMs (App engine Flex uses Compute Engine VMs) and the scaling strategy on your app.yaml is exceeding the quotas.
Please try to add one of the following blocks in your app.yaml file
For automatic scaling
automatic_scaling:
min_num_instances: 1
max_num_instances: 2
For manual scaling
manual_scaling:
instances: 2
To avoid exhaust these quotas please, delete/stop the App Engine service versions that you don't need.
For more information about scaling strategies please check this reference guide
For example:
Every VM takes 1 IP and your project have a quota of 4.
If your app engine service has 3 VMs running (3 IPs used), in the next deploy you only has 1 IP available, if your min_instances or instances in your app.yaml file is greater than 1, the deploy will fails.
This is because is not possible allocate more than 4 IPs on your project, and App engine first turn on the new instances and after shutdown the old instances, this is to avoid a service interruption
If you need increase this resource quotas it is necessary to contact a GCP sales rep.

gcloud - Can't configure my VPC connector to work with my Redis instance

I'm facing a problem with gcloud and their support can't seem to help me.
So, to put my app in prod I need to use a redis instance to host some data. I'm using memorystore because I like to have everything on gcloud.
My app is in the standard environment on app engine so on their doc (https://cloud.google.com/memorystore/docs/redis/connect-redis-instance-standard) they ask me to configure a VPC connector. But I think that the CIDR that I put is always wrong, can someone help me finding the good CIDR.
connectMode: DIRECT_PEERING
createTime: '2020-03-13T17:20:51.590448560Z'
currentLocationId: europe-west1-d
displayName: APP-EPG
host: 10.240.224.179
locationId: europe-west1-d
memorySizeGb: 1
name: projects/*************/locations/europe-west1/instances/app-epg
persistenceIamIdentity: *************
port: 6379
redisVersion: REDIS_4_0
reservedIpRange: 10.240.224.176/29
state: READY
tier: BASIC
Thank you all !
First in order to VPC connector work yor App Engine instances have to be in the same VPC & region that your Redis instance is. If not there will not be connectivity between the two.
Also make sure you redis and app use one of the approved locations. By now it's a lot of them.
Your redis instance is in europe-west1 region so to create your VPC connector you have to set the name of the VPC network your redis instance is in
(for example "default").
IP range you were asking about is any range (not reserved by the network redis instance is in).
So - for example if your "default" network is 10.13.0.0/28 then you have to specify something else like 10.140.0.0/28 etc. It has to be /29 - otherwise you won't be able to create the connector.
Why 10.13.0.0 or any other addresses ? They are going to be assigned as the source network for you apps to connect to the Redis (or any
other VM's) in the specified network.
I've tested it using the command:
cloud compute networks vpc-access connectors create conn2 --network default /
--range 10.13.0.0/28 --region=europe-west1
Or you can do it using console in Serverless VPC Access and clicking "Add new connector";
You can also read documentation on how to create a connector.

GAE Region - Exact Location

My GAE (Standard) app is hosted in europe-west region.
I am looking into creating a Cloud SQL instance in support of this app and would like to place it closest to the GAE.
Current Cloud SQL instances are available at below locations:
europe-west1 Belgium
europe-west2 London
europe-west3 Frankfurt
Is there any way to find out additional location details of my GAE app in order to decide which Cloud SQL location to use?
You can use gcloud app describe from the cloud shell in order to find out your app's location info. Typing this command (gcloud app describe) in the shell returns something like this:
authDomain: gmail.com
codeBucket: staging.my-project-id.appspot.com
defaultBucket: my-project-id.appspot.com
defaultHostname: my-project-id.appspot.com
featureSettings:
splitHealthChecks: true
gcrDomain: eu.gcr.io
id: my-project-id
locationId: europe-west2
name: apps/my-project-id
servingStatus: SERVING
See the command description here.
Next you can create your SQL instance by typing gcloud sql instances create [your-instance-name] --region=[region-of-your-choice]. For example:
user-id#my-project-id:~$ gcloud sql instances create test-instance --region=europe-west2
Creating Cloud SQL instance...done.
Created [https://www.googleapis.com/sql/v1beta4/projects/my-project-id/instances/test-instance].
NAME DATABASE_VERSION REGION TIER ADDRESS STATUS
test-instance MYSQL_5_6 europe-west2 db-n1-standard-1 00.000.000.000 RUNNABLE
user-id#my-project-id:~$
All the available options are here.

Using h2 database for local development in Google App Engine

I'm developing a Google App Engine application. In production I'm planning on using Google Cloud SQL, but in development I wanted to use an h2 database. It works fine when using an in-memory and embeded db (jdbc:h2:mem/dev/jdbc:h2:~/folder/dev) but not with a server-enabled one (jdbc:h2:~/folder/dev;AUTO_SERVER=TRUE), it fails with the following error:
org.h2.jdbc.JdbcSQLException: IO Exception: "java.net.SocketException: Permission denied: Not allowed to issue a socket bind: permission denied."; "port: 0 ssl: false" [90031-168]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:158)
at org.h2.message.DbException.convertIOException(DbException.java:315)
at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:193)
at org.h2.util.NetUtils.createServerSocket(NetUtils.java:156)
at org.h2.server.TcpServer.start(TcpServer.java:222)
at org.h2.tools.Server.start(Server.java:455)
at org.h2.engine.Database.startServer(Database.java:672)
at org.h2.engine.Database.open(Database.java:544)
at org.h2.engine.Database.openDatabase(Database.java:222)
at org.h2.engine.Database.<init>(Database.java:217)
at org.h2.engine.Engine.openSession(Engine.java:56)
at org.h2.engine.Engine.openSession(Engine.java:159)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:138)
at org.h2.engine.Engine.createSession(Engine.java:121)
at org.h2.engine.Engine.createSession(Engine.java:28)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:305)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
at org.h2.Driver.connect(Driver.java:72)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
As I understand, this is GAE's limitations playing up.This is stopping me from having easy access to the db while the application is running. Is there a way to circumvent this issue in local server mode only? I'm curious if there's an alternative to just using a MySql db in dev mode.
Currently trying out H2 in GAE too. Standard environment is a no go for me because of mem and writable disk: a combined max of 1Gb with a F4-1G instance.
If h2 embedded mode works, you must be using Flexible as Standard only allows /tmp for write operations.
On the error itself, GAE does not allow listening sockets so no auto-server unfortunately regardless of the type of environment (https://cloud.google.com/appengine/docs/standard/java/sockets/)

Resources