I am developing go api backend using google appengine. When i run the project locally the output says:
INFO 2018-07-11 12:31:17,502 devappserver2.py:120] Skipping SDK update check.
INFO 2018-07-11 12:31:17,576 api_server.py:274] Starting API server at: http://localhost:38628
INFO 2018-07-11 12:31:17,588 dispatcher.py:270] Starting module "default" running at: http://localhost:38629
What is the purpose of two endpoints "API Server" and "module default"?
My init is just like this:
http.HandleFunc("/signup", signUp)
http.HandleFunc("/whitelist", whitelist)
http.HandleFunc("/signin", signIn)
http.HandleFunc("/signout", signOut)
Those 2 "endpoints"–or rather servers–serve different purposes.
First some background:
"An App Engine app is made up of a single application resource that consists of one or more services." (source) Note: Services were previously called "modules".
So an app consists of one or multiple services (or modules). If you don't specify services in your app configuration, there is a default service.
This line:
Starting module "default" running at: http://localhost:38629
tells a web server has been started that will serve the default service (or module), which is the web server you register your handlers to ("/signup", "/whitelist", etc.).
The other server:
Starting API server at: http://localhost:38628
Starts an API server which is not directly used by you. It is an App Engine specific server that acts as a proxy so the local app engine environment can access remote App Engine services (such as Memcache, Datastore) over HTTP. This API server uses the Remote API protocol for communication, and the local dev environment connects to it using HTTP.
You do not need to worry about this API server, and you do not need to configure it or interact with it. It is part of the App Engine local dev environment which aids to access your remote services, those used by your production environment (should you need it).
The default module is the default route into your application. You can check the behavior of your routes in your app.yaml file to see/alter how requests get routed as well in the "handlers" section.
Check out the documentation on how requests are routed for more detailed information.
Related
I have a React application that I have been trying to run on GKE for weeks now but I cannot figure out the the GKE Ingress. There are a total of 7 microservices running including the React App.
My React App makes 4 API calls in total
"/posts/create" //creates a new post
'/posts/comments/*' //adds a comment to a post
'/posts' // gets posts+comments, returns empty object since no posts are created
'/posts/save' // saves post to cloudSQL
The application uses an event bus that handles communication between the different microservices so I created a ClusterIP service for each app and created additional NodePort services to use on the Ingress. After the Ingress is created I can access the React App but it says all of the backend services are unhealthy and I can't access them. I have tried calling the API's in several ways through the React Client including (calls // error in Chrome console
"http://query-np-srv:4002/posts" //Failed to load resource: net::ERR_NAME_NOT_RESOLVED
"http://10.96.11.196:4002/posts"(this is the endpoint for the service) //xhr.js:210 GET http://10.96.11.196:4002/posts net::ERR_CONNECTION_TIMED_OUT
"http://posts.com/posts // GET http://posts.com/posts 502 (Bad Gateway)
If i run any of the follwoing commands from the client pod I get an object returned as intended
curl query-srv:4002/posts
curl 10.96.12.242:4002/posts
curl query-np-srv:4002/posts
The only way I have been able to get this application to actually work on GKE is by exposing the client, posts, comments, and query pods on LoadBalancers and hard coding the LB IP's into the API calls, which cannot be a best practice. At least this way I know the project is functional and leads me to believe this is an ingress issue
Here is my Github repo for project
All of the yaml files are located in the infra/k8s folder and I am using the test.yaml to deploy the ingress, not the ingress-srv.yaml. Also, I am not using skaffold to deploy so that can be ignored as it is not causing the issues. If anyone can figure this out I would be very appreciative.
If after you create the ingress object the backends services are unhealthy, you need to review your Health checks. Did you review if GKE created Health checks for each backend service?
Health checks connect to backends on a configurable, periodic basis.
Each connection attempt is called a probe. Google Cloud records the
success or failure of each probe. Google Cloud considers backends to
be unhealthy when the unhealthy threshold has been met. Unhealthy
backends are not eligible to receive new connections; however,
existing connections are not immediately terminated. Instead, the
connection remains open until a timeout occurs or until traffic is
dropped.
Our AppEngine app is connecting to a remote service which requires a VPN and also required me to add entries to the hosts file on my local machine in order to connect to their endpoints.
e.g.
10.200.30.150 foo.bar.com
This is working fine when running the app locally, but I can't figure out how to set this up on Google Cloud to work once deployed.
I can't use the IP addresses directly because it errors that the IP is not on the cert's list.
How do I map the host names to the IPs in Google Cloud so that AppEngine can use them?
From the error mentioned in the comment I suspect connecting directly through the IP fails because the certificate doesn't recognize the IP to DNS mapping as valid and therefore the secure connection setup breaks. Based on the requirements of connecting to the API by VPN and tweaking the hosts mapping there are few things you may try.
The simplest approach that may work would be using a Google Compute Engine VM instance, since there you would able to manipulate the etc/hosts file and replicate the local machine setup. This VM could be used either as the main app service or as a proxy from App Engine to the 3rd party API endpoint. To go that route I would suggest taking a look at these two posts which explain how to change the etc/hosts file on GCE (Changing the file once wouldn't work as the VM periodically overrides it, see the posts for cronjob like workaround).
Separately, as your app runs in App Engine flexible environment there is the chance to provide a docker container with the app packaged. It may be possible to set the workaround above in the docker file and have it working in App Engine too.
I'm trying to implement a simple design in google cloud using app engine standard and flexible with datastore. App1 lives in GAE standard environment. When a user interacts with this app, it writes some data to datastore and queues a task. The target of the queued task is App2 that lives in app engine flexible environment (the task can take a longer time to complete than standard environment allows). The idea is for App2 to read the data from datastore, perform the task using the data, once complete it should write a report entity to datastore. I've attached a simple diagram.
In App1 I've set up a Service Account named flexKey with Owner permissions, downloaded the json file.
when I run App2 locally I first export the path to the credentials json file as an environment variable:
GOOGLE_APPLICATION_CREDENTIALS: "path/to/flexKey.json"
then launch the app with mvn jetty:run-exploded and everything works fine, App2 is able to authenticate with live datastore (not local emulation), and read the data written by App1. When I unset the environment variable, I get an 'Unauthenticated' error (expected)
To use the same service account when App2 is deployed I've added the following in app.yaml for App2 to set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the service account json file flexKey.json (this is the path to the file on the deployed instance):
env: flex
env_variables:
GOOGLE_APPLICATION_CREDENTIALS: "/var/lib/jetty/webapps/root/WEB-INF/classes/flexKey.json"
runtime: java
However, when I deploy App2 to app engine flexible environment, there is an error authenticating with datastore when trying to do the read query (this works fine when querying datastore with the same credentials from a locally running instance of App2):
com.google.cloud.datastore.DatastoreException: Missing or insufficient permissions.
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:114)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:182)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:178)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:174)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:173)
.....
the code is PERMISSION_DENIED
If I leave out the environment variable GOOGLE_APPLICATION_CREDENTIALS from the app.yaml file for App2, then I get an 'Unauthenticated' error, so I think it's reading the file so I'm not sure what the issue is
I'm using Objectify v6
I'm not able to see why the same credentials (created with Owner role) work fine when querying the datastore from a locally running instance of the app but don't work when datastore is queried from the deployed version of the same app (deployed to flexible environment). setting the path to the credentials file via an environment variable in app.yaml is the method recommended in the documentation unless I am mistaken.
Is the GOOGLE_APPLICATION_CREDENTIALS environment variable not properly set in app.yaml?
Is there something conceptually problematic about my design?
All help appreciated.
You have a blocking issue in your design: it is not possible for one application to enqueue tasks into a push queue targeted at a service from another application. From the <target> (push queues) rows in the Syntax tables for both queue.yaml and queue.xml references:
The string is prepended to the domain name of your app when
constructing the HTTP request for a task. For example, if your app ID
is my-app and you set the target to my-version.my-service, the
URL hostname will be set to
my-version.my-service.my-app.appspot.com.
If you want to use the task queue then you have to make the 2 services part of the same application. As a (positive) side effect you don't have to worry about setting up the authentication for datastore access anymore - both services can directly access the app's datastore.
I have a GAE project (flexible) consisting of 1 default and 2 subservices:
foo.appspot.com
service1.foo.appspot.com
service2.foo.appspot.com
Now I want to use foo.appspot.com as API proxy & auth gateway to the internal services service1 and service2. The proxy itself I wrote and it is working fine.
I am struggling with adjusting the GAE Firewall to forbid incoming world traffic to service1 and service2 because I would like force an API user to send requests to foo.appspot.com. Traffic to the default service foo should be allowed.
It seems I can just enter IPs in the Firewall settings but not service names. The docs says that it should work but does not show how.
Thanks for the help!
App engine Flex environment is built on the Google Compute Engine and consequently, it supports the Virtual Private Cloud networking system.
With the VPC networks, you can configure firewall rules that would use Instance Tags to determine the target or source component in a firewall rule. Hence, you simply have to configure the app.yaml files of the target service/version to use the appropriate instance tags.
I would like to debug my Google App Engine (GAE) app locally but without using localhost. Since my application is made up of microservices, the urls in a production environment would be along the lines of:
https://my-service.myapp.appspot.com/
But code in one service can call another service and that means that the urls are hardcoded. I could of course use a mechanism in code to determine whether the app is running locally or on GAE and use urls that are different although I don't see how a local url would handle the since the only way to run an app locally is to use localhost. Hence:
http://localhost:8080/some-service
Notice that "some-service" maps to a servlet, whereas "my-service" is a name assigned to a service when the app is uploaded. These are really two different things.
The only possible solution I was able to find was to use a reverse proxy which would map one url to a different one. Still, it isn't clear whether the GAE development SDK even supports this.
Personally I chose to detect the local development vs GAE environment and build my inter-services URLs accordingly. I feel it was a well-worthy effort, I've been (re)using it a lot. No reverse proxy or any other additional ops necessary, it just works.
Granted, I'm using Python, so I'm not 100% sure a complete similar Java solution exists. But maybe it can point you in the right direction.
To build the per-service URLs I used modules.get_hostname() (the implementation is presented in Resolve Discovery path on App Engine Module). I believe the Java equivalent would be getInstanceHostname() from com.google.appengine.api.modules.
This method, when executed on the local server, automatically provides the particular port the server listens to for each service.
BTW, all my services for an app are executed by a single development server process, which listens on multiple ports (this is, I guess, how it can provide the modules.get_hostname() info). See Running multiple services using dev_appserver.py on different ports. This is part I'm unsure about: if/how the java local dev server can simultaneously run multiple services. Apparently this used to be supported some time ago (when services were still called modules):
Serving multiple GAE modules from one development server?
GAE modules on development server
This can be accomplished with the following steps:
Create an entry in the hosts file
Run the App Engine Dev server from a Terminal using certain options
Use IntelliJ with Remote debugging to attach the App Engine Dev server.
To edit the hosts file on a Mac, edit the file /etc/hosts and supply the domain that corresponds to your service:. Example:
127.0.0.1 my-service.myapp.com
After you save this, you need to restart your computer for the changes to take place.
Run the App Engine Dev server manually:
dev_appserver.sh --address=0.0.0.0 --jvm_flag=-Xdebug
--jvm_flag=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
[path_to_exploded_war_directory]
In IntelliJ, create a debug configuration. Use the Remote template to create this configuration. Set the host to the url you set in the hosts file and set the port to 8000.
You can set a breakpoint and run the app in IntelliJ. IntelliJ will attach to the running instance of App Engine Dev server.
Because you are using a port during debugging and no port is actually used when the app is uploaded to the GAE during production, you need to add code that identifies when the app is running locally and when it's running on GAE. This can be done as follows:
private String mServiceUrl = "my-service.my-app.appspot.com";
...
if (SystemProperty.environment.value() != SystemProperty.Environment.Value.Production) {
mServiceUrl += ":8000";
}
See https://cloud.google.com/appengine/docs/standard/java/tools/using-local-server
An improved solution is to avoid including the port altogether and not having to use code to determine whether your app is running locally or on the production server. One way to do this is to use Charles (an application for monitoring and interacting with requests) and use a feature called Remote Mapping which lets you map one url to another. When enabled, you could map something like:
https://my-service.my-app.appspot.com/
to
https://localhost:8080
You would then enable the option to include the original host, so that this gets delivered to the local dev server. As far as your code is concerned it only sees:
https://my-service.my-app.appspot.com/
although the ip address will be 127.0.0.1:8080 when remote mapping is enabled. To use https on local host however does require that you enable ssl certificates for Charles.
For a complete overview on how to setup and debug microservices for a GAE Java app in IntelliJ, see:
https://github.com/JohannBlake/gae-microservices