I started using Google App Engine today, so hopefully the answer will be obvious to someone with experience. After simply following this method to get App Engine interfacing with Cloud SQL, I'm having a frustrating issue: goapp deploy succeeds but goapp serve fails (using the cloud shell).
Here's the error:
XXXX#cloudshell:~/src/XXXX/app-engine (XXXX)$ goapp serve app.yaml
INFO 2018-06-21 07:29:08,115 devappserver2.py:764] Skipping SDK update check.
INFO 2018-06-21 07:29:08,235 api_server.py:268] Starting API server at: http://0.0.0.0:60628
INFO 2018-06-21 07:29:08,305 dispatcher.py:199] Starting module "default" running at: http://0.0.0.0:8080
INFO 2018-06-21 07:29:08,306 admin_server.py:116] Starting admin server at: http://0.0.0.0:8000
ERROR 2018-06-21 07:29:12,254 go_runtime.py:181] Failed to build Go application: (Executed command: /google/go_appengine/goroot/bin/go-app-builder -app_base /home/XXX/app-engine -arch 6 -dynamic -goroot /google/go_appengine/goroot -gopath /home/XXX/gopath:/google/gopath -nobuild_files ^^$ -incremental_re
build -unsafe -binary_name _go_app -extra_imports appengine_internal/init -work_dir /tmp/tmpT1RTRMappengine-go-bin -gcflags -I,/google/go_appengine/goroot/pkg/linux_amd64_appengine -l
dflags -L,/google/go_appengine/goroot/pkg/linux_amd64_appengine hello.go)
/home/XXX/gopath/src/github.com/go-sql-driver/mysql/connection.go:12: can't find import: "context"
2018/06/21 07:29:09 Can't find package "context" in $GOPATH: cannot find package "context" in any of:
/google/go_appengine/goroot/src/context (from $GOROOT)
/home/XXX/gopath/src/context (from $GOPATH)
/google/gopath/src/context
2018/06/21 07:29:12 go-app-builder: build timing: 0×skip (3ms total), 9×compile (2.731s total), 0×link (0 total)
2018/06/21 07:29:12 go-app-builder: failed running compile: exit status 2
I'm suspicious that the version of go being used is wrong. The stated version is 1.6.3 but as I understand "context" requires 1.7. Here's the relevant cloud shell output for that:
XXXX#cloudshell:~ (XXX)$ goapp version
go version go1.6.3 (appengine-1.9.48) linux/amd64
XXXX#cloudshell:~ (XXX)$ go version
go version go1.10 linux/amd64
However, despite my best efforts I have been unable to find any resources on how to manipulate the "goapp" go version.
My app.yaml includes:
runtime: go
api_version: go1.8
And goapp get yields this:
XXXX#cloudshell:~/src/XXX/app-engine (XXX)$ goapp get
package context: unrecognized import path "context" (import path does not begin with hostname)
The go file itself is an exact copy of the demo from the linked above.
As confirmed on a comment on the question the solution was using dev_appserver.py, instead of goapp serve.
goapp is the old tool to handle GAE tasks. Now it's recommended to use dev_appserver.py for the local development, and gcloud app commands for tasks in the cloud (e.g. gcloud app deploy instead of goapp deploy).
Note that goapp is not deprecated, but dev_appserver.py provides more flexibility on local dev, while gcloud app uses the properly documented and quite useful App Engine Admin API (unlike the older tools), making debugging a lot easier in case something goes wrong with your deployment.
In Google App Engine, if you need to get a context, you should probably use google.golang.org/appengine.NewContext() for a context during an HTTP request, or google.golang.org/appengine.BackgroundContext() if it's outside a request. Both return context.Context, which is the same that you would get by creating a context through the context package, but with a context prepared to be used within GAE.
This does not tell you why the imports are not working (other than goapp looks to be using 1.6), but will just work around the issue avoiding you to import any context package.
Is it possible to deploy Elasticsearch on App engine flex environment using a docker image.
I have tried the following
My files on the local machine
Folder : elasticsearch
app.yaml
Dockerfile
docker-entrypoint.sh
config folder(containing elasticsearch.yml)file
Contents of app.yaml
runtime: custom
env: flex
Dockerfile and docker-entrypoint.sh copied from https://github.com/GoogleCloudPlatform/elasticsearch-docker/tree/master/5/5.2.0
Modifications to the Dockerfile
replaced EXPOSE 9200 9300 to EXPOSE 8080
Modification to the elasticsearch.yml
cluster.name: "beaconinside-docker-cluster"
path.data: /usr/share/elasticsearch/data
http.host: 0.0.0.0
http.port: 8080
discovery.zen.minimum_master_nodes: 1
I build a container using the docker file on my local machine
docker build -t elasticdemo .
Then, I run the container
docker run -p 8080:8080 elasticdemo
I am able to access elasticsearch on 0.0.0.0:8080
Problem:
I am trying to deploy elasticsearch as an app to Google app engine flex environment
gcloud app deploy app.yaml --version elasticdocker --project myproject
The deployment fails with the following error
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
I was expected elasticsearch to deploy as an app and be available on the deployed url.
Could you please provide pointers/help/suggestions with this approach?
While you can deploy ES to App Engine Flexible environment it's not particularly useful. The VMs hosting GAE Flexible containers are restarted regularly as part of maintenance and whatever data is stored on the local disk will be lost on restart. If you want to use local disk for long term storage, I'd suggest to deploy the GCE VM's (or alternatively use a solution from the GCP Marketplace) or deploy to GKE which supports persistent disks
As for the actual question: you probably don't have a health check handler and therefore App Engine Flexible environment doesn't consider your app healthy after deploying it. The error message is useless, I agree.
From the GAE Flexible docs for building custom images:
"A health check is an HTTP request to the URL /_ah/health. A healthy application should respond with status code 200."
Alternatively you can turn off health checks by adding into app.yaml
enable_health_check: False
I followed this tutorial
to get a Bigtable client up and running in Google Managed VMs. But is there a way to run this locally? Reason is that deploying the code remotely in development is a pain.
Normally I can use dev_appserver.sh to run GAE app locally. But when I run it, I'm getting this error:
Caused by: java.lang.IllegalStateException: Jetty ALPN has not been
properly configured.
Which means we need to include ALPN library? Since our codebase is in Java 7, I used this ALPN version: 7.1.3.v20150130.
I then tried again with this:
dev_appserver.sh --jvm_flag=-Xbootclasspath/p:/Users/shouguoli/tmp/alpn-boot-7.1.3.v20150130.jar
still getting this error:
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException:
The API package 'urlfetch' or call 'Fetch()' was not found.
How do you get it to work locally?
The sample was updated last week. It's based on the java 8 compat runtime, which means that you have access to most of the App Engine API's including Users, Task Queues, and Datastore.
There is a new Netty TCNative module that uses Boring SSL.
To use it with the pom.xml in the sample, do:
mvn clean -Pmac jetty:run -Dbigtable.projectID=<your-project> -Dbigtable.clusterID=<your-cluster> -Dbigtable.zone=<your-zone>
To use on Windows, use -Pwindows instead of -Pmac. For linux, omit the Profile -P as it's the default.
To deploy:
mvn clean gcloud:deploy -Dbigtable.projectID=<your-project> -Dbigtable.clusterID=<your-cluster> -Dbigtable.zone=<your-zone>
NOTE - it is advisable to do the clean between running locally and running remotely as the TCNative module is currently specific to the platform the code runs on.
We are in the process of updating all of our samples to use TCNative, we hope to have this by 3/10/16.
Background
"In the Python runtime, we've added support for the Python SSL
Library, so you can now open secure connections to remote services
such as Apple's Push Notification service."
This quote is taken from a recent post on the Google App Engine blog.
Implementation
If you want to use native python ssl, you must enable it using the libraries configuration in your application's app.yaml file where you specify the library name "ssl" . . .
These instructions are provided for developers through the Google App Engine documentation.
The following lines have been added to the app.yaml file:
libraries:
- name: ssl
version: latest
This much is in line with the advice provided through the Google App Engine documentation.
Problem
I have tried running my project in three different configurations. Two are working, and one is not.
Working ...
After I upload my application to Google App Engine, and run my project through the live server, everything works fine.
Working ...
When I run my project with manage.py runserver and include the Google App Engine SKD in my PYTHONPATH, everything works fine.
Not Working ...
However, when I run my project with dev_appserver.py, I get the following error:
ImportError at /
No module named _ssl
Request Method: GET
Request URL: http://localhost:8080/
Django Version: 1.4.3
Exception Type: ImportError
Exception Value:
No module named _ssl
Exception Location: /usr/local/lib/google_appengine_1.7.7/google/appengine/tools/devappserver2/python/sandbox.py in load_module, line 856
Python Executable: /home/rbose85/Code/venvs/appserver/bin/python
Python Version: 2.7.3
Python Path:
['/home/rbose85/Code/product/site',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7/lib/protorpc',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7/lib/protorpc',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7/lib/protorpc',
'/home/rbose85/Code/venvs/appserver/lib/python2.7',
'/home/rbose85/Code/venvs/appserver/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/local/lib/google_appengine',
u'/usr/local/lib/google_appengine_1.7.7/lib/django-1.4',
u'/usr/local/lib/google_appengine_1.7.7/lib/ssl-2.7',
u'/usr/local/lib/google_appengine_1.7.7/lib/webapp2-2.3',
u'/usr/local/lib/google_appengine_1.7.7/lib/webob-1.1.1',
u'/usr/local/lib/google_appengine_1.7.7/lib/yaml-3.10']
Server time: Wed, 24 Apr 2013 11:23:49 +0000
For the current GAE version (1.8.0 at least until 1.8.3), if you want to be able to debug SSL connections in your development environment, you will need to tweak a little bit the gae sandbox:
add "_ssl" and "_socket" keys to the dictionary _WHITE_LIST_C_MODULES in /path-to-gae-sdk/google/appengine/tools/devappserver2/python/sandbox.py
Replace the socket.py file provided by google in /path-to-gae-sdk/google/appengine/dis27 from the socket.py file from your Python framework.
IMPORTANT: Tweaking the sandbox environment might end up with functionality working on your local machine but not in production (for example, GAE only supports outbound sockets in production). I will recommend you to restore your sandbox when you are done developing that specific part of your app.
The solution by jmg works, but instead of changing the sdk files, you could monkey patch the relevant modules.
Just put something like this on the beginning of your project setup.
# Just taking flask as an example
app = Flask('myapp')
if environment == 'DEV':
import sys
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
from lib import copy_of_stdlib_socket.py as patched_socket
sys.modules['socket'] = patched_socket
socket = patched_socket
I had to use a slightly different approach to get this working in CircleCI (unsure what peculiarity about their venv config caused this):
appengine_config.py
import os
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
I had this problem because I wasn't vendoring ssl in my app.yaml file. I know the OP did that, but for those landing here for the OP's error, it's worth making sure lines like the following are in your app.yaml file:
libraries:
- name: ssl
version: latest
Stumbled upon this thread trying to work with Apples Push notification service and appengine... I was able to get this working without any monkey patching, by adding the SSL library in my app.yaml, as recommended in the official docs, hope that helps someone else :)
I added the code to appengine_config.py as listed by Spain Train, but had to also add the following code as well to get this to work:
phttplib = os.path.join(os.path.dirname(real_os_src_path), 'httplib.py')
imp.load_source('httplib', phttplib)
You can test if ssl is available at your local system by opening a python shell and typing import ssl. If no error appears then the problem is something else, otherwise you don't have the relevant libraries installed on your system. If you are using a Linux operating system try sudo apt-get install openssl openssl-devel or the relevant instructions for your operating system to install them locally. If you are using windows, these are the instructions.
I saw this link - App engine CPU times when downloading logs that refers to a question about how to get the performance information when downloading logs from google app engine with --include_all option.
I've tried it with the Java command line options and read the documetnation and it is not mentioned there at all!
How can I get the performance infromation such as cpu time when I download the logs from app engine?
The command I'm currently using (it works) is:
appcfg.cmd --num_days=3 --severity=0 request_logs . logs.txt
In the admin dashboard you can see this information:
"<my_app_name>.appspot.com" ms=13 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.000102
I want to be able to get this information in the logs as well.
thanks,
Li
The Java version of appcfg doesn't currently support that flag. You can create an app.yaml and use the Python version of the SDK to download the logs, though.