Sending HTTPS requests in Google App Engine using requests python library - google-app-engine

The title pretty much sums it up, I am making a request to the google translate api like this:
payload = {"key":translate_api_key, "q":str(sentence)}
try:
api_response = requests.get("https://www.googleapis.com/language/translate /v2/detect", params=payload)
except Exception, e:
print e
this works fine normally (by which i mean just running it as a script on my desktop) but on the google app engine server I am getting this response:
{u'error': {u'message': u'SSL is required to perform this operation.', u'code': 403, u'errors': [{u'message': u'SSL is required to perform this operation.', u'domain': u'global', u'reason': u'sslRequired'}]}}
How can I fix this?
EDIT:
Seems request's https doesn't play nicely with GAE. Using urlfetch and urllib seems to fix this.
payload = dict(key=translate_api_key, q=sentence)
payload = urllib.urlencode(payload)
url = "https://www.googleapis.com/language/translate/v2/detect?"
api_response = urlfetch.fetch(url+payload)

I have no experience of using the requests library, but it may be that it is not fully implemented on App Engine.
The preferred method on App Engine is to use urlfetch
from google.appengine.api import urlfetch
url = "https://www.googleapis.com/language/translate/v2/detect"
payload = {"key":translate_api_key, "q":str(sentence)}
result = urlfetch.fetch(url=url, payload=payload)
if result.status_code == 200:
api_response = result.content

You can try in the app.yaml
handlers:
- url: /youraccount/.*
script: accounts.py
login: required
secure: always
The secure:always makes your site ssl. I'm not sure this is that actual problem though.

GCP does not support the use of Requests library out of the box. So we will have to make some tweeks to make it work.
In order to deploy an application on the Google App Engine, we need to make a main.py(where main python flask app resides) and app.yaml(configuration file needed to run it in GCP).
Here is a sample code for the app.yaml file for python 2.7 environment
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
redirect_http_response_code: 301
script: main.app
libraries:
- name: flask
version: 0.12
Now we need to configure the requests library to use URLfetch internally.
To use requests, we'll need to install both requests and requests-toolbelt using the vendoring instructions. (https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27#installing_a_library).
Basically we need to install our custom libraries.
Create a directory to store your third-party libraries, such as lib/
mkdir lib
Upload the requests and requests-toolbelt libraries from your system or download them directly into the lib folder created in earlier step.
Use pip (version 6 or later) with the -t flag to copy the libraries into the folder you created in the previous step. For example:
pip install -t lib/ (pip install -t lib/ requests)
Create a file named appengine_config.py in the same folder as your app.yaml file.
Edit the appengine_config.py file and provide your library directory to the vendor.add() method.
Sample appengine_config.py file
from google.appengine.ext import vendor
Add any libraries install in the "lib" folder.
vendor.add('lib/requests')
vendor.add('lib/requests_toolbelt')
Once installed, use the requests_toolbelt.adapters.appengine module to configure requests to use URLFetch.
Copy the below code to the start of your main.py file
import requests
from requests_toolbelt.adapters import appengine
appengine.monkeypatch(validate_certificate=True)
(https://cloud.google.com/appengine/docs/standard/python/issue-requests)
Now we can easily use the requests library to make get/post requests.
Test your app:
dev_appserver.py --port=<port number> app.yaml

Related

How to pass -ldflags to GAE build?

I have an HTTP service written in Go. Inside main.go I have a global version string.
package main
var version string
Locally, I build using -ldflags "-X main.version=$VERSION where $VERSION is determined by the shell environment, like so:
VERSION=v0.16.0 go build ./cmd/app -ldflags "-X main.version=$VERSION
I've recently decided to trial Google App Engine and started with a basic YAML file:
runtime: go111
handlers:
- url: /.*
script: auto
What can I set in the YAML file in order to instruct GAE to build with the equivalent ldflags to bake in my version string?
I should also mention I use go modules with GO111MODULE=on locally when building.
You can't do it with you app.yaml file.
However, you can use Cloud build to build and deploy your app to App Engine.
In your cloudbuild.yaml you can add a line to the build step
args: ['build', '-a', '-installsuffix', 'cgo', '-ldflags', '''-w''', '-o', 'main', './main.go']

Consume SQS tasks from App Engine

I'm attempting to integrate with a third party that is posting messages on an Amazon SQS queue. I need my GAE backend to receive these messages.
Essentially, I want the following script to launch and always be running
import boto3
sqs_client = boto3.client('sqs',
aws_access_key_id=KEY,
aws_secret_access_key=SECRET,
region_name=REGION)
while True:
sqs_client.receive_message(QueueUrl=QUEUE_URL, WaitTimeSeconds=60)
for message in msgs_response.get('Messages', []):
deferred.defer(process_and_delete_message, message)
My main appengine web app is on Automatic Scaling (with the 60-second &10-minute task timeouts), but I'm thinking of setting up a micro-service set to either Manual Scaling or Basic Scaling because:
Requests can run indefinitely. A manually-scaled instance can choose to handle /_ah/start and execute a program or script for many hours without returning an HTTP response code. Task queue tasks can run up to 24 hours.
https://cloud.google.com/appengine/docs/standard/python/an-overview-of-app-engine
Apparently both Manual & Basic Scaling also allow "Background Threads", but I am having a hard-time finding documentation for it and I'm thinking this may be a relic from the days before they deprecated Backends in favor of Modules (although I did find this https://cloud.google.com/appengine/docs/standard/python/refdocs/modules/google/appengine/api/background_thread/background_thread#BackgroundThread).
Is Manual or Basic Scaling suited for this? If so, what should I use to listen on sqs_client.receive_message()? One thing I'm concerned about is this task/background thread dieing and not relaunching itself.
This maybe a possible solution:
Try to use a Google Compute Engine micro instance to run that script continuously and send a REST call to your app engine app. Easy Python Example For Compute Engine
OR:
I have used modules that run instance type B2/B1 for long running jobs; and I have never had any trouble; but those jobs do start and stop. I use the basic scaling: with max_instances set to 1. The jobs I have run take around 6 hours to complete.
I ended up creating a manual scaling app engine standard micro-service for this. This micro-service has handeler for /_ah/start never returns and runs indefinitely (many days at a time) and when it does get stopped, then app engine restarts it immediately.
Requests can run indefinitely. A manually-scaled instance can choose
to handle /_ah/start and execute a program or script for many hours
without returning an HTTP response code. Task queue tasks can run up
to 24 hours.
https://cloud.google.com/appengine/docs/standard/python/an-overview-of-app-engine
My /_ah/start handler listens to the SQS queue, and creates Push Queue tasks that my default service is set up to listen for.
I was looking into the Compute Engine route as well as the App Engine Flex route (which is essentially Compute Engine managed by app engine), but there were other complexities like not getting access to ndb and the taskqueue sdk and I didn't have time to dive into that.
Below are all of the files for this micro-service, not included is my lib folder that contains the source code for boto3 & some other libraries I needed.
I hope this helpful for someone.
gaesqs.yaml:
application: my-project-id
module: gaesqs
version: dev
runtime: python27
api_version: 1
threadsafe: true
manual_scaling:
instances: 1
env_variables:
theme: 'default'
GAE_USE_SOCKETS_HTTPLIB : 'true'
builtins:
- appstats: on #/_ah/stats/
- remote_api: on #/_ah/remote_api/
- deferred: on
handlers:
- url: /.*
script: gaesqs_main.app
libraries:
- name: jinja2
version: "2.6"
- name: webapp2
version: "2.5.2"
- name: markupsafe
version: "0.15"
- name: ssl
version: "2.7.11"
- name: pycrypto
version: "2.6"
- name: lxml
version: latest
gaesqs_main.py:
#!/usr/bin/env python
import json
import logging
import appengine_config
try:
# This is needed to make local development work with SSL.
# See http://stackoverflow.com/a/24066819/500584
# and https://code.google.com/p/googleappengine/issues/detail?id=9246 for more information.
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
import sys
# this is socket.py copied from a standard python install
from lib import stdlib_socket
socket = sys.modules['socket'] = stdlib_socket
except ImportError:
pass
import boto3
import os
import webapp2
from webapp2_extras.routes import RedirectRoute
from google.appengine.api import taskqueue
app = webapp2.WSGIApplication(debug=os.environ['SERVER_SOFTWARE'].startswith('Dev'))#, config=webapp2_config)
KEY = "<MY-KEY>"
SECRET = "<MY-SECRET>"
REGION = "<MY-REGION>"
QUEUE_URL = "<MY-QUEUE_URL>"
def process_message(message_body):
queue = taskqueue.Queue('default')
task = taskqueue.Task(
url='/task/sqs-process/',
countdown=0,
target='default',
params={'message': message_body})
queue.add(task)
class Start(webapp2.RequestHandler):
def get(self):
logging.info("Start")
for loggers_to_suppress in ['boto3', 'botocore', 'nose', 's3transfer']:
logger = logging.getLogger(loggers_to_suppress)
if logger:
logger.setLevel(logging.WARNING)
logging.info("boto3 loggers suppressed")
sqs_client = boto3.client('sqs',
aws_access_key_id=KEY,
aws_secret_access_key=SECRET,
region_name=REGION)
while True:
msgs_response = sqs_client.receive_message(QueueUrl=QUEUE_URL, WaitTimeSeconds=20)
logging.info("msgs_response: %s" % msgs_response)
for message in msgs_response.get('Messages', []):
logging.info("message: %s" % message)
process_message(message['Body'])
sqs_client.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=message['ReceiptHandle'])
_routes = [
RedirectRoute('/_ah/start', Start, name='start'),
]
for r in _routes:
app.router.add(r)
appengine_config.py:
import os
from google.appengine.ext import vendor
from google.appengine.ext.appstats import recording
appstats_CALC_RPC_COSTS = True
# Add any libraries installed in the "lib" folder.
# Use pip with the -t lib flag to install libraries in this directory:
# $ pip install -t lib gcloud
# https://cloud.google.com/appengine/docs/python/tools/libraries27
try:
vendor.add('lib')
except:
print "Unable to add 'lib'"
def webapp_add_wsgi_middleware(app):
app = recording.appstats_wsgi_middleware(app)
return app
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
print "gaesqs development"
import imp
import os.path
import inspect
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
os.environ['HTTP_HOST'] = "my-project-id.appspot.com"
else:
print "gaesqs prod"
# Doing this on dev_appserver/localhost seems to cause outbound https requests to fail
from lib import requests
from lib.requests_toolbelt.adapters import appengine as requests_toolbelt_appengine
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()

App Engine deploy with Go libraries

I'm new on Google App Engine. And, I'm getting an issue that I can't solve.
I've a very simple app (developped in Go) like this :
main/
| model/
| | user.go
| main.go
| app.yaml
These are the imports of main.go :
import (
"github.com/julienschmidt/httprouter"
"log"
"net/http"
)
My code works well when I run it locally.
But, when I try to publish it on my Google App Engine instance, I receive this error :
$ gcloud app deploy
You are about to deploy the following services:
- <MY_APP_ENGINE_URL> (from [<MY_LOCAL_YAML_PATH>])
Deploying to URL: [<MY_APP_ENGINE_URL>]
Do you want to continue (Y/n)? Y
Beginning deployment of service [default]...
Some files were skipped. Pass `--verbosity=info` to see which ones.
You may also view the gcloud log file, found at
[<LOCAL_APP_ENGINE_LOG>].
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
2017/05/27 14:48:24 go-app-builder: build timing: 5×compile (301ms total), 0×link (0s total)
2017/05/27 14:48:24 go-app-builder: failed running compile: exit status 2
main.go:4: can't find import: "github.com/julienschmidt/httprouter"
What did I do wrong ?
EDIT :
This is the content of my app.yaml file :
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
App Engine environment doesn't contain your dependencies, you can add an script to do a go get ... for each one but it's too hacky and Go has a solution for that, we can save our dependencies in a vendor folder on the root of our project.
Quick solution:
# Instal godep:
go get -v -u github.com/tools/godep
cd your/project/path
godep save
Now try to deploy again, you'll see a vendor folder in your project, don't remove it and add it to your git source, that folder contains all your third party dependencies like your httprouter (it's my favorite :) )
Note You can use other tools to save your dependencies
I haven't used the gcloud tool, but back in the day when goapp was the tool you had to create github.com/julienschmidt/httprouter (with the lib's source in it, of course) directly under you'r main and then deploy.
AFAIK the App Engine's go version is currently 1.6 which means that while the vendoring is on by default, it can be switched off - perhaps thats the case and thats why #Yandry Pozo's suggestion doesn't work.

Google Cloud Endpoints files generated, server not working

I run endpoints.sh
get-client-lib com.my.app.FooService and successfully generate the files for Google Cloud Endpoints[1] (2 .discovery files, 1 .api file and 1 .zip file).
The script doesn't add anything to my war folder, so I assume server-side is already handled by magic configured in web.xml and #Api annotations (a'la Spring Framework).
But http://localhost:8080/_ah/api/explorer redirects me to a blank Google cloud console. Uploading my app yields the same result.
Am I missing something? I think the documentation is a bit lacking, it doesn't even explain what the generated files are for.
I'm using Google App Engine Java.
[1] https://developers.google.com/appengine/docs/java/endpoints/gen_clients
To make the endpoints work, need follow the steps stated in the Google GAE page
Write your API backend code first.
Annotate your API backend code, so classes and client libraries can be generated from it. (Alternatively, use the Google Plugin for Eclipse, which annotates automatically for you.)
Generate the client library using the endpoints.sh utility. (Alternatively, use the Google Plugin for Eclipse to generate the client library.)
Write your client app, using the client library when making calls to the API backend.
But the above steps miss out some important steps, which the Google Plugins (Google Plugin for Eclipse) will generated for you automatically.
Configure the servlet in your web.xml. (Replace the your-full-class-name with your own class name)
<servlet>
<servlet-name>com.google.api.server.spi.SystemServiceServlet</servlet-name>
<servlet-class>com.google.api.server.spi.SystemServiceServlet</servlet-class>
<init-param>
<param-name>services</param-name>
<param-value>your-full-class-name</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>com.google.api.server.spi.SystemServiceServlet</servlet-name>
<url-pattern>/_ah/spi/*</url-pattern>
</servlet-mapping>
you need ensure the file (.api) generated by the endpoints.sh is copied into the folder WEB-INF in your web app root folder.
For better understanding, you can invoke the endpoints.sh to see all the available options as the following
Available commands:
get-client-lib: Generates a client library
usage: get-client-lib ...
Options:
--classpath=CLASSPATH Additional class path entries
-cp CLASSPATH (default: ./war/WEB-INF/classes).
--language=LANGUAGE The target output programming language
-l LANGUAGE (java) (default: java).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
--war=WAR_PATH The path to a directory or .war with a WAR
-w WAR_PATH directory layout (default: ./war).
gen-api-config: Generates API configuration files from service classes
usage: gen-api-config ...
Options:
--classpath=CLASSPATH Additional class path entries
-cp CLASSPATH (default: ./war/WEB-INF/classes).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
--war=WAR_PATH The path to a directory or .war with a WAR
-w WAR_PATH directory layout (default: ./war).
gen-discovery-doc: Generates API Discovery document
usage: gen-discovery-doc
Options:
--format=FORMAT The requested API protocol type (rest|rpc)
-f FORMAT (default: rest).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
gen-client-lib: Generates a client library
usage: gen-client-lib
Options:
--language=LANGUAGE The target output programming language
-l LANGUAGE (java) (default: java).
--output=OUTPUT_DIR The directory to store output files
-o OUTPUT_DIR (default: ./).
Then you are able to view the services in the explorer UI as the following.
when you access the http: //localhost:8080/_ah/api/explorer, it will be redirected to the developers.google.com first. But the contents is from the localhost.
You can always rely on your normal Google APE local development server. For example, IntelliJ Google APE plugin start the Google APE local server also can support the "_ah/api/explorer" access
As mentioned by Dan Holevoet in the comment, the script needs to be ran from war. As for my case, I just need to copy the files into that directory.

"ImportError: No module named _ssl" with dev_appserver.py from Google App Engine

Background
"In the Python runtime, we've added support for the Python SSL
Library, so you can now open secure connections to remote services
such as Apple's Push Notification service."
This quote is taken from a recent post on the Google App Engine blog.
Implementation
If you want to use native python ssl, you must enable it using the libraries configuration in your application's app.yaml file where you specify the library name "ssl" . . .
These instructions are provided for developers through the Google App Engine documentation.
The following lines have been added to the app.yaml file:
libraries:
- name: ssl
version: latest
This much is in line with the advice provided through the Google App Engine documentation.
Problem
I have tried running my project in three different configurations. Two are working, and one is not.
Working ...
After I upload my application to Google App Engine, and run my project through the live server, everything works fine.
Working ...
When I run my project with manage.py runserver and include the Google App Engine SKD in my PYTHONPATH, everything works fine.
Not Working ...
However, when I run my project with dev_appserver.py, I get the following error:
ImportError at /
No module named _ssl
Request Method: GET
Request URL: http://localhost:8080/
Django Version: 1.4.3
Exception Type: ImportError
Exception Value:
No module named _ssl
Exception Location: /usr/local/lib/google_appengine_1.7.7/google/appengine/tools/devappserver2/python/sandbox.py in load_module, line 856
Python Executable: /home/rbose85/Code/venvs/appserver/bin/python
Python Version: 2.7.3
Python Path:
['/home/rbose85/Code/product/site',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7/lib/protorpc',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7/lib/protorpc',
'/usr/local/lib/google_appengine_1.7.7',
'/usr/local/lib/google_appengine_1.7.7/lib/protorpc',
'/home/rbose85/Code/venvs/appserver/lib/python2.7',
'/home/rbose85/Code/venvs/appserver/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/local/lib/google_appengine',
u'/usr/local/lib/google_appengine_1.7.7/lib/django-1.4',
u'/usr/local/lib/google_appengine_1.7.7/lib/ssl-2.7',
u'/usr/local/lib/google_appengine_1.7.7/lib/webapp2-2.3',
u'/usr/local/lib/google_appengine_1.7.7/lib/webob-1.1.1',
u'/usr/local/lib/google_appengine_1.7.7/lib/yaml-3.10']
Server time: Wed, 24 Apr 2013 11:23:49 +0000
For the current GAE version (1.8.0 at least until 1.8.3), if you want to be able to debug SSL connections in your development environment, you will need to tweak a little bit the gae sandbox:
add "_ssl" and "_socket" keys to the dictionary _WHITE_LIST_C_MODULES in /path-to-gae-sdk/google/appengine/tools/devappserver2/python/sandbox.py
Replace the socket.py file provided by google in /path-to-gae-sdk/google/appengine/dis27 from the socket.py file from your Python framework.
IMPORTANT: Tweaking the sandbox environment might end up with functionality working on your local machine but not in production (for example, GAE only supports outbound sockets in production). I will recommend you to restore your sandbox when you are done developing that specific part of your app.
The solution by jmg works, but instead of changing the sdk files, you could monkey patch the relevant modules.
Just put something like this on the beginning of your project setup.
# Just taking flask as an example
app = Flask('myapp')
if environment == 'DEV':
import sys
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
from lib import copy_of_stdlib_socket.py as patched_socket
sys.modules['socket'] = patched_socket
socket = patched_socket
I had to use a slightly different approach to get this working in CircleCI (unsure what peculiarity about their venv config caused this):
appengine_config.py
import os
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
I had this problem because I wasn't vendoring ssl in my app.yaml file. I know the OP did that, but for those landing here for the OP's error, it's worth making sure lines like the following are in your app.yaml file:
libraries:
- name: ssl
version: latest
Stumbled upon this thread trying to work with Apples Push notification service and appengine... I was able to get this working without any monkey patching, by adding the SSL library in my app.yaml, as recommended in the official docs, hope that helps someone else :)
I added the code to appengine_config.py as listed by Spain Train, but had to also add the following code as well to get this to work:
phttplib = os.path.join(os.path.dirname(real_os_src_path), 'httplib.py')
imp.load_source('httplib', phttplib)
You can test if ssl is available at your local system by opening a python shell and typing import ssl. If no error appears then the problem is something else, otherwise you don't have the relevant libraries installed on your system. If you are using a Linux operating system try sudo apt-get install openssl openssl-devel or the relevant instructions for your operating system to install them locally. If you are using windows, these are the instructions.

Resources