I'm following the docs and yet it appears the requests are still being made synchronously.
https://cloud.google.com/appengine/docs/standard/python/issue-requests
Here is my code:
rpcs = []
for url in urls:
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, url)
rpcs.append(rpc)
result = []
for rpc in rpcs:
result.append(rpc.get_result().content)
return result
I did some profiling and compared using requests.get and they both take exactly the same amount of time.
The urls i'm fetching are from different sites so I'm sure that I don't have concurrent limitations on the server side.
Running on GAE Standard, Python 2.7
I got it working but for some reason only with callbacks. Also It only works on production and not on local env. :D. Here is the working code:
from google.appengine.api import urlfetch
import functools
class ClassName(object):
responses = []
def fetch_concurrent_callback(self, rpc):
response = rpc.get_result()
json_response = json.loads(response.content)
self.responses.append(json_response)
def fetch_concurrent(self, urls):
rpcs = []
for url in urls:
rpc = urlfetch.create_rpc()
rpc.callback = functools.partial(self.fetch_concurrent_callback, rpc)
urlfetch.make_fetch_call(rpc, url)
rpcs.append(rpc)
for rpc in rpcs:
rpc.wait()
return self.responses
Related
The code:
package simulations
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class StarWarsBasicExample extends Simulation
{
// 1 Http Conf
val httpConf = http.baseUrl("https://swapi.dev/api/films/")
// 2 Scenario Definition
val scn = scenario("Star Wars API")
.exec(http("Get Number")
.get("4")
.check(jsonPath("$.episode_id")
.saveAs("episodeId"))
)
.exec(session => {
val movie = session("episodeId").as[String]
session.set("episode",movie)
}).pause(4)
.exec(http("$episode")
.get("$episode"))
// 3 Load Scenario
setUp(
scn.inject(atOnceUsers(1)))
.protocols(httpConf)
}
Trying to grab a variable from the first Get request, and inject that variable into a second request, but unable to do so despite using the documentation. There might be something I'm not understanding.
When I use breakpoints, and navigate through the process, it appears the session execution happens AFTER both of the other requests have been completed (by which time is too late). Can't seem to make that session execution happen between the two requests.
Already answered on Gatling's community mailing list.
"$episode" is not correct Gatling Expression Language syntax. "${episode}" is correct.
I am building a web crawler app on Google App Engine. I want to use a post method to pass the variable to Flask. Then, the received variable became the input of my web crawler app. However, Flask only accept single variable from post. If I add another variable in the funciton， Flask would crash.
I have limited knowledge in Flask and Google app engine. I struggled with the problem for several days and your help will be highly appreciated.
Failed function
#server-side function that does not work,with 2 variable passed
#app.route('/bac',methods=['GET', 'POST'])
def bac():
request_json = request.get_json()
filename = request_json["filename"]
url = request_json["url"]
#baseconnect.Baseconnect(url=url,filename=filename).run()
return filename,url
#The function to post on client side
import requests
req = requests.Session()
data = req.post('https://project.appspot.com/bac',json={"filename":"yuan","url":"https:...f5"})
print(data.text)
#output:
Internal server eror 500
Succeeded function
#server-side function that works,with 1 variable passed
#app.route('/bac',methods=['GET', 'POST'])
def bac():
request_json = request.get_json()
filename = request_json["filename"]
#url = request_json["url"]
#baseconnect.Baseconnect(url=url,filename=filename).run()
return filename
#The function to post on client side
import requests
req = requests.Session()
data = req.post('https://project.appspot.com/bac',json={"filename":"yuan"})
print(data.text)
#output:
yuan
Flask seems only accept single variable. What is the problem....
The problem you have here is that Flask only returns Response object, and Flask will consider return filename, url a shortcut of return Response, status or header.
In this case, url becomes the http status code or header, which is obviously not right.
You need flask.jsonify() to return the proper format of so called 'multiple variables'.
Something like this: (only the important part)
# In server-side code
from flask import jsonify
#app.route('/bac',methods=['GET', 'POST'])
def bac():
request_json = request.get_json()
filename = request_json["filename"]
url = request_json["url"]
# Do your logic here
return jsonify({ filename_returned: filename, url_returned: url })
# client-side
import requests
req = requests.Session()
json_data = req.post('https://project.appspot.com/bac',json={"filename":"yuan", "url": "http:xxxxxxx"})
real_data = json.loads(json_data)
# real_data should be the result you want
Following is my App Engine Endpoint. I annotate it as ApiMethod.HttpMethod.GET because I want to be able to make a get call through the browser. The class itself has a few dozen methods understandably. Some of them using POST. But getItems is annotated with GET. When I try to call the url through a browser, I get a 405 error
Error: HTTP method GET is not supported by this URL
The code:
#Api(name = "myserver",
namespace = #ApiNamespace(ownerDomain = "thecompany.com", ownerName = "thecompany", packagePath = ""),
version = "1", description = "thecompany myserver", defaultVersion = AnnotationBoolean.TRUE
)
public class myserver {
#ApiMethod(name = "getItems", httpMethod = ApiMethod.HttpMethod.GET)
public CollectionResponse<Item> getItems(#Named("paramId") Long paramId) {
…
return CollectionResponse.<Item>builder().setItems(ItemList).build();
}
}
This is not for localhost, it’s for the real server. Perhaps I am forming the url incorrectly. I have tried a few urls such as
https://thecompanymyserver.appspot.com/_ah/spi/com.thecompany.myserver.endpoint.myserver.getItems/v1/paramId=542246400
https://thecompanymyserver.appspot.com/_ah/spi/myserver/NewsForVideo/v1/542246400
The proper path for this is /_ah/api/myserver/1/getItems. /_ah/spi refers to the backend path, which only takes POST requests of a different format.
Side note: API versions are typical "vX" instead of just "X".
You can use the api explorer to find out whether you're using the correct url. Go to
https://yourprojectid.appspot.com/_ah/api/explorer
this works on the devserver as well:
http://localhost:8080/_ah/api/explorer
Also if you're not planning to use the google javascript api client you should add path="..." to your #ApiMethods, so you are sure about what the path actually is.
what im trying to do is to load different applications (webapp2.WSGIApplication) depending on the request domain.
for example www.domain_1.com should load the application in app1.main.application while www.domain_2.com should load app2.main.appplication.
of course im on the same GAE appid and im using namespaces to separate the apps data.
this works pretty good with 'threadsafe:false' and a runner.py file where a function determines which application to return
it seems that with 'threadsafe:true' the first request loads the wsgiapplication into the instance and further requests dont execute the 'application dispatching' logic any more so the request gets a response from the wrong app.
im using python2.7 and webapp2
what is the best way to do this?
edit:
a very simplified version of my runner.py
def main():
if domain == 'www.mydomain_1.com':
from app_1 import application
namespace = 'app_1'
elif domain == 'www.domain_2.com':
from app_2 import application
namespace = 'app_2'
namespace_manager.set_namespace(namespace)
return wsgiref.handlers.CGIHandler().run(application)
if __name__ == '__main__':
main()
and in app.yaml
- url: /.*
script: app-runner.py
Your runner script is a CGI script. The full behavior of a CGI script with multithreading turned on is not documented, and the way the docs are written I'm guessing this won't be supported fully. Instead, the docs say you must refer to the WSGI application object directly from app.yaml, using the module path to a global variable containing the object, when multithreading is turned on. (CGI scripts retain their old behavior in Python 2.7 with multithreading turned off.)
The behavior you're seeing is explained by your use of imports. Within a single instance, each import statement only has an effect the first time it is encountered. After that, the module is assumed to be imported and the import statement has no effect on subsequent requests. You can import both values into separate names, then call run() with the appropriate value.
But if you want to enable multithreading (and that's a good idea), your dispatcher should be a WSGI application itself, stored in a module global referred to by app.yaml. I don't know offhand how to dispatch a request to another WSGI application from within a WSGI application, but that might be a reasonable thing to do. Alternatively, you might consider using or building a layer above WSGI to do this dispatch.
made it happen by subclassing webapp2.WSGIApplication and overriding __call__() which is called before dispatching to a RequestHandler.
prefixing routes (and removing the prefix in the handlers initialize) and substructuring config to be able to use the instance memory.
class CustomWSGIApplication(webapp2.WSGIApplication):
def __call__(self, environ, start_response):
routes, settings, ns = get_app(environ)
namespace_manager.set_namespace(ns)
environ['PATH_INFO'] = '/%s%s' %(ns, environ.get('PATH_INFO'))
for route in routes:
r, h = route # returns a tuple with mapping and handler
newroute = ('/%s%s'%(ns, r), h,)
self.router.add(newroute)
if settings:
self.config[ns] = settings
self.debug = debug
with self.request_context_class(self, environ) as (request, response):
try:
if request.method not in self.allowed_methods:
# 501 Not Implemented.
raise exc.HTTPNotImplemented()
rv = self.router.dispatch(request, response)
if rv is not None:
response = rv
except Exception, e:
try:
# Try to handle it with a custom error handler.
rv = self.handle_exception(request, response, e)
if rv is not None:
response = rv
except HTTPException, e:
# Use the HTTP exception as response.
response = e
except Exception, e:
# Error wasn't handled so we have nothing else to do.
response = self._internal_error(e)
try:
return response(environ, start_response)
except Exception, e:
return self._internal_error(e)(environ, start_response)
I use Amazon web service api from within my Google app engine application. Amazon have said that they will only accept signed requests from Aug 15, 2009. While they have given simple instructions for signing, I am not so knowledgeable of Python libraries for SHA256. The app engine documentation says it supports pycrypto but I was just wondering (read being lazy) if anyone has already done this. Any code snippets you could share? Any issues I might be missing here?
Here is an example of a REST request based on lower level (then boto) libraries. Solution was taken from http://cloudcarpenters.com/blog/amazon_products_api_request_signing.
All you need is valid entries for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
def amazon_test_url():
import base64, hashlib, hmac, time
from urllib import urlencode, quote_plus
AWS_ACCESS_KEY_ID = 'YOUR_KEY'
AWS_SECRET_ACCESS_KEY = 'YOUR_SECRET_KEY'
TEST_ISBN = '9780735619678' #http://stackoverflow.com/questions/1711/what-is-the-single-most-influential-book-every-programmer-should-read
base_url = "http://ecs.amazonaws.com/onca/xml"
url_params = dict(
Service='AWSECommerceService',
Operation='ItemLookup',
IdType='ISBN',
ItemId=TEST_ISBN,
SearchIndex='Books',
AWSAccessKeyId=AWS_ACCESS_KEY_ID,
ResponseGroup='Images,ItemAttributes,EditorialReview,SalesRank')
#Can add Version='2009-01-06'. What is it BTW? API version?
# Add a ISO 8601 compliant timestamp (in GMT)
url_params['Timestamp'] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
# Sort the URL parameters by key
keys = url_params.keys()
keys.sort()
# Get the values in the same order of the sorted keys
values = map(url_params.get, keys)
# Reconstruct the URL parameters and encode them
url_string = urlencode(zip(keys,values))
#Construct the string to sign
string_to_sign = "GET\necs.amazonaws.com\n/onca/xml\n%s" % url_string
# Sign the request
signature = hmac.new(
key=AWS_SECRET_ACCESS_KEY,
msg=string_to_sign,
digestmod=hashlib.sha256).digest()
# Base64 encode the signature
signature = base64.encodestring(signature).strip()
# Make the signature URL safe
urlencoded_signature = quote_plus(signature)
url_string += "&Signature=%s" % urlencoded_signature
print "%s?%s\n\n%s\n\n%s" % (base_url, url_string, urlencoded_signature, signature)
Pycrypto will work fine - it's supported on App Engine, though the public ciphers are implemented in Python rather than C. You also ought to be able to use one of the existing AWS libraries, now that urlfetch/httplib are supported on App Engine.
I have an app that uploads images to S3, and I've implemented the request signing myself, but mostly because I wrote it before urlfetch/httplib were available. It works just fine, however.
Got this to work based on code sample at http://jjinux.blogspot.com/2009/06/python-amazon-product-advertising-api.html
Here is a minor improved version that lets you merge a dict of call specific params with the basic params before making the call.
keyFile = open('accesskey.secret', 'r')
# I put my secret key file in .gitignore so that it doesn't show up publicly
AWS_SECRET_ACCESS_KEY = keyFile.read()
keyFile.close()
def amz_call(self, call_params):
AWS_ACCESS_KEY_ID = '<your-key>'
AWS_ASSOCIATE_TAG = '<your-tag>'
import time
import urllib
from boto.connection import AWSQueryConnection
aws_conn = AWSQueryConnection(
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=Amz.AWS_SECRET_ACCESS_KEY, is_secure=False,
host='ecs.amazonaws.com')
aws_conn.SignatureVersion = '2'
base_params = dict(
Service='AWSECommerceService',
Version='2008-08-19',
SignatureVersion=aws_conn.SignatureVersion,
AWSAccessKeyId=AWS_ACCESS_KEY_ID,
AssociateTag=AWS_ASSOCIATE_TAG,
Timestamp=time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime()))
params = dict(base_params, **call_params)
verb = 'GET'
path = '/onca/xml'
qs, signature = aws_conn.get_signature(params, verb, path)
qs = path + '?' + qs + '&Signature=' + urllib.quote(signature)
print "verb:", verb, "qs:", qs
return aws_conn._mexe(verb, qs, None, headers={})
Sample usage:
result = self.amz_call({'Operation' : 'ItemSearch' , 'Keywords' : searchString , 'SearchIndex' : 'Books' , 'ResponseGroup' : 'Small' })
if result.status == 200:
responseBodyText = result.read()
# do whatever ...
See http://sowacs.appspot.com/AWS/Downloads/#python for a GAE Python signing service webapp. Uses native Python libraries.
I wrote another simple example that uses only the core python 3 libraries (not boto) and uses version 2 of the AWS signature protocol:
http://xocoatl.blogspot.com/2011/03/signing-ec2-api-request-in-python.html
I know it won't work in GAE, but might be useful for anyone just looking for AWS authentication examples like I was.
I use this one using pycrypto to generate a custom policy:
import json
import time
from Crypto.Hash import SHA
from Crypto.PublicKey import RSA
from Crypto.Signature import PKCS1_v1_5
from base64 import b64encode
url = "http://*"
expires = int(time.time() + 3600)
pem = """-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----"""
key_pair_id = 'APK.....'
policy = {}
policy['Statement'] = [{}]
policy['Statement'][0]['Resource'] = url
policy['Statement'][0]['Condition'] = {}
policy['Statement'][0]['Condition']['DateLessThan'] = {}
policy['Statement'][0]['Condition']['DateLessThan']['AWS:EpochTime'] = expires
policy = json.dumps(policy)
private_key = RSA.importKey(pem)
policy_hash = SHA.new(policy)
signer = PKCS1_v1_5.new(private_key)
signature = b64encode(signer.sign(policy_hash))
print '?Policy=%s&Signature=%s&Key-Pair-Id=%s' % (b64encode(policy),
signature,
key_pair_id)
This allows me to use one key for multiple items, something like:
http://your_domain/image1.png?Policy...
http://your_domain/image2.png?Policy...
http://your_domain/file1.json?Policy...
Don't forget to enable pycrypto by adding this lines to the app.yaml
libraries:
- name: pycrypto
version: latest