I am trying to do concurrent JSONRPC calls to remote agents in Volttron. And am using Volttron 5.1.0 (with gevent 1.1.2 & requests 2.11.1).
The code works as expected. However, from the log files, I notice that the requests are not run concurrently. I am not sure what I am missing.
I tried suggestions mentioned in how enable requests async mode? (i.e,, handling async part in the request, monkey patch, etc.,). But none work-out, either the solution is outdated or required modules not available in volttron env (I am a bit worried about version requirements).
Any suggestions or inputs would be of great help.
The relevant part of the agent code is as follows:
Agent Code
import gevent
import requests
url_roots = ['http://192.168.1.51:8080/', 'http://192.168.1.52:8080/']
jobs = [gevent.spawn(do_rpc, self._agent_id, url_root, 'pricepoint'
, get_json_params()
) for url_root in url_roots
]
gevent.joinall(jobs, timeout=11)
def do_rpc(id, url_root, method, params=None):
result = False
json_package = {
'jsonrpc': '2.0',
'id': id,
'method':method,
}
json_package['params'] = params
response = requests.post(url_root, data=json.dumps(json_package), timeout=10)
if response.ok:
if 'result' in response.json().keys():
success = response.json()['result']
if success:
result = True
return result
Log
2020-02-19 21:12:15,913 (xyzagent-0.4 28079) xyz.ispace_msg_utils DEBUG: validate_bustopic_msg()
2020-02-19 21:12:15,918 (xyzagent-0.4 28079) xyz.agent DEBUG: New price point (pp) msg on the local-bus, topic: building/pricepoint ...
2020-02-19 21:12:15,919 (xyzagent-0.4 28079) xyz.agent DEBUG: ***** New bid price point from local: 0.20 price_id: 2218566
2020-02-19 21:12:15,931 (xyzagent-0.4 28079) xyz.agent DEBUG: post_ds_new_pp()...
2020-02-19 21:12:15,932 (xyzagent-0.4 28079) xyz.agent DEBUG: us pp messages count: 1...
2020-02-19 21:12:15,933 (xyzagent-0.4 28079) xyz.agent DEBUG: processing pp msg 1/1, price id: 2218566
2020-02-19 21:12:15,938 (xyzagent-0.4 28079) xyz.agent DEBUG: new ttl: 28.
2020-02-19 21:12:15,942 (xyzagent-0.4 28079) xyz.agent DEBUG: _ds_rpc_1_to_m()...
2020-02-19 21:12:15,953 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.51
2020-02-19 21:12:16,079 () volttron.platform.web DEBUG: {'jsonrpc': '2.0', 'id': '2503402', 'result': True}
2020-02-19 21:12:16,080 () volttron.platform.web DEBUG: res is a dictionary.
2020-02-19 21:12:16,238 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-19 21:12:16,245 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.52
2020-02-19 21:12:16,526 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-19 21:12:16,529 (xyzagent-0.4 28079) xyz.agent DEBUG: post pp to ds (ZoneController-51), result: success!!!
2020-02-19 21:12:16,529 (xyzagent-0.4 28079) xyz.agent DEBUG: post pp to ds (ZoneController-52), result: success!!!
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: _ds_rpc_1_to_m()...done
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: msg successfully posted to all ds, removing it from the queue
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: reset the retry counter for success ds msg
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: post_ds_new_pp()...done
The below changes did the trick. Please note that just monkey.patch_all() breaks volttron. Need to set flag thread=False.
Agent Code
import gevent
from gevent import monkey
monkey.patch_all(thread=False, select=False)
import requests
Logs
2020-02-20 14:36:29,981 (xyzagent-0.4 8657) xyz.agent DEBUG: _ds_rpc_1_to_m()...
2020-02-20 14:36:29,987 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.51
2020-02-20 14:36:29,992 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.52
2020-02-20 14:36:30,260 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-20 14:36:30,333 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-20 14:36:30,341 (xyzagent-0.4 8657) xyz.agent DEBUG: post pp to ds (ZoneController-51), result: success!!!
2020-02-20 14:36:30,342 (xyzagent-0.4 8657) xyz.agent DEBUG: post pp to ds (ZoneController-52), result: success!!!
2020-02-20 14:36:30,344 (xyzagent-0.4 8657) xyz.agent DEBUG: _ds_rpc_1_to_m()...done
Related
I want to iterate over array and pass each array element value to role from playbook but it is not working in ansible, Can some one help
---
#play book
- name: create config for instance
hosts: all
vars:
LIST: [Asia, Americas, Artic, Antartic ,Oceania,Europe,Africa]
connection: local
roles:
- role: create_config
debug:
msg : "{{ item }}"
vars:
VENUE: "{{ item }}"
with_items:
- "{{ LIST }}"
## Role
- name: create directory structure
file:
path: "{{item}}"
state: directory
mode: 0755
with_items:
- "{{dest_folder}}/{{instance_name}}/{{VENUE}}"
I am getting below error
ansible-playbook -i inventory/AlgoTest_SP create_pkg_1.yml
PLAY [create config for instance] **************************************************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************************************************
ok: [localhost]
TASK [create_config : create directory structure] *******************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "{{ item }}: 'item' is undefined"}
PLAY RECAP ****************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
To be able to loop over roles, you need the include_role task, as in:
- name: create config for instance
hosts: localhost
vars:
LIST: [Asia, Americas, Artic, Antartic ,Oceania,Europe,Africa]
tasks:
- include_role:
name: create_config
vars:
VENUE: "{{ item }}"
with_items:
- "{{LIST}}"
cat roles/create_config/tasks/main.yml
- debug:
msg: "{{VENUE}}"
- name: create directory structure
file:
path: "{{item}}"
state: directory
mode: 0755
with_items:
- "{{inventory_hostname}}/{{VENUE}}"
Resulting in:
$ ansible-playbook 69045121.yml
PLAY [create config for instance] ********************************************************************************************************************************************************************************************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [include_role : create_config] ******************************************************************************************************************************************************************************************************************************************************************************************
TASK [create_config : debug] *************************************************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "Asia"
}
TASK [create_config : create directory structure] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [localhost] => (item=localhost/Asia)
TASK [create_config : debug] *************************************************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "Americas"
}
[...] and so on
As a final note, you could, and should, likely handle this differently. Also, you have clash in the item variable name from the outer (playbook) and inner (role) with_items, you can use loop_var to set a different looping varname.
I can get information for Individual Package Version like this
- name: Print zsh Version
debug:
msg: "{{ ansible_facts.packages['zsh'][0].version }}"
when: " 'zsh' in ansible_facts.packages"
I am trying to use a loop for a list, but I am unable to quote the {{item}}.
software: ['ksh','zsh','bash']
- name: Print Softwre Versions
debug:
msg: "{{ ansible_facts.packages['{{item}}'][0].version }}"
with_items: "{{ software }}"
I get the following error message
"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute '{{item}}'
How do I make this work ?
You don't need to quote it or put it in curly bracers, you are already in curly bracers:
- name: Print software versions
debug:
msg: "{{ ansible_facts.packages[item][0].version }}"
vars:
software:
- 'ksh'
- 'zsh'
- 'bash'
loop: "{{ software }}"
Fully working playbook:
- hosts: localhost
gather_facts: no
tasks:
- name: Gather package facts
package_facts:
manager: auto
- name: Print software versions
debug:
msg: "{{ ansible_facts.packages[item][0].version }}"
vars:
software:
- 'ksh'
- 'zsh'
- 'bash'
loop: "{{ software }}"
Gives this recap:
PLAY [localhost] ***************************************************************
TASK [Gather package facts] ****************************************************
ok: [localhost]
TASK [Print software versions] *************************************************
ok: [localhost] => (item=ksh) => {
"msg": "2020.0.0-5"
}
ok: [localhost] => (item=zsh) => {
"msg": "5.8-3ubuntu1"
}
ok: [localhost] => (item=bash) => {
"msg": "5.0-6ubuntu1"
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
PS: try not to mix YAML and JSON notation, your software array is in JSON, while the rest of your playbook is in YAML.
When I send a task to a task queue it keeps failing and shows a http 404 (not found) error in the logs.
The project has been whitelisted for cloud tasks alpha on flexible.
I can send HTTP post requests to /endpointpath & /tasks/worker locally without any errors.
The endpoint works fine and adds the task to the task queue.
13:37:41.300 POST 200 0 B 422 ms curl/7.54.0 /endpointspath?key=keyremoved 0.0.0.0 - "POST endpointspath?key=keyremoved" 200 undefined "-" "curl/7.54.0"
The app is running as the default service.
app.go main func:
func main() {
r := mux.NewRouter()
r.HandleFunc("/", handler)
r.HandleFunc("/_ah/health", healthCheckHandler)
// Task handlers
r.Path("/tasks/worker").Methods("POST", "GET", "PUT").HandlerFunc(workerTaskHandler)
// Endpoints
r.Path("/endpointpath").Methods("POST").HandlerFunc(searchHandler)
http.Handle("/", r)
port := 8080
if portStr := os.Getenv("PORT"); portStr != "" {
port, _ = strconv.Atoi(portStr)
}
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", port), nil))
}
abbreviated app.yaml:
runtime: go
env: flex
handlers:
- url: /tasks/.*
script: _go_app
- url: /.*
script: _go_app
Logs HTTP 404 response when queue dispatches request to worker:
10.0.0.1 - "POST /tasks/worker" 404 200 "-" "AppEngine-Google; (+http://code.google.com/appengine)"
Expand all | Collapse all {
httpRequest: {
latency: "0s"
referer: "-"
remoteIp: "10.0.0.1"
requestMethod: "POST"
requestUrl: "/tasks/worker"
responseSize: "200"
status: 404
userAgent: "AppEngine-Google; (+http://code.google.com/appengine)"
}
insertId: "......."
jsonPayload: {
appLatencySeconds: "-"
latencySeconds: "0.000"
time: null
trace: "......."
}
labels: {
appengine.googleapis.com/instance_name: "......"
appengine.googleapis.com/trace_id: "......."
compute.googleapis.com/resource_id: "......."
compute.googleapis.com/resource_name: "......"
compute.googleapis.com/zone: "us-central1-b"
}
logName: "projects/projectname/logs/appengine.googleapis.com%2Fnginx.request"
receiveTimestamp: "2017-12-09T10:56:14.794726383Z"
resource: {
labels: {
module_id: "default"
project_id: "projectname"
version_id: "....."
}
type: "gae_app"
}
timestamp: "2017-12-09T10:56:10.301Z"
}
The closest I can get GAE to find the tasks/worker url is by setting login:admin in app.yaml (even tho flex doesn't use this for authentication). This returns a 403 unauthorised error.
handlers:
- url: /tasks/.*
script: _go_app
login: admin
Here is the 403 response in the logs
{
httpRequest: {
latency: "0s"
referer: "-"
remoteIp: "10.0.0.1"
requestMethod: "POST"
requestUrl: "/tasks/worker"
responseSize: "162"
status: 403
userAgent: "AppEngine-Google; (+http://code.google.com/appengine)"
}
insertId: "....."
jsonPayload: {
appLatencySeconds: "-"
latencySeconds: "0.000"
time: null
trace: "....."
}
labels: {
appengine.googleapis.com/instance_name: "...."
appengine.googleapis.com/trace_id: "...."
compute.googleapis.com/resource_id: "...."
compute.googleapis.com/resource_name: "....."
compute.googleapis.com/zone: "us-central1-b"
}
logName: "projects/projectname/logs/appengine.googleapis.com%2Fnginx.request"
receiveTimestamp: "2017-12-09T13:35:59.986118082Z"
resource: {
labels: {
module_id: "default"
project_id: "projectname"
version_id: "....."
}
type: "gae_app"
}
timestamp: "2017-12-09T13:35:54.764Z"
}
Not sure if it's related but projectname.appspot.com/_ah/health returns this error:
{
"code": 5,
"message": "Method does not exist.",
"details": [
{
"#type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "service_control"
}
]
}
It turns out endpoints can't run on the same service as task handlers. Task handler url requests are blocked by the ESP proxy if they run on the same service in the flexible environment, and the service has the endpoints service enabled.
Run task handlers on a separate service and do not set "endpoints_api_service:" in the task handler service app.yaml file.
Doing so will prevent the queue from being able to dispatch to workers in the flexible environment.
This isn't mentioned in the app engine documentation which is kinda bizarre.
The "/_ah/health" issue was caused by this path not being set in the open api file. If this path isn't set the url isn't recognised by the proxy.
I've implemented ExpressCheckoutDG in cakePHP3.x website, based on the ExpressCheckout Wizard. (https://devtools-paypal.com/integrationwizard/)
So the payment procedure is realized in a dedicated opened frame.
Everything goes well until payment is achived and Paypal call the return URL.
At the end of the called method named confirm, I don't know what to do to close the Paypal frame and go back to a specified url.
My confirm method is:
public function confirm() {
$this->log($this->request->url . ' confirm', 'debug' );
$this->loadModel('Orders');
$PaymentOption = "PayPal";
if ( $PaymentOption == "PayPal" )
{
$res = $this->GetExpressCheckoutDetails( $_REQUEST['token'] );
/**
* I removed this part of code as it doesn't concern the problem
*/
$resArray = $this->ConfirmPayment ( $token, $paymentType, $currencyCodeType, $payerID, $finalPaymentAmount, $items );
$ack = strtoupper($resArray["ACK"]);
$this->log($this->request->url . ' confirm :' . $ack, 'debug' );
if( $ack == "SUCCESS" || $ack == "SUCCESSWITHWARNING" )
{
/**
* I removed this part of code as it doesn't concern the problem
*/
/*
* Here I save the transaction
*/
// Add javascript to close Digital Goods frame. You may want to add more javascript code to
// display some info message indicating status of purchase in the parent window
$this->Flash->success(__("transaction successfully completed"));
$this->log($this->request->url . ' confirm : display confirm.ctp', 'debug' );
/*
* So the problem is here: What to do to close the Paypal frame, AND return to a given page of my website??
*/
//$this->redirect(['controller' => 'Sites', 'action' => 'view']);
$this->set(compact('ack'));
}
else
{
//Display a user friendly Error on the page using any of the following error information returned by PayPal
$ErrorCode = urldecode($resArray["L_ERRORCODE0"]);
$ErrorShortMsg = urldecode($resArray["L_SHORTMESSAGE0"]);
$ErrorLongMsg = urldecode($resArray["L_LONGMESSAGE0"]);
$ErrorSeverityCode = urldecode($resArray["L_SEVERITYCODE0"]);
echo "DoExpressCheckoutDetails API call failed. ";
echo "Detailed Error Message: " . $ErrorLongMsg;
echo "Short Error Message: " . $ErrorShortMsg;
echo "Error Code: " . $ErrorCode;
echo "Error Severity Code: " . $ErrorSeverityCode;
$this->Flash->error(__("votre achat n'a pas été accepté"));
$this->set(compact('ack'));
}
}
$this->set(compact('ack'));
}
As you can see, the payment process goes well until trying to finish:
2015-02-20 11:00:38 Debug: pros/Sitemessages/checkout checkout
2015-02-20 11:00:38 Debug: pros/Sitemessages/checkout SetExpressCheckoutDG
2015-02-20 11:00:38 Debug: pros/Sitemessages/checkout hash_call
2015-02-20 11:00:41 Debug: pros/Sitemessages/checkout deformatNVP
2015-02-20 11:00:41 Debug: pros/Sitemessages/checkout deformatNVP
2015-02-20 11:00:41 Debug: pros/Sitemessages/checkout hash_call: closing
2015-02-20 11:00:41 Debug: pros/Sitemessages/checkout RedirectToPayPalDG
2015-02-20 11:01:15 Debug: pros/Sitemessages/confirm confirm
2015-02-20 11:01:15 Debug: pros/Sitemessages/confirm GetExpressCheckoutDetails
2015-02-20 11:01:15 Debug: pros/Sitemessages/confirm hash_call
2015-02-20 11:01:18 Debug: pros/Sitemessages/confirm deformatNVP
2015-02-20 11:01:18 Debug: pros/Sitemessages/confirm deformatNVP
2015-02-20 11:01:18 Debug: pros/Sitemessages/confirm hash_call: closing
2015-02-20 11:01:18 Debug: pros/Sitemessages/confirm ConfirmPayment
2015-02-20 11:01:18 Debug: pros/Sitemessages/confirm hash_call
2015-02-20 11:01:23 Debug: pros/Sitemessages/confirm deformatNVP
2015-02-20 11:01:23 Debug: pros/Sitemessages/confirm deformatNVP
2015-02-20 11:01:23 Debug: pros/Sitemessages/confirm hash_call: closing
2015-02-20 11:01:23 Debug: pros/Sitemessages/confirm confirm :SUCCESS
2015-02-20 11:01:23 Notice: pros/Sitemessages/confirm transaction successfully saved
2015-02-20 11:01:23 Debug: pros/Sitemessages/confirm confirm : display confirm.ctp
here is my confirm.ctp file:
<?php if ( $ack == "SUCCESS" || $ack == "SUCCESSWITHWARNING" ):?>
<script>
alert("Payment Successful");
// add relevant message above or remove the line if not required
window.onload = function(){
if(window.opener){
window.close();
}
else{
if(top.dg.isOpen() == true){
top.dg.closeFlow();
return true;
}
}
};
</script>
<?php else:?>
<script>
alert("Payment failed");
// add relevant message above or remove the line if not required
window.onload = function(){
if(window.opener){
window.close();
}
else{
if(top.dg.isOpen() == true){
top.dg.closeFlow();
return true;
}
}
};
</script>
<?php endif;?>
Any idea on the solution to close the Paypal Frame and go back to a given url?
You need to have a page with nothing on it except for the javascript to close the window. That's what you would set as your return URL and your cancel URL, so that the window simply closes.
Here's a demo I put together using my PHP class library for PayPal that will show you how to make it work.
Sorry, I found my mistake.
I had to add the following line at the begining of the ctp file.
<?php $this->layout=false?>
The following code generates a CGI method instead of WSGI as indicated in the log below. Is this the normal execution for a WSGI app on the dev server? If not what needs to change to get the application to execute as WSGI?
main.py
import webapp2
import wsgiref.handlers
import logging
from google.appengine.api import users
class HomeHandler(webapp2.RequestHandler):
def get(self):
self.response.out.write("Hi World")
app = webapp2.WSGIApplication([
(r'/', HomeHandler),
], debug=True)
app.run()
app.yaml
application: WSGITEST
version: 1
runtime: python27
api_version: 1
threadsafe: yes
libraries:
- name: webapp2
version: latest
handlers:
- url: /.*
script: main.app
Log
DEBUG 2012-05-09 21:31:14,921 dev_appserver.py:656] Matched "/" to CGI dispatcher with path main.app
DEBUG 2012-05-09 21:31:14,926 dev_appserver_import_hook.py:1246] Enabling webapp2: None
DEBUG 2012-05-09 21:31:14,928 dev_appserver.py:1624] Executing CGI with env:
{'REQUEST_ID_HASH': '77DE68DA', 'SERVER_SOFTWARE': 'Development/1.0', 'SCRIPT_NAME': '', 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': '', 'CONTENT_LENGTH': '', 'USER_ID': '', 'APPENGINE_RUNTIME': 'python27', 'TZ': 'UTC', 'SERVER_NAME': 'localhost', 'REMOTE_ADDR': '127.0.0.1', 'SDK_VERSION': '1.6.5', 'PATH_TRANSLATED': '/home/bear/dev/appengine/code/ae-baseapp/401/main3.app', 'SERVER_PORT': '8080', '_AH_THREADSAFE': '1', 'CURRENT_VERSION_ID': '1.1', 'USER_ORGANIZATION': '', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0', 'HTTP_HOST': 'localhost:8080', 'HTTP_CONNECTION': 'keep-alive', 'USER_EMAIL': '', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'APPLICATION_ID': 'dev~WSGITEST', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_DNT': '1', 'CONTENT_TYPE': 'application/x-www-form-urlencoded', '_AH_ENCODED_SCRIPT_NAME': '/', 'AUTH_DOMAIN': 'gmail.com'}
INFO 2012-05-09 21:31:14,933 dev_appserver.py:2891] "GET / HTTP/1.1" 200 -
this log is a hardcoded string:
/usr/local/google_appengine/google/appengine/tools/dev_appserver.py:
1622 sys.modules['__builtin__'] = __builtin__
1623
1624: logging.debug('Executing CGI with env:\n%s', repr(env))
First, you should remove this line of code because it's incorrect:
app.run()
As written in my comment and well pointed out by aschmid00, could be an old debug statement.