Volttron enters what seems like an almost endless loop - volttron

Question

I have installed about 40 agents to run some simulation. But I run into some threading errors. The errors are shown below.
2019-08-06 10:56:24,120 (socialagent-3.0 16231) ERROR:
Exception in thread Thread-31: 2019-08-06 10:56:24,124
(socialagent-3.0 16231) ERROR: Traceback (most recent call
last): 2019-08-06 10:56:24,135 (socialagent-3.0 16231) ERROR:
File "/usr/lib/python2.7/threading.py", line 801, in
__bootstrap_inner
2019-08-06 10:56:24,162 (socialagent-3.0 16231) ERROR:
self.run()
2019-08-06 10:56:24,177 (socialagent-3.0 16231) ERROR: File
"/usr/lib/python2.7/threading.py", line 1073, in run
2019-08-06 10:56:24,198 (socialagent-3.0 16231) ERROR:
self.function(*self.args, **self.kwargs)
2019-08-06 10:56:24,201 (socialagent-3.0 16231) ERROR: File
"/home/jenny/.volttron/agents/50b42ee5-b6af-4a25-aa89-97e2b1a823e1/socialagent-3.0/social/agent.py",
line 145, in enter_phase3
2019-08-06 10:56:24,240 (socialagent-3.0 16231) ERROR:
self.vip.pubsub.publish('pubsub', ids_topic, message=message)
2019-08-06 10:56:24,265 (socialagent-3.0 16231) ERROR: File
"/home/jenny/workspace/volttron/volttron/platform/vip/agent/subsystems/pubsub.py",
line 607, in publish
2019-08-06 10:56:24,267 (socialagent-3.0 16231) ERROR:
self.vip_socket.send_vip(b'', 'pubsub', frames, result.ident,
copy=False)
2019-08-06 10:56:24,279 (socialagent-3.0 16231) ERROR: File
"/home/jenny/workspace/volttron/volttron/platform/vip/socket.py", line
404, in send_vip
2019-08-06 10:56:24,326 (socialagent-3.0 16231) ERROR:
with self._sending(flags) as flags:
2019-08-06 10:56:24,327 (socialagent-3.0 16231) ERROR: File
"/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2019-08-06 10:56:24,377 (socialagent-3.0 16231) ERROR:
return self.gen.next()
2019-08-06 10:56:24,402 (socialagent-3.0 16231) ERROR: File
"/home/jenny/workspace/volttron/volttron/platform/vip/green.py", line
78, in _sending
2019-08-06 10:56:24,413 (socialagent-3.0 16231) ERROR:
while not lock.acquire(not flags & NOBLOCK):
2019-08-06 10:56:24,422 (socialagent-3.0 16231) ERROR: File
"/home/jenny/workspace/volttron/env/local/lib/python2.7/site-packages/gevent/lock.py",
line 210, in acquire
2019-08-06 10:56:24,427 (socialagent-3.0 16231) ERROR: rc
= self._block.acquire(blocking)
2019-08-06 10:56:24,442 (socialagent-3.0 16231) ERROR: File
"gevent/_semaphore.py", line 198, in
gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:4117)
2019-08-06 10:56:24,451 (socialagent-3.0 16231) ERROR:
def acquire(self, blocking=True, timeout=None):
2019-08-06 10:56:24,485 (socialagent-3.0 16231) ERROR: File
"gevent/_semaphore.py", line 226, in
gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:3944)
2019-08-06 10:56:24,493 (socialagent-3.0 16231) ERROR:
timeout = self._do_wait(timeout)
2019-08-06 10:56:24,503 (socialagent-3.0 16231) ERROR: File
"gevent/_semaphore.py", line 166, in
gevent._semaphore.Semaphore._do_wait (gevent/gevent._semaphore.c:3178)
2019-08-06 10:56:24,546 (socialagent-3.0 16231) ERROR:
result = get_hub().switch()
2019-08-06 10:56:24,552 (socialagent-3.0 16231) ERROR: File
"/home/jenny/workspace/volttron/env/local/lib/python2.7/site-packages/gevent/hub.py",
line 609, in switch
2019-08-06 10:56:24,574 (socialagent-3.0 16231) ERROR:
return greenlet.switch(self)
2019-08-06 10:56:24,589 (socialagent-3.0 16231) ERROR:
LoopExit: ('This operation would block forever', )
2019-08-06 10:56:24,640 (socialagent-3.0 16231) ERROR:
This is just what I have on line 145 in the agent.py file:
self.vip.pubsub.publish('pubsub', ds_topic, message=message)
I have already defined a message as message = “something”
When I install fewer of the agents, I don't run into this problem at all. I'm running a 1.9GHz Intel core i3 machine; could this be related? In order to give as much information as would be helpful in diagnosing the problem, I have shown some snippets of my agent here:
from threading import Timer
#Core.receiver("onstart")
def starting(self, sender, **kwargs):
_log.debug('Subscribing to all relevant topics')
self.vip.pubsub.subscribe('pubsub', pred_topic, callback=self.from_pred)
def from_pred(self, peer, sender, bus, topic, headers, message):
message_dict = message_parser.mess_conv(message)
message = "{}: {}".format(statID, message_dict['feat_pro'])
# Publish to all neighbour topics and to our own topic
for k in neighbours:
self.vip.pubsub.publish('pubsub', k, message=message)
self.vip.pubsub.publish('pubsub', social_topic, message=message)
# ...and start timer to allow communication for 8s
timer = Timer(8, self.enter_phase)
timer.start()
def enter_phase(self):
# Send a message to IDS to begin phase
_log.debug("In agent {}. We have {}".format(stationID, sent_prob))
if stationID in sent_prob.keys():
message = "phase3: 1"
else:
message = "phase3: 0"
self.vip.pubsub.publish('pubsub', ds_topic, message=message)
Also, could timer = Timer(8, self.enter_phase) be the/part of the problem? I'm not such a dev guru and will appreciate any help (in simple language :)).

Answer 1

This could happen due to monkey patch issues. VOLTTRON platform code uses gevent patched socket and thread that is not patched.
Does your agent do a monkey.patchall() or does it import modules that do gevent monkey.patchall()? Does your agent depend on packages/other libraries that expect unpatched Socket? If your agent needs unpatched Socket module you can do a reload(socket) after import Socket statement in your code. VOLTTRON does this for mongo historian(see volttron/platform/dbutils/mongoutils.py) since pymongo library requires unpatch socket and thread modules.