Volttron throws an Attribute error when attempting to publish - volttron

I am combining Scapy with Volttron and I want to publish to a topic when a packet comes in and has certain features. However, I keep running into this error:
Traceback (most recent call last): File "sniff.py", line 373, in
sys.exit(main()) File "sniff.py", line 342, in main
utils.vip_main(sniffer, version=version) File "/home/jenny/workspace/volttron/volttron/platform/agent/utils.py",
line 314, in vip_main
version=version, **kwargs) File "sniff.py", line 336, in sniffer
Sniffer(**kwargs) File "sniff.py", line 138, in init
self.vip.pubsub.publish('pubsub', "some/topic", message="blah")
File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/subsystems/pubsub.py",
line 602, in publish
self._save_parameters(result.ident, **kwargs)
File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/subsystems/pubsub.py",
line 706, in _save_parameters
event = self.core().schedule(end_time, self._cancel_event, result_id)
File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/core.py",
line 409, in schedule
self._schedule_callback(deadline, event) File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/core.py",
line 417, in _schedule_callback
self._schedule_event.set()
AttributeError: 'NoneType' object has no attribute 'set'
The closest to a solution I found is here RPC crashes with AttributeError: 'NoneType' object has no attribute 'call'. However, I notice they aren't exactly the same issue and so there isn't much surprise when I tried the solution offered there and it didn't work for me.
My code looks like this:
def sniffer(config_path, **kwargs):
''' Initializations '''
global pkt_counter
global IP_counter
# Defined other parameters here
class Sniffer(Agent):
def __init__(self, **kwargs):
super(Sniffer, self).__init__(**kwargs)
# I am just testing the publish function here
self.vip.pubsub.publish('pubsub', "some/topic", message="blah")
sniff(count=0, iface=conf.iface, prn = self.pkt_action, store=0)
def pkt_action(self, pkt):
#Process every packet, updates values and rises and alert if necessary
# some checks are run here and later a publish is called
Can someone please let me know what I'm doing wrong?
Edit: I didn't add that I want to run this script as I would a simple python script on the terminal (eg. python somescript.py): no installation. The reason I'm trying to do this is that I get an error when I install the agent and start it. The platform doesn't allow Scapy to create and connect to sockets.

You are attempting to use the vip module from in a context that isn't expected. The agent's lifecycle doesn't actually start executing until the start agent or configure events are triggered. Try to put the publish within those contexts.
The following excerpt is from the VOLTTRON readthedocs site located: https://volttron.readthedocs.io/en/develop/devguides/agent_development/Agent-Development-Cheatsheet.html#agent-lifecycle-events
Core Agent Functionality
These tools volttron.platform.vip.agent module. Try importing
Agent Lifecycle Events
Each agent has four events that are triggered at different stages of its life. These > are onsetup, onstart, onstop, and onfinish. Registering callbacks to these events are commonplace in agent development, with onstart being the most frequently used.
The easiest way to register a callback is with a function decorator:
#Core.receiver('onstart')
def function(self, sender, **kwargs):
function_body

Related

Why am I getting a timeout exception with flask on googles app engine when trying to append a value to a wtforms validtor?

I'm having trouble deploying a flask app to the Google app engine. It works fine when I run it locally but as soon as it's deployed to the app engine it fails. With two exceptions (I'm always getting both, I'm unsure which one comes first).
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/logging/handlers.py", line 1008, in emit
smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 338, in connect
(code, msg) = self.getreply()
File "/opt/python3.7/lib/python3.7/smtplib.py", line 391, in getreply
+ str(e))
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out
and
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/smtplib.py", line 387, in getreply
line = self.file.readline(_MAXLINE + 1)
File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
It consistently happens within the __init__ function of a form. And specifically in there when I try to append a validator to a validators list.
self.password.validators.append(InputRequired(message='Please enter password!'))
It is not specific to this one validator, the same thing happens with others too. I am at a complete loss as to why and how appending validators would interact with SMTP or sockets.
One thing I noticed is that I had an issue with the Email validator which also worked fine locally but required me to install the module 'email_validator' to work. When I did that it complained that it couldn't use append on a tuple when I tried to append a validator to - what was a list in my local app - but for some reason turned out to be a tuple when I ran the exact same code on the app engine. I removed the Email validator (and the import/requirements) entirely and now I'm at the problem I've described above. It may or may not be related to the Email validator issue.
Update: The error message does indeed seem to be related to the validators.append() call seeing as by setting up the validators at object creation and avoiding all further changes via validators.append() fixed the problem for me. This is the requirements.txt that goes with the application:
blinker>=1.4
Click>=7.0
Flask>=1.1.1
Flask-Login>=0.5.0
Flask-Mail>=0.9.1
Flask-SQLAlchemy>=2.4.1
Flask-WTF>=0.14.3
itsdangerous>=1.1.0
Jinja2>=2.11.1
MarkupSafe>=1.1.1
PyJWT>=1.7.1
PyMySQL>=0.9.3
python-dotenv>=0.11.0
SQLAlchemy>=1.3.13
Werkzeug>=1.0.0
WTForms>=2.2.1
Instead of a full minimal, reproducible example I can offer a minimized version of the form that was causing the issues when instanciated on the GAE:
class ExampleForm(FlaskForm):
name = StringField('Name')
original_name = HiddenField('Original Name')
def add_validators(self):
names = [a.name for a in Users.query.all()]
if self.original_name.data:
names.remove(self.original_name.data)
self.name.validators.append(InputRequired(message='Please enter name!'))
self.name.validators.append(NoneOf(names, message='Username already exists!'))
def __init__(self):
super().__init__(*args, **kwargs)
self.add_validators()
As I'm writing this it occurs to me that maybe I need to always initialize an empty validators list on creation to ensure new entries can be appended to it (like with a normal list) like this name = StringField('Name', validators=[]). I'd still be confused as to why it works locally, but if it fixes the problem that'd be what really counts for me.

Error while calling cloud framework endpoint api

I am getting below error while accessing the api via explorer '_ah/api/explorer' or when api is called from the app.
Below is relevant piece from app.yaml and api implementation.
From other questions on stackoverflow it looks like this could happen if endpoint config is not deployed using gcloud service-management but event after uploading gatewayv1openapi.json I am getting similar error.
Is there something i need to change in api code or app.yaml? One more thing is that I have uploaded exact same codebase (only difference is other project id instead of ixp-test in app.yaml) to another old project and it works fine. Any newly created project is giving this error.
/_ah/api/gateway/v1/invoke?alt=json
no scheduler thread, scheduler.run() will be invoked by report(...) (/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/client.py:235)
Traceback (most recent call last):
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/client.py", line 231, in start
self._thread.start()
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/background_thread/background_thread.py", line 108, in start
start_new_background_thread(self.bootstrap, ())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/background_thread/background_thread.py", line 87, in start_new_background_thread
raise ERROR_MAPerror.application_error
FrontendsNotSupported
.
.
.
(/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 96, in LoadObject
__import(cumulative_path)
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/main.py", line 18, in
api = endpoints.api_server([GatewayApi])
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints/apiserving.py", line 514, in api_server
controller)
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/wsgi.py", line 126, in add_all
a_service = loader.load()
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/control/service.py", line 111, in load
return self._load_func(**kw)
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/config/service_config.py", line 79, in fetch_service_config
_log_and_raise(Exception, message_template.format(status_code))
File "/base/data/home/apps/s~ixp-test/20171013t215957.404774686885375478/server/lib/endpoints_management/config/service_config.py", line 127, in _log_and_raise
raise exception_class(message)
Exception: Fetching service config failed (status code 403)
Below is the relevant config
app.yaml
runtime: python27
api_version: 1
threadsafe: false
automatic_scaling:
max_idle_instances: 0
env_variables:
ENDPOINTS_SERVICE_NAME: ixp-test.appspot.com
ENDPOINTS_SERVICE_VERSION: 2017-08-13r0
API
#endpoints.api(name='gateway', version='v1')
class GatewayApi(remote.Service):
#endpoints.method(
GatewayRequest,
GatewayResponse,
path='invoke',
http_method='POST',
name='invoke')
def invoke(self, request):
pass
#endpoints.method(
GatewayRequest,
GatewayResponse,
path='invokeSecure',
http_method='POST',
name='invokeSecure',
scopes=[endpoints.EMAIL_SCOPE])
def invoke_secure(self, request):
pass
This might well happen if the service name in the gatewayv1openapi.json doesn't match the service name in the app.yaml; can you double-check that?
Managed to resolve it. Issue was due to not updating
ENDPOINTS_SERVICE_VERSION: 2017-08-13r0
for new project. Value from old deployed project for which it was working caused the error.
from getting started tutorial
gcloud service-management deploy echov1openapi.json
Service Configuration [2017-02-13-r2] uploaded for service [example-project.appspot.com]

No longer able to create an entity using the dev_appserver admin console

I cleared all entities from the datastore and cleared the memcache, created an entity using the API Explorer and then attempted to create another one of the same kind in the Datastore console running on localhost:
Internal Server Error
The server has either erred or is incapable of performing the requested operation.
Traceback (most recent call last):
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/admin/admin_request_handler.py", line 97, in dispatch
super(AdminRequestHandler, self).dispatch()
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/admin/datastore_viewer.py", line 760, in get
count=20)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/admin/datastore_viewer.py", line 121, in _get_entities
query = datastore.Query(kind, _namespace=namespace)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/google/appengine/api/datastore.py", line 1361, in __init__
datastore_errors.BadArgumentError)
File "/home/tom/work/google-cloud-sdk/platform/google_appengine/google/appengine/api/datastore_types.py", line 181, in ValidateString
raise exception('%s must not be empty.' % name)
BadArgumentError: kind must not be empty.
Here's the gcloud version info:
$ gcloud --version
Google Cloud SDK 157.0.0
app-engine-python 1.9.54
beta 2017.03.24
bq 2.0.24
core 2017.05.30
gcloud
gsutil 4.26
Any hints as to what might be wrong?
I think it's a bug in the Datastore console.
I was able to replicate the same error on my machine. If you look at the traceback in the exception, you see (fourth from the bottom):
File "/.../google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/admin/datastore_viewer.py", line 760, in get
count=20)
If you look at that file from line 735, you see:
def get(self, entity_key_string=None):
super(DatastoreEditRequestHandler, self).get(entity_key_string)
if entity_key_string:
entity_key = datastore.Key(entity_key_string)
entity_key_name = entity_key.name()
entity_key_id = entity_key.id()
namespace = entity_key.namespace()
kind = entity_key.kind()
entities = [datastore.Get(entity_key)]
parent_key = entity_key.parent()
if parent_key:
parent_key_string = _format_datastore_key(parent_key)
else:
parent_key_string = None
else:
entity_key = None
entity_key_string = None
entity_key_name = None
entity_key_id = None
namespace = self.request.get('namespace')
kind = self.request.get('kind')
entities, _ = _get_entities(kind,
namespace,
order=None,
start=0,
count=20) ### line 760 #######
...
In that last line, kind is supposed to be string containing the model name of the model you are working with. Instead, if you follow the traceback down into that function call, you see that kind is an empty string. I don't think there's much you can do to fix it at this time. I'd recommend using the interactive console for now to create/edit your Datastore entities, and just use the viewer to view them. I'm sure Google will fix this with an update soon.
Edit: I actually found a hacky fix if you really want the Datastore console to work, but I would still recommend just using the interactive console. The hacky fix is you manually enter the kind variable into the URL sent by the Datastore console.
When you click "Create New Entity" and get the error you show in your question, look at the url:
http://localhost:8000/datastore/edit?kind=&next=http%3A%2F%2Flocalhost%3A8000%2Fdatastore%3Fkind%3DUnique
You see ...:8000/datastore/edit?kind=&.... Right there, just type in the model name. For example, let's say your model name is User:
http://localhost:8000/datastore/edit?kind=User&next=http%3A%2F%2Flocalhost%3A8000%2Fdatastore%3Fkind%3DUnique
This brought me to the "Create new entity" form for class User (one of my models), and I was able to create a user with that form.
P.S. You may want to also change the model name on the very end of that url (where mine says Unique) to match the kind you are creating, because that model name is the kind that will be displayed after you create your new entity. So mine created a User, but then displayed the Unique entities (Unique is one of my model names).

Getting error "ImportError: Could not find 'input_readers' on path 'map reduce'" trying to start mapReduce job

I'm getting this error... "ImportError: Could not find 'input_readers' on path 'map reduce'" when trying to Run my map reduce job via the http://localhost:8080/mapreduce launcher page.
It looks like my problem is similar to this post, AppEngine mapper API import error. Unfortunately, no definitive answers were given.
I've simplified it down to this tiny testmapreduce.py:
from google.appengine.ext import db
class TestEntity(db.Model):
value = db.StringProperty()
def mapperhandler(test):
print test.value
return
And my mapreduce.yaml:
mapreduce:
- name: Simplest MapReduce
mapper:
handler: testmapreduce.mapperhandler
input_reader: mapreduce.input_readers.DatastoreInputReader
params:
- name: entity_kind
default: testmapreduce.TestEntity
One possible clue is the presence of __init__.py has no effect (whether in the project root, the mapreduce directory, or both). I'm sure I'm making a beginner mistake, but over the last couple of days I have read every bit of documentation, and all the examples I can find. Thanks.
UPDATE:
I get the same error trying to invoke it via...
control.start_map(
"Give awards",
"testmapreduce.mapperhandler",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "testmapreduce.TestEntity"},
shard_count=10)
UPDATE:
As requested, the stack trace -- let me know what else would be helpful...
ERROR 2011-10-16 17:09:27,216 _webapp25.py:464] Could not find 'input_readers' on path 'mapreduce'
Traceback (most recent call last):
File "/Develop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/lc/PycharmProjects/mrtest/testmapreduce.py", line 22, in get
shard_count=10) # handle web form test interface
File "/Users/lc/PycharmProjects/mrtest/mapreduce/control.py", line 94, in start_map
transactional=transactional)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/handlers.py", line 811, in _start_map
mapper_input_reader_class = mapper_spec.input_reader_class()
File "/Users/lc/PycharmProjects/mrtest/mapreduce/model.py", line 393, in input_reader_class
return util.for_name(self.input_reader_spec)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 94, in for_name
module = for_name(module_name, recursive=True)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 102, in for_name
short_name, module_name))
ImportError: Could not find 'input_readers' on path 'mapreduce'
INFO 2011-10-16 22:09:27,253 dev_appserver.py:4247] "GET /giveawards HTTP/1.1" 500 -
This problem turned out to be that I was using the 2.7 version of the Python Interpreter in my local environment. When I switched to 2.5, it works fine.

Google app engine send mail service raises exception

I was trying to use the app engine mail sending service on my site.
Currently, I've put something as simple as:
message = mail.EmailMessage(sender="Subhranath Chunder <subhranath#gmail.com>", \
subject="Your account has been approved")
message.to = "Subhranath Chunder <subhranathc#yahoo.com>"
message.body = "Just another message"
message.send()
But every time the code handler raises this error:
2011-08-20 04:44:36.988
ApplicationError: 1 Internal error
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~subhranath/0-2.352669327317975990/index.py", line 71, in post
message.send()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 895, in send
raise e
ApplicationError: ApplicationError: 1 Internal error
My code as a different version, and not the default one. I'm not able to change this version as the default version, unless this error gets resolved. Any idea why this is happening?
The sender must either have privilegies or be the logged in user. So subhranath#gmail.com should have admin rights to your location or be the logged in user to run your code.
Most basic example would be just a HTTP get to send an email:
def get(self):
message = mail.EmailMessage(sender='subhranath_app_admin#gmail.com', subject='Email')
message.to='any...#gmail.com'
message.body = os.environ.get('HTTP_HOST')
message.send()
If you want to send an email as the logged in user you can make the sender a variable:
senderemail = users.get_current_user().email() if users.get_current_user() else 'subhranath#gmail.com'
I think the error message you posted was not very clear and if it persists I recommend to get more info about the error message using logging and exceptions.
Seems to be a app engine system glitch. The same code is now working, without any new deployment.

Resources