Why am I getting a timeout exception with flask on googles app engine when trying to append a value to a wtforms validtor? - google-app-engine

I'm having trouble deploying a flask app to the Google app engine. It works fine when I run it locally but as soon as it's deployed to the app engine it fails. With two exceptions (I'm always getting both, I'm unsure which one comes first).
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/logging/handlers.py", line 1008, in emit
smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 338, in connect
(code, msg) = self.getreply()
File "/opt/python3.7/lib/python3.7/smtplib.py", line 391, in getreply
+ str(e))
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out
and
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/smtplib.py", line 387, in getreply
line = self.file.readline(_MAXLINE + 1)
File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
It consistently happens within the __init__ function of a form. And specifically in there when I try to append a validator to a validators list.
self.password.validators.append(InputRequired(message='Please enter password!'))
It is not specific to this one validator, the same thing happens with others too. I am at a complete loss as to why and how appending validators would interact with SMTP or sockets.
One thing I noticed is that I had an issue with the Email validator which also worked fine locally but required me to install the module 'email_validator' to work. When I did that it complained that it couldn't use append on a tuple when I tried to append a validator to - what was a list in my local app - but for some reason turned out to be a tuple when I ran the exact same code on the app engine. I removed the Email validator (and the import/requirements) entirely and now I'm at the problem I've described above. It may or may not be related to the Email validator issue.
Update: The error message does indeed seem to be related to the validators.append() call seeing as by setting up the validators at object creation and avoiding all further changes via validators.append() fixed the problem for me. This is the requirements.txt that goes with the application:
blinker>=1.4
Click>=7.0
Flask>=1.1.1
Flask-Login>=0.5.0
Flask-Mail>=0.9.1
Flask-SQLAlchemy>=2.4.1
Flask-WTF>=0.14.3
itsdangerous>=1.1.0
Jinja2>=2.11.1
MarkupSafe>=1.1.1
PyJWT>=1.7.1
PyMySQL>=0.9.3
python-dotenv>=0.11.0
SQLAlchemy>=1.3.13
Werkzeug>=1.0.0
WTForms>=2.2.1
Instead of a full minimal, reproducible example I can offer a minimized version of the form that was causing the issues when instanciated on the GAE:
class ExampleForm(FlaskForm):
name = StringField('Name')
original_name = HiddenField('Original Name')
def add_validators(self):
names = [a.name for a in Users.query.all()]
if self.original_name.data:
names.remove(self.original_name.data)
self.name.validators.append(InputRequired(message='Please enter name!'))
self.name.validators.append(NoneOf(names, message='Username already exists!'))
def __init__(self):
super().__init__(*args, **kwargs)
self.add_validators()
As I'm writing this it occurs to me that maybe I need to always initialize an empty validators list on creation to ensure new entries can be appended to it (like with a normal list) like this name = StringField('Name', validators=[]). I'd still be confused as to why it works locally, but if it fixes the problem that'd be what really counts for me.

Related

Google AppEngine application log assigned to the wrong request log

When I look at the logs in the Google Log Viewer for my GAE project, I see that often the logs that I write myself in the code are assigned to the wrong request. Most of the time the log is assigned to the request directly after the request that produced the log entry.
As the root of every application log in GAE must be a request, this means that the wrong request is sometimes marked as error, because another request before produced an error, but the log is somehow assigned to the request after that.
I don't really do anything special, I use Ktor as my servlet and have an interceptor that creates a log when an exception occurs before returning status 500.
I use Java logging via SLF4J with the google cloud logging handler, but before that I used logback via SLf4J and had the same problem.
The content of the logs itself is also correct, the returned status of the request, the level of the log entry, the message, everything is ok.
I thought that it may be because I use kotlin and switch coroutine contexts during a single request, but in some cases the point where I write the log and where I send the response are exactly next to each other, so I'm not sure if kotlin has anything to do with it.
My logging.properties:
# To use this configuration, add to system properties : -Djava.util.logging.config.file="/path/to/file"
#
.level = INFO
# it is recommended that io.grpc and sun.net logging level is kept at INFO level,
# as both these packages are used by Stackdriver internals and can result in verbose / initialization problems.
io.grpc.netty.level=INFO
sun.net.level=INFO
handlers=com.google.cloud.logging.LoggingHandler
# default : java.log
com.google.cloud.logging.LoggingHandler.log=custom_log
# default : INFO
com.google.cloud.logging.LoggingHandler.level=INFO
# default : ERROR
com.google.cloud.logging.LoggingHandler.flushLevel=WARNING
# default : auto-detected, fallback "global"
#com.google.cloud.logging.LoggingHandler.resourceType=container
# custom formatter
com.google.cloud.logging.LoggingHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.SimpleFormatter.format=%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS %4$-6s %2$s %5$s%6$s%n
#optional enhancers (to add additional fields, labels)
#com.google.cloud.logging.LoggingHandler.enhancers=com.example.logging.jul.enhancers.ExampleEnhancer
My logging relevant dependencies:
implementation "org.slf4j:slf4j-jdk14:1.7.30"
implementation "com.google.cloud:google-cloud-logging:1.100.0"
An example logging call:
exception<Throwable> { e ->
logger().error("Error", e)
call.respondText(e.message ?: "", ContentType.Text.Plain, HttpStatusCode.InternalServerError)
}
with logger() being:
import org.slf4j.Logger
import org.slf4j.LoggerFactory
inline fun <reified T : Any> T.logger(): Logger = LoggerFactory.getLogger(T::class.java)
Edit:
An example of the log in Google cloud. The first request has the query parameter GAID=cdda802e-fb9c-47ad-0794d394c913, but as you can see the error log for that request is in the one below, marked in red.

Volttron throws an Attribute error when attempting to publish

I am combining Scapy with Volttron and I want to publish to a topic when a packet comes in and has certain features. However, I keep running into this error:
Traceback (most recent call last): File "sniff.py", line 373, in
sys.exit(main()) File "sniff.py", line 342, in main
utils.vip_main(sniffer, version=version) File "/home/jenny/workspace/volttron/volttron/platform/agent/utils.py",
line 314, in vip_main
version=version, **kwargs) File "sniff.py", line 336, in sniffer
Sniffer(**kwargs) File "sniff.py", line 138, in init
self.vip.pubsub.publish('pubsub', "some/topic", message="blah")
File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/subsystems/pubsub.py",
line 602, in publish
self._save_parameters(result.ident, **kwargs)
File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/subsystems/pubsub.py",
line 706, in _save_parameters
event = self.core().schedule(end_time, self._cancel_event, result_id)
File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/core.py",
line 409, in schedule
self._schedule_callback(deadline, event) File "/home/jenny/workspace/volttron/volttron/platform/vip/agent/core.py",
line 417, in _schedule_callback
self._schedule_event.set()
AttributeError: 'NoneType' object has no attribute 'set'
The closest to a solution I found is here RPC crashes with AttributeError: 'NoneType' object has no attribute 'call'. However, I notice they aren't exactly the same issue and so there isn't much surprise when I tried the solution offered there and it didn't work for me.
My code looks like this:
def sniffer(config_path, **kwargs):
''' Initializations '''
global pkt_counter
global IP_counter
# Defined other parameters here
class Sniffer(Agent):
def __init__(self, **kwargs):
super(Sniffer, self).__init__(**kwargs)
# I am just testing the publish function here
self.vip.pubsub.publish('pubsub', "some/topic", message="blah")
sniff(count=0, iface=conf.iface, prn = self.pkt_action, store=0)
def pkt_action(self, pkt):
#Process every packet, updates values and rises and alert if necessary
# some checks are run here and later a publish is called
Can someone please let me know what I'm doing wrong?
Edit: I didn't add that I want to run this script as I would a simple python script on the terminal (eg. python somescript.py): no installation. The reason I'm trying to do this is that I get an error when I install the agent and start it. The platform doesn't allow Scapy to create and connect to sockets.
You are attempting to use the vip module from in a context that isn't expected. The agent's lifecycle doesn't actually start executing until the start agent or configure events are triggered. Try to put the publish within those contexts.
The following excerpt is from the VOLTTRON readthedocs site located: https://volttron.readthedocs.io/en/develop/devguides/agent_development/Agent-Development-Cheatsheet.html#agent-lifecycle-events
Core Agent Functionality
These tools volttron.platform.vip.agent module. Try importing
Agent Lifecycle Events
Each agent has four events that are triggered at different stages of its life. These > are onsetup, onstart, onstop, and onfinish. Registering callbacks to these events are commonplace in agent development, with onstart being the most frequently used.
The easiest way to register a callback is with a function decorator:
#Core.receiver('onstart')
def function(self, sender, **kwargs):
function_body

Images from Google Cloud Storage no longer display after GAE 1.8.8 update

After updating to GAE 1.8.8 and the GCS Client to r127 my code that displays images no longer works using the Development Server or Production. Code that worked previously now throughs the following error in development:
Blob with key 'encoded_gs_file:blahblahblah' does not exist
Note blahblahblah is just my shorthand for a long string.
Below is how I generate my url. This has always worked in the past.
bkey = blobstore.create_gs_key('/gs/mybucket/myfile.jpg')
url = images.get_serving_url(bkey, size=170, crop=True, secure_url=True)
Below is a stack trace from the development server:
ERROR 2013-11-21 20:35:31,767 images_stub.py:405] Blob with key 'encoded_gs_file:c2l0ZS1hdWRpdC1waW5uYWNsZWNyZS9CYXR0ZXJ5L3Bob3Rvcy8yMDEzMTEyMTAwMTEyOUBhNzllYmQwNC05YWY3LTRhMWItYTg3My1mZGIwYjI5MjNjNzguanBn' does not exist
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/images/images_stub.py", line 401, in _OpenBlob
blobstore_stub.BlobstoreServiceStub.ToDatastoreBlobKey(blob_key))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 651, in Get
return GetAsync(keys, **kwargs).get_result()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1541, in __get_hook
entities = extra_hook(entities)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 620, in local_extra_hook
raise datastore_errors.EntityNotFoundError()
EntityNotFoundError
ApplicationError(1,)
Reading the release notes for GAE 1.8.8 I cannot see anything else I need to change in my code. Any pointers as to what I am doing wrong would be great.

Getting error "ImportError: Could not find 'input_readers' on path 'map reduce'" trying to start mapReduce job

I'm getting this error... "ImportError: Could not find 'input_readers' on path 'map reduce'" when trying to Run my map reduce job via the http://localhost:8080/mapreduce launcher page.
It looks like my problem is similar to this post, AppEngine mapper API import error. Unfortunately, no definitive answers were given.
I've simplified it down to this tiny testmapreduce.py:
from google.appengine.ext import db
class TestEntity(db.Model):
value = db.StringProperty()
def mapperhandler(test):
print test.value
return
And my mapreduce.yaml:
mapreduce:
- name: Simplest MapReduce
mapper:
handler: testmapreduce.mapperhandler
input_reader: mapreduce.input_readers.DatastoreInputReader
params:
- name: entity_kind
default: testmapreduce.TestEntity
One possible clue is the presence of __init__.py has no effect (whether in the project root, the mapreduce directory, or both). I'm sure I'm making a beginner mistake, but over the last couple of days I have read every bit of documentation, and all the examples I can find. Thanks.
UPDATE:
I get the same error trying to invoke it via...
control.start_map(
"Give awards",
"testmapreduce.mapperhandler",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "testmapreduce.TestEntity"},
shard_count=10)
UPDATE:
As requested, the stack trace -- let me know what else would be helpful...
ERROR 2011-10-16 17:09:27,216 _webapp25.py:464] Could not find 'input_readers' on path 'mapreduce'
Traceback (most recent call last):
File "/Develop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/lc/PycharmProjects/mrtest/testmapreduce.py", line 22, in get
shard_count=10) # handle web form test interface
File "/Users/lc/PycharmProjects/mrtest/mapreduce/control.py", line 94, in start_map
transactional=transactional)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/handlers.py", line 811, in _start_map
mapper_input_reader_class = mapper_spec.input_reader_class()
File "/Users/lc/PycharmProjects/mrtest/mapreduce/model.py", line 393, in input_reader_class
return util.for_name(self.input_reader_spec)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 94, in for_name
module = for_name(module_name, recursive=True)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 102, in for_name
short_name, module_name))
ImportError: Could not find 'input_readers' on path 'mapreduce'
INFO 2011-10-16 22:09:27,253 dev_appserver.py:4247] "GET /giveawards HTTP/1.1" 500 -
This problem turned out to be that I was using the 2.7 version of the Python Interpreter in my local environment. When I switched to 2.5, it works fine.

Google app engine send mail service raises exception

I was trying to use the app engine mail sending service on my site.
Currently, I've put something as simple as:
message = mail.EmailMessage(sender="Subhranath Chunder <subhranath#gmail.com>", \
subject="Your account has been approved")
message.to = "Subhranath Chunder <subhranathc#yahoo.com>"
message.body = "Just another message"
message.send()
But every time the code handler raises this error:
2011-08-20 04:44:36.988
ApplicationError: 1 Internal error
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~subhranath/0-2.352669327317975990/index.py", line 71, in post
message.send()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 895, in send
raise e
ApplicationError: ApplicationError: 1 Internal error
My code as a different version, and not the default one. I'm not able to change this version as the default version, unless this error gets resolved. Any idea why this is happening?
The sender must either have privilegies or be the logged in user. So subhranath#gmail.com should have admin rights to your location or be the logged in user to run your code.
Most basic example would be just a HTTP get to send an email:
def get(self):
message = mail.EmailMessage(sender='subhranath_app_admin#gmail.com', subject='Email')
message.to='any...#gmail.com'
message.body = os.environ.get('HTTP_HOST')
message.send()
If you want to send an email as the logged in user you can make the sender a variable:
senderemail = users.get_current_user().email() if users.get_current_user() else 'subhranath#gmail.com'
I think the error message you posted was not very clear and if it persists I recommend to get more info about the error message using logging and exceptions.
Seems to be a app engine system glitch. The same code is now working, without any new deployment.

Resources