I am getting a ClientPayloadError when reading data from a server. The data is JSON. I am using python 3.6.5 and aiohttp 3.4.4
On debugging it seems that there is a close connection detected, and upon getting that aiohttp tries to see if it was done reading, and since it wasn't it throws this exception.
I was reading the data using
await response.text()
But I tried to change it to reading the response from content in chunks and see if that makes the problem go away but I had the same issue.
So what I am wondering is if there is something going on where the wrong connection is being closed. Maybe a connection I was done reading on is getting closed and a connection that's still not finished with its data is getting that signal ? Is that possible ? It seems like perhaps something mentioned here:
Python asyncio/aiohttp: What are the requirements regarding BaseProtocol.connection_lost()?
Seems that the answer was ultimately in calling JSON decode in the async portion of the code. That was a blocking operation. So something was timing out a transport layer while the decoding took place. I took that operation out of the async portion and it seems to work.
But ultimately, I don't know what was timing out. Would be good to know.
This seems to work:
return_data = await response.text()
And then I decode the return data in a synchronous part of the code. The weird thing is that response.json() didn't work (or caused the same problem)...but I thought that would have been ok since it came from the same package.
Related
I have a question, I would like to know whether ConectionRequest returns response code is 502 or 0 when internet connection lost during the query on Mobile device. Testing Tool returns 502 when there is no internet and debugging the code in Simulator returns response code 0. I would like to know what would be the response code when there is no connectivity during network call. Please advise.
Also, is there a better method to detect offline mode during the network call. I'm aware of Connectivity lib already.
Thanks
This will behave in an inconsistent way because the underlying platforms behave very differently. You need to test for positive responses e.g. server returned 200 instead of testing for negative outcome such as error codes.
You should get an error response callback but we can't guarantee it.
I am executing the ipmimonitoring-sensors.c example provided in the freeipmi library.
It throws internal error sometimes. Issue is reproducible when i execute the program back to back couple of times. I need to wait approximately 30 sec after the last execution for the program to run properly. Has anyone faced this issue before? If yes, can you tell me how to avoid it.
This is the error ipmi_monitoring_sensor_readings_by_record_id: internal error
Thanks
FreeIPMI maintainer here. The "internal error" indicates some logical error that the library doesn't know how to handle. Given its coming from ipmi_monitoring_sensor_readings_by_record_id and it occurs when you run the program back to back, I would bet there is some internal IPMI issue on your system.
Perhaps the motherboard has some issue with a high amount of IPMI traffic or a sensor has issues with a high number of requests. Many of these situations are handled more gracefully (perhaps give a BUSY error or minimally SYSTEM error), but perhaps there is some combo of error situations I haven't yet seen. (Lots of motherboards return errors that would be considered non-standard or unexpected).
If you're interested in working through that, just send something onto the FreeIPMI mailing list.
Set the driver_type = -1 (for default) and it works.
I'm currently working on a C application sending message to a Meteor server over websocket.
I'm using jansson for JSON conversion and nopoll as websocket library.
Everything is working fine in both way (sending / receiving) except when I try to send very large messages (about 15 000 000 characters). I think (I'm not sure) that the message is sent to the server so the nopoll library should not be the source of the issue. But, I'm sure that the message is not treated by Meteor as he should be, because the method (RPC) is never called.
I found that the websocket limitation is equal to the maximum value of a 64-bit unsigned value so this shouldn't be the problem.
On the other hand, I didn't found the maximum length for a DDP message even in the DDP specification.
Have you any idea of the DDP limitation or other parameters that I didn't think about ?
As I was working on an architecture where the client and the server was on the same machine, I was not limited by the network. I think I was pushing too much informations too fast, and the socket was simply full of data.
The solution is simple : cut the data in several fragments as LPs suggests and implement flow-control.
I also found the Mongo C driver that could be another solution because I should be able to push data directly on the database as suggests Mikkel.
Thank you both for your help.
I'm trying to write a HTTP to ZeroMQ proxy with libevent (2.0.4) which should be able to handle very large (up to 4GB) file upload.
The problem is I don't know how large post requests (larger than memory) are handled by libevent so if you have hints on how to implement large file uploading, please led me on the right path.
have you read the libevent source code? it's very readable.
If you're using it's HTTP code, i think it uses the 'bufferedevent' (or is it evented buffers?) feature. You can simply set callbacks when the input buffer reaches the highwater mark.
Maybe you will find some info in http://mongrel2.org/home since this is HTTP server and proxy that uses ZeroMQ for processing (backend handlers).
I'm attempting to write my own http 1.1 server, just for fun and learning about more about HTTP, sockets, and threading.
I've gotten a good start i think with only delivering static pages (using c, which I would prefer to stay in for the time being). I have a test page I wrote a while ago and deliver it's ~50 files in 124ms according to chrome, without using threads or keep-alive sockets.
I've found it very difficult to get threading/keep-alive working at all. There are little to no resources on the web (that I can find in my hours of Googling) that explain keep-alive connections in detail. If anyone could recommend a good book on HTTP server programming, I would greatly appreciate it.
I've done some threading and socket programming before by making a simple chat program, so I have at least some experience with it.
The issue I'm having is that when I attempt to incorporate threads, the client browser sets up multiple connections. Somewhere along the line, the server gets confused and the client just sits there waiting for responses and the server stops doing anything. I send the Connection: Keep-Alive header, but that doesn't change anything and when I incorporate keep-alive it and create a loop for getting requests in the threaded function, it stalls until the connection is closed.
I would appreciate it if someone could give me some pseudo code on how to get keep alive/threading working for this so the client stops creating multiple connections at a time.
A brief description of whats going on:
main function
load in static pages to large array of fileinfo struct that hold the file data and length
create the socket
set it to listen to port 80
set it to listen for 10 connections at a time(i know this is low...)
start an endless loop
block while waiting for someone to connect
check if it's a localhost connection
shutdown the server
otherwise
start a thread(with pthread), sending it the socket variable
loop
Thread Function
setsock opt for 3 sec timeout on send/recv and enable Keep-alive
start endless loop
read in request
if request timed out, break the loop
Validate Request function call
Create Reponse function call
Send response
if request contained Connection: close header break the loop
loop
close socket
return
I would recommend looking at GNU libmicrohttpd. It focuses squarely on providing a framework upon which to build HTTP 1.1 servers. It is small and supports keep-alive with and without threading. (Personally I use it without threading. It has several threading models too.)
Even if you decide to write your web server from scratch, I would suggest looking at libmicrohttpd to gain insight in not only how the protocol works, but how the library models "the work flow" of a web server in a very clean way. I think it is a mistake to imagine that keep-alive implies threading and I think it is an impediment to understanding keep-alive.
(Regarding Apaches' credits as a web server, it is pretty huge, and there is a lot in there not related to protocols, but rather things like its plugin system and so on.)
I'd recommend grabbing the source for Apache and seeing how they handle it. There's not much point in psuedo code when you can see how the real thing works.
Perhaps you could look at Apache's code for some clues. It is written in C.
Hopefully someone will come along and give a more detailed answer :)