akka http with connecting http server to outgoing tcp server throught nginx - akka-stream

Problem when try to using akka http like proxy to tcp.
Start http server, which responsibility only to get json request from the client and send message on Tcp server through nginx, take response from it and send response back to the client. So, only like proxy.
The problem is that server try to response before tcp server response. But when I remove nginx proxy between http and tcp servers everything all right. In tcp level akka http server send RST packet and right after that get RST from nginx and connection close. Tcp respond to nginx but connection already closed. With simple tcp server the same situation.
Akka version 2.4.17, akka http 10.0.10
The code below
val connections: Flow[ByteString, ByteString,
Future[Tcp.OutgoingConnection]] =
Tcp().outgoingConnection(new InetSocketAddress(settings.tcpServerHost,
settings.tcpServerPort))
def start(): Unit ={
val requestHandler: HttpRequest => Future[HttpResponse] = {
case HttpRequest(POST, uri#Uri.Path("/v1/doc"), _, entry, _) =>
proxyTcp(100, uri, entry)
case HttpRequest(POST, uri#Uri.Path("/v5/doc"), _, entry, _) =>
proxyTcp(105, uri, entry)
case _ => Future(HttpResponse(status = StatusCodes.NotFound))
}
Http().bind(settings.host, settings.port).runForeach(connection => {
connection.handleWithAsyncHandler(requestHandler)
}) }
private def proxyTcp(version: Short, uri: Uri, entry: RequestEntity):
Future[HttpResponse] = {
val transform = Transformations(connections, version)
entry.withoutSizeLimit().dataBytes
.via(transform.json2tlv)
.via(transform.connections)
.runWith(Sink.head)
.map(result => HttpResponse(status = StatusCodes.OK, entity =
HttpEntity(ContentTypes.`application/json`, result.get)))
}

Related

Unble to open multiple simultaneous websockets to the same endpoint in case where the consumer is busy for the first websocket

I am using Django channels with React websockets. I am unable to open multiple simulataneous websockets if the consumer of the first websocket channel is busy with some activity. I want to open multiple simultaneous websockets where consumers is doing something for each individual websocket.
The module versions are:
asgiref 3.6.0
daphne 3.0.2
django-cors-headers 3.13.0
djangorestframework 3.14.0
djangorestframework-simplejwt 5.2.2
From the snippet below, once one websocket completes the task the second websocket can be connected (in a separate tab) but while 1st websocket is busy (sleeping), the other websockets in the new browser tabs fails after handshake with the following error:
**
WebSocket HANDSHAKING /ws/socket-server/monitor-rack-kpis/ushoeuhrti/ [127.0.0.1:49228]
django.channels.server INFO WebSocket HANDSHAKING /ws/socket-server/monitor-rack-kpis/ushoeuhrti/ [127.0.0.1:49228]
daphne.http_protocol DEBUG Upgraded connection ['127.0.0.1', 49228] to WebSocket
daphne.ws_protocol DEBUG WebSocket closed for ['127.0.0.1', 49228]
WebSocket DISCONNECT /ws/socket-server/monitor-rack-kpis/ushoeuhrti/ [127.0.0.1:49228]
django.channels.server INFO WebSocket DISCONNECT /ws/socket-server/monitor-rack-kpis/ushoeuhrti/ [127.0.0.1:49228]
daphne.server WARNING Application instance <Task pending name='Task-56' coro=<StaticFilesWrapper.__call__() running at /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/channels/staticfiles.py:44> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py:387, Task.task_wakeup()]>> for connection <WebSocketProtocol client=['127.0.0.1', 49216] path=b'/ws/socket-server/monitor-rack-kpis/vhofoefwmr/'> took too long to shut down and was killed.
**
I tried using url in routing instead of re_path but that didn't help either
settings.py
`
ASGI_APPLICATION = 'backend.asgi.application'
CHANNEL_LAYERS={'default':{'BACKEND':'channels.layers.InMemoryChannelLayer'}}
`
asgi.py
application = ProtocolTypeRouter({
'http':get_asgi_application(),
'websocket':AuthMiddlewareStack(
URLRouter(logstatus.routing.websocket_urlpatterns)
)
})
routing.py
websocket_urlpatterns=[
re_path(r'^ws/socket-server/monitor-rack-kpis/(?P<username>[A-Za-z]+)/', consumers.InfluxWritePromesthusConsumer.as_asgi())
]
consumer.py
class InfluxWritePromesthusConsumer(WebsocketConsumer):
def ws_connect(message):
message.reply_channel.send({"accept": True})
def receive(self, text_data):
print(f"\nReceiving from: {self.channel_name}\n")
t = 0
while t<=100:
self.send(text_data=json.dumps({
'type':"LearnSocket",
'message': "Received Messages"
}))
t += 10
time.sleep(10)
Frontend- React.JS
const randN = generate()
console.log(randN)
const socket = new WebSocket('ws://127.0.0.1:8000/ws/socket-server/monitor-rack-kpis/'+ randN+ '/');
console.log(socket)
socket.onopen=function(e){
console.log(socket.readyState)
if (user && user.access) {
socket.send("Hi");
//Receiving response from WebSocket
socket.onmessage=function(e){
//let data1=JSON.parse(e.data);
console.log(e.data, new Date());
//setOutputResponse(data1['message']);
}
}
}

ESP-32 can't connect to MQTT broker: mqtt_client: Error network response

I am trying to connect my ESP32 which runs using the ESP-IDF framework to MQTT. I have imported this MQTT library successfully and have set up the configuration to look like this:
static void mqtt_app_start(void)
{
const esp_mqtt_client_config_t mqtt_cfg = {
// .host = "m15.cloudmqtt.com",
.uri = "mqtt://rxarkckf:smNb81Ppfe7T#m15.cloudmqtt.com:10793", // uri in the format (username:password#domain:port)
// .host = "m15.cloudmqtt.com", // config with host, port, user, password seperated
// .port = 10793,
// .username = "rxarkckf",
// .password = "smNb81Ppfe7T",
.event_handle = mqtt_event_handler,
// .user_context = (void *)your_context
};
esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
esp_mqtt_client_start(client);
}
I call mqtt_app_start(); in my app_main function. After uploading the code my ESP-32 doesn't connect to the MQTT broker and outputs this:
␛[0;32mI (12633410) MQTT_CLIENT: Sending MQTT CONNECT message, type: 1, id: 0000␛[0m
␛[0;31mE (12633710) MQTT_CLIENT: Error network response␛[0m
␛[0;32mI (12633710) MQTT_CLIENT: Error MQTT Connected␛[0m
␛[0;32mI (12633710) MQTT_CLIENT: Reconnect after 10000 ms␛[0m
␛[0;32mI (12633710) MQTT_SAMPLE: MQTT_EVENT_DISCONNECTED␛[0m
I have double checked that the values for the host, username, password, and port are all correct. When I look at the logs on the web interface hosted at cloudmqtt.com, I can see this output:
2018-11-17 03:50:53: New connection from 73.94.66.49 on port 10793.
2018-11-17 03:50:53: Invalid protocol "MQIs�" in CONNECT from 73.94.66.49.
2018-11-17 03:50:53: Socket error on client <unknown>, disconnecting.
2018-11-17 03:51:20: New connection from 73.94.66.49 on port 10793.
I had similar experience using mosquitto.
Adding this line to mqtt_config.h made my mqtt working.
#define CONFIG_MQTT_PROTOCOL_311
I think the more correct way to set this configuration is in sdkconfig.h, either manually or using "make menuconfig"
The problem is very simple. The library you are using implements the MQTT 3.1 protocol. The server you are trying to connect to implements the MQTT 3.1.1 protocol or higher.
As specified in the document (https://www.oasis-open.org/committees/download.php/55095/mqtt-diffs-v1.0-wd01.doc):
4.1 Protocol Name
The Protocol Name is present in the variable header of a MQTT CONNECT control packet. The Protocol Name is a UTF-8 encoded
string. In MQTT 3.1 the protocol name is "MQISDP". In MQTT 3.1.1 the
protocol name is represented as "MQTT".
For technical info:
https://mqtt.org/mqtt-specification/

not connecting to web socket properly in gatling

I want to connect to a web socket through gatling. But it is not working. The socket listener does not get any messsage. The code is given below.Can anyone suggest the problem? Is there any recording option for websocket in gatling. the recorder only record http requests.
`val scn = scenario("RecordedSimulation")
.exec(http("login")
.post("/user/login")
.formParam("email", "username")
.formParam("password", "*******")
.check(headerRegex("Set-Cookie", "localhost:1337.sid=(.*); Path=/").saveAs("cookie")))
.pause(5)
.exec(http("get sid")
.get("/socket.io/?EIO=3&transport=polling&t=1468496890883-0")
.headers(headers_3)
.check(regex("\"sid\":\"(.*)\",").saveAs("sid"))
)
.pause(4)
.exec(ws("Connect WebSocket").open("/socket.io/?EIO=3&transport=websocket&sid=${sid}")
.headers(Map(
"Accept-Encoding" -> "gzip, deflate, sdch",
"Accept-Language" -> "en-US,en;q=0.8",
"Pragma" -> "no-cache",
"Host" -> "localhost:1337",
"Cache-Control" -> "no-cache",
"Connection" -> "Upgrade",
"Origin" -> "http://localhost:1337",
"Sec-WebSocket-Extensions" -> "permessage-deflate; client_max_window_bits",
"Sec-WebSocket-Key" -> "sBWXugNrGCMSXmO3BEm4yw==",
"Sec-WebSocket-Version" -> "13",
"Upgrade" ->"websocket",
"Cookie" -> "io=${sid}; __cfduid=d1cf62d5cf275e2c709080ad7610da8b61465800778; cf_clearance=42068d23995e3243b3ee748ac616389d5cc27d92-1468865525-1800; _gat=1; _ga=GA1.2.1134427855.1467369017; localhost:1337.sid=${cookie}"
)))
.pause(1)
.exec(http("Run")
.post("/posturl")
.headers(headers_13)
.body(RawFileBody("RecordedSimulation_0013_request.txt")))
.exec(ws("Set Check for instance ID")
.check(wsAwait.within(30).until(1).regex("\"intInstanceID\":\"(.*-.*-.*-.*-.*)\",").saveAs("instanceID")))
.pause(1)
.exec(ws("Say Hello WS")
.sendText("""{"text": "Hello, I'm ${sid}"}"""))
.exec( session =>{
println(session("cookie").as[String])
session
})
setUp(scn.inject(atOnceUsers(1))).protocols(httpProtocol)
}`
You don't say how your expected incoming messages are related to your other actions, but beware that we don't currently buffer unmatched incoming messages.
So if a messages arrives before you actually set a check, it's lost.
If you have some feedback about your WebSocket load test usage, please share and discuss on our mailing list. There's an ongoing effort to redesign WebSocket support for Gatling 3.

C HTTP Proxy, the browser shows "waiting for" indefinetely

I have a hard time implementing a proxy server in C. It works for a few first webpages but then I get blocked while waiting for a new request.
Design:
Firefox -> Proxy -> Webserver --.
Firefox <- Proxy <- Webserver <-'
So each request is a round-trip from the browser to the proxy and the server and then back. Until the response from the request comes back, nothing should happen. I use no pipelining, threads or anything like that but rather only recv() and send() in a linear serial manner (for simplicity and intuition). I also don't close any sockets as I want to have a persistent connection.
I expect to be able and fetch one whole webpage including subrequested resources like css, img, js, etc..
In my implementation I manage to fetch a few first requests for a webpage. Then it hangs at step 1.
Implementation:
puts("Waiting for user to connect..");
int sock_user = accept(sock, (struct sockaddr*)NULL, NULL);
int sock_host = -1;
printf("User connected.\n");
// Accept requests
while(1){
http_request req;
http_response resp;
// 1. Client ==> Proxy Server
http_parse_request(sock_user, &req); // uses recv()
// 2. Client Proxy ==> Server
if (sock_host < 0)
sock_host = proxy_connect_host(req.header->host);
write(sock_host, req.header->raw_data, req.header->raw_size);
// 3. Client Proxy <== Server
http_parse_response(sock_host, &resp); // uses recv()
// 4. Client <== Proxy Server
write(sock_user, resp.header->raw_data, resp.header->raw_size);
write(sock_user, resp.body ->first_block->data, resp.body ->first_block->size);
}
Log:
---- ......................................... ----
---- after succesfully responded to 4 requests ----
Client ==> Proxy Server
Received 389
Client Proxy ==> Server
Sending.. 389
Sent 389
Client Proxy <== Server
Got header 312
Got body 1437
Response total 1749
Client <== Proxy Server
Sending header.. 312
Sent 312
Sending body.. 1437
Sent 1437
Client ==> Proxy Server
---- Hangs/blocks here ----
Firebug:
Wireshark:
I have no intuition as to what the reason for that block is and I have spent a whole week trying to resolve this problem without a breakthrough.
Among things tried to resolve problem:
Sending some extra CRLF for each response body
Checked the return value from each recv() and send(). (in the log above,
the values getting printed are the return values from recv and
send)
I hope someone could give at least some direction as to how to troubleshoot this or my brain will soon explode :)
You have to be very careful not to read too much data. E.g. make sure that:
header is read only up to the double CRLF; store extra data and sent them with the body
sending of body starts not before the server has sent the complete header (does not apply to this GET case but is important for POST or CONNECT)
there are received and sent only Content-Length bytes from the body
This applies to both the client -> proxy request and the server -> proxy response.
Your sample code stays in an endless loop (while (1) ...). How do you abort this? Do you honor the "Proxy-Connection" header?

CXF - catching/handling HTTP exceptions

This is rather a basic cxf usage question. How/where can we catch the actual HTTP exception/error. I kind of followed the Interceptor/MessageObserver concept but could not capture the HTTP error using them.
I see this error in the log4j log file.
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response
'401: Unauthorized' when communicating with http://10.107.172.79/test/_vti_bin/lists.asmx
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1502)
at org.apache.cxf.transpot.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1448)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1356)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:614)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 9 more
Only javax.xml.ws.WebServiceException with "Could not send Message."
message is thrown while calling the service
try{
GetListCollectionResult result = port.getListCollection();
}catch (javax.xml.ws.WebServiceException excep){
}
This is how we call the service.
To provide NTLM credentials:
Authenticator.setDefault( extended class of Authenticator);
Create the service.
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setServiceClass(ListsSoap.class);
factory.setAddress(list_url);
ListsSoap port = (ListsSoap) factory.create();
Update the conduit.
..
Client client = ClientProxy.getClient(port);
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(36000);
httpClientPolicy.setAllowChunking(false);
http.setClient(httpClientPolicy);
Call service and get the result.
GetListCollectionResult result = port.getListCollection();
Nevermind, I found the answer in the CXF mailing list.
excep.getCause()
gives access to the underlying exception, in my case it is the HTTP Transport exception.

Resources