I am writing a custom plugin for kong. The plugin will transform request/response in accordance with my server. I am getting [info] 27#0: *588 client <x> closed keepalive connection.
After some debugging, I found that the error occurs whenever I set the
ngx.arg[1] with my transformed response. I have followed the existing response-transformer plugin provided by kong.
This is the body of the kong body_filter function:
local ctx = ngx.ctx
local chunk, eof = ngx.arg[1], ngx.arg[2]
ctx.rt_body_chunks = ctx.rt_body_chunks or {}
ctx.rt_body_chunk_number = ctx.rt_body_chunk_number or 1
if eof then
local someChunks = concat(ctx.rt_body_chunks)
local aBody = responseTransformer.transform(theConf, someChunks)
aBody = unEscapeString(aBody)
ngx.arg[1] = aBody or someChunks
else
ctx.rt_body_chunks[ctx.rt_body_chunk_number] = chunk
ctx.rt_body_chunk_number = ctx.rt_body_chunk_number + 1
ngx.arg[1] = nil
end
I ran the same plugin with a local dummy server. It worked properly. But when I proxied to my actual server, I got the closed keepalive connection error.
From kong logs, I can see that the response got transformed properly.
Using curl, I get about half the body of the response.
Found the cause. The server was sending Content-Length header. During rewrite of body this remained unchanged. So, connection was being closed before the full content was being delivered.
To solve this, I had to clear the Content-Length header in the header_filter function:
kong.response.clear_header("Content-Length")
Related
I created an API endpoint using Google Cloud Functions and am trying to call it from a JS fetch function.
I am running into errors that I am pretty sure are related to either CORS or the output format, but I'm not really sure what is going on. A few other SO questions are similar, and helped me realize I needed to remove the mode: "no-cors". Most mention enabling CORS on the BE, so I added response.headers.set('Access-Control-Allow-Origin', '*') - which I learned of in this article - to ensure CORS would be enabled... But I still get the "Failed to fetch" error.
The Full Errors (reproducible in the live demo linked below) are:
Uncaught Error: Cannot add node 1 because a node with that id is
already in the Store. (This one is probably unrelated?)
Access to fetch at
'https://us-central1-stargazr-ncc-2893.cloudfunctions.net/nearest_csc?lat=37.75&lon=-122.5'
from origin 'https://o2gxx.csb.app' has been blocked by CORS policy:
Request header field access-control-allow-origin is not allowed by
Access-Control-Allow-Headers in preflight response.
GET
https://us-central1-stargazr-ncc-2893.cloudfunctions.net/nearest_csc?lat=37.75&lon=-122.5 net::ERR_FAILED
Uncaught (in promise) TypeError: Failed to fetch
See Code Snippets below, please note where I used <---- *** Message *** to denote parts of the code that have recently changed, giving me one of those two errors.
Front End Code:
function getCSC() {
let lat = 37.75;
let lng = -122.5;
fetch(
`https://us-central1-stargazr-ncc-2893.cloudfunctions.net/nearest_csc?lat=${lat}&lon=${lng}`,
{
method: "GET",
// mode: "no-cors", <---- **Uncommenting this predictably gets rid of CORS error but returns a Opaque object which seems to have no data**
headers: {
// Accept: "application/json", <---- **Originally BE returned stringified json. Not sure if I should be returning it as something else or if this is still needed**
Origin: "https://lget3.csb.app",
"Access-Control-Allow-Origin": "*"
}
}
)
.then(response => {
console.log(response);
console.log(response.json());
});
}
Back End Code:
import json
import math
import os
import flask
def nearest_csc(request):
"""
args: request object w/ args for lat/lon
returns: String, either with json representation of nearest site information or an error message
"""
lat = request.args.get('lat', type = float)
lon = request.args.get('lon', type = float)
# Get list of all csc site locations
with open(file_path, 'r') as f:
data = json.load(f)
nearby_csc = []
# Removed from snippet for clarity:
# populate nearby_csc (list) with sites (dictionaries) as elems
# Determine which site is the closest, assigned to var 'closest_site'
# Grab site url and return site data if within 100 km
if dist_km < 100:
closest_site['dist_km'] = dist_km
// return json.dumps(closest_site) <--- **Original return statement. Added 4 lines below in an attempt to get CORS set up, but did not seem to work**
response = flask.jsonify(closest_site)
response.headers.set('Access-Control-Allow-Origin', '*')
response.headers.set('Access-Control-Allow-Methods', 'GET, POST')
return response
return "No sites found within 100 km"
Fuller context for code snippets above:
Here is a Code Sandbox Demo of the above.
Here is the full BE code on GitHub, minus the most recent attempt at adding CORS.
The API endpoint.
I'm also wondering if it's possible that CodeSandbox does CORS in a weird way, but have had the same issue running it on localhost:3000, and of course in prod would have this on my own personal domain.
The Error would appear to be CORS-related ( 'https://o2gxx.csb.app' has been blocked by CORS policy: Request header field access-control-allow-origin is not allowed by Access-Control-Allow-Headers in preflight response.) but I thought adding response.headers.set('Access-Control-Allow-Origin', '*') would solve that. Do I need to change something else on the BE? On the FE?
TLDR;
I am getting the Errors "Failed to fetch" and "field access-control-allow-origin is not allowed by Access-Control-Allow-Headers" even after attempts to enable CORS on backend and add headers to FE. See the links above for live demo of code.
Drop the part of your frontend code that adds a Access-Control-Allow-Origin header.
Never add Access-Control-Allow-Origin as a request header in your frontend code.
The only effect that’ll ever have is a negative one: it’ll cause browsers to do CORS preflight OPTIONS requests even in cases when the actual (GET, POST, etc.) request from your frontend code would otherwise not trigger a preflight. And then the preflight will fail with this message:
Request header field Access-Control-Allow-Origin is not allowed by Access-Control-Allow-Headers in preflight response
…that is, it’ll fail with that unless the server the request is being made to has been configured to send an Access-Control-Allow-Headers: Access-Control-Allow-Origin response header.
But you never want Access-Control-Allow-Origin in the Access-Control-Allow-Headers response-header value. If that ends up making things work, you’re actually just fixing the wrong problem. Because the real fix is: never set Access-Control-Allow-Origin as a request header.
Intuitively, it may seem logical to look at it as “I’ve set Access-Control-Allow-Origin both in the request and in the response, so that should be better than just having it in the response” — but it’s actually worse than only setting it in the response (for the reasons described above).
So the bottom line: Access-Control-Allow-Origin is solely a response header, not a request header. You only ever want to set it in server-side response code, not frontend JavaScript code.
The code in the question was also trying to add an Origin header. You also never want to try to set that header in your frontend JavaScript code.
Unlike the case with the Access-Control-Allow-Origin header, Origin is actually a request header — but it’s a special header that’s controlled completely by browsers, and browsers won’t ever allow your frontend JavaScript code to set it. So don’t ever try to.
I'm programming a simple web server in C. The HTTP message the server generates is stored in a buffer and sent (via send()) as follows:
Scenario 1:
"200 OK\nContent-Type: text/html\nContent-Length: " + resource size in bytes + "\r\n\n"
where the resource size in bytes is converted to a char array using snprintf and then concatenated into the string.
Scenario 2:
"HTTP1.1 404 Not Found\r\nContent-Length: 0\r\n\n"
Scenario 3:
"HTTP1.1 405 Method Not Allowed\r\nAllow: GET, HEAD\r\n"
These are the headers, they are sent beforehand. The message body is sent afterwards as follows:
char resource[length];
int numRead;
while ( (numRead= read(filefd, resource, length)) > 0 )
send(client, resource, length, 0);
When I use wireshark, it doesn't recognize it as an HTTP response. When I use firefox, the web page continues loading until I shut down the server, at which point it displays the HTTP response instead of a webpage (index.html):
image
Do I have to encode the message before sending? Or is there something wrong with my message format?
There are many errors here which suggest that you just made up your own version of HTTP instead of reading the standard. Please refer to the standard for all the details if you want to implement HTTP.
In short:
"200 OK\nContent-Type: text/html\nContent-Length: " + resource size in bytes + "\r\n\n"
It should be HTTP/1.1 200 OK... and not just 200 OK.... All line ends must be \r\n and not \n and there must be a single \r\n at the end of the HTTP header.
"HTTP1.1 404 Not Found\r\nContent-Length: 0\r\n\n"
"HTTP1.1 405 Method Not Allowed\r\nAllow: GET, HEAD\r\n"
It should be HTTP/1.1 not HTTP1.1. And then all the other problems.
Again, if you really want to implement your own HTTP stack then study the standard. It is far more complex than you might think. Just because HTTP is a text based protocol does not mean that it is simple nor that any text is a correct HTTP message. Additionally implementations are tolerant in different ways, so just because it works with one client (i.e. browser) does not mean it is correct and that it will work with a different client too.
I have made a simple HTTP server that listens to socket connections. The server code limits total number of connections that it can hold simultaneously.
So, I have these lines:
do {
new_fd = accept(lfd, NULL, NULL);
nfds += 1;
...
if(nfds + 1 > ntotal){ // connection limit exceeded
set_headers( new_fd, /* HTTP status code here */ );
/* close socket after error had been sent */
}
}while(1);
In this situation I'm interested with HTTP status code that server should send before closing socket.
From this link, it appears 503 is the appropriate HTTP status code to send for an overloaded server.
10.5.4 503 Service Unavailable
The server is currently unable to handle the request due to a
temporary overloading or maintenance of the server. The implication is
that this is a temporary condition which will be alleviated after some
delay. If known, the length of the delay MAY be indicated in a
Retry-After header. If no Retry-After is given, the client SHOULD
handle the response as it would for a 500 response.
Note: The existence of the 503 status code does not imply that a
server must use it when becoming overloaded. Some servers may wish
to simply refuse the connection.
(bold emphasis mine)
I am trying to develop a server application using mongoose C library. In the initial stage of the my try, I am stuck at sending a response for an HTTP request. I am trying to send a simple response of status 200 using following line of code:
mg_send_response_line(nc, 200, "Access-Control-Allow-Origin: *");
printf("Response sent...\n");
But the response is not received to the client(Postman or web-browser).
There is no error, and even the printf line of Response sent is printed.
As against to this, following lines are getting executed successfully:
mg_http_send_error(nc,404, "Fatal Error!"); // I get this error at client side.
The basic simplest_web_server also works fine. Why is my single line of code sending response failing. I am not able to understand/ debug this.
Regards,
Neeraj.
The issue is that there is no content length or transfer encoding specified for the HTTP response and the server does not close the connection so the client hangs waiting for the response.
If you look through the source code, you will see that in mg_http_send_error(), the MG_F_SEND_AND_CLOSE flag is set but it is not set within mg_send_response_line() (though, like you, I assumed that this would be handled by the function).
To fix the issue in your context,
mg_send_response_line(nc, 200, "Access-Control-Allow-Origin: *");
printf("Response sent...\n");
nc->flag |= MG_F_SEND_AND_CLOSE;
I have a hard time implementing a proxy server in C. It works for a few first webpages but then I get blocked while waiting for a new request.
Design:
Firefox -> Proxy -> Webserver --.
Firefox <- Proxy <- Webserver <-'
So each request is a round-trip from the browser to the proxy and the server and then back. Until the response from the request comes back, nothing should happen. I use no pipelining, threads or anything like that but rather only recv() and send() in a linear serial manner (for simplicity and intuition). I also don't close any sockets as I want to have a persistent connection.
I expect to be able and fetch one whole webpage including subrequested resources like css, img, js, etc..
In my implementation I manage to fetch a few first requests for a webpage. Then it hangs at step 1.
Implementation:
puts("Waiting for user to connect..");
int sock_user = accept(sock, (struct sockaddr*)NULL, NULL);
int sock_host = -1;
printf("User connected.\n");
// Accept requests
while(1){
http_request req;
http_response resp;
// 1. Client ==> Proxy Server
http_parse_request(sock_user, &req); // uses recv()
// 2. Client Proxy ==> Server
if (sock_host < 0)
sock_host = proxy_connect_host(req.header->host);
write(sock_host, req.header->raw_data, req.header->raw_size);
// 3. Client Proxy <== Server
http_parse_response(sock_host, &resp); // uses recv()
// 4. Client <== Proxy Server
write(sock_user, resp.header->raw_data, resp.header->raw_size);
write(sock_user, resp.body ->first_block->data, resp.body ->first_block->size);
}
Log:
---- ......................................... ----
---- after succesfully responded to 4 requests ----
Client ==> Proxy Server
Received 389
Client Proxy ==> Server
Sending.. 389
Sent 389
Client Proxy <== Server
Got header 312
Got body 1437
Response total 1749
Client <== Proxy Server
Sending header.. 312
Sent 312
Sending body.. 1437
Sent 1437
Client ==> Proxy Server
---- Hangs/blocks here ----
Firebug:
Wireshark:
I have no intuition as to what the reason for that block is and I have spent a whole week trying to resolve this problem without a breakthrough.
Among things tried to resolve problem:
Sending some extra CRLF for each response body
Checked the return value from each recv() and send(). (in the log above,
the values getting printed are the return values from recv and
send)
I hope someone could give at least some direction as to how to troubleshoot this or my brain will soon explode :)
You have to be very careful not to read too much data. E.g. make sure that:
header is read only up to the double CRLF; store extra data and sent them with the body
sending of body starts not before the server has sent the complete header (does not apply to this GET case but is important for POST or CONNECT)
there are received and sent only Content-Length bytes from the body
This applies to both the client -> proxy request and the server -> proxy response.
Your sample code stays in an endless loop (while (1) ...). How do you abort this? Do you honor the "Proxy-Connection" header?