Curl doesn't send entire form-data in HTTP POST request - c

Question

Edit:
Problem: 2 and Problem: 3 solved by following #melpomene comment i.e., by using number of bytes read to print the buffer.
But still struck on Problem: 1.
I have written a TCP server-client program. Later out of curiosity, I want to know about HTTP server.
My previous question: Simple TCP server can't output to web browser
Now I'm just seeing what and how the data is transferred to the server by using GET and POST( form-data and x-www-form-urlencoded for now).
I'm following How to cURL POST from the Command Line to send POST requests.
When I send x-www-form-urlencoded as:
curl -d "data=example1&data2=example2" localhost:8080
Output on Server:
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
Content-Length: 28
Content-Type: application/x-www-form-urlencoded
data=example1&data2=example2
This is as expected.
Problem: 1
Now comes the problem. When I try to send form-data, the output is not expected.
When I send form-data as:
curl -X POST -F "name=user" -F "password=test" localhost:8080
Output on server:
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
Content-Length: 244
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------78b7f8917ad1992c
I'm getting the boundary but I'm not getting the next part like the data I'm sending.
Problem: 2
One more odd thing is when I try to send x-www-form-urlencoded after sending form-data.
When I send x-www-form-urlencoded after form-data as:
curl -d "data=example1&data2=example2" localhost:8080
Output on server:
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
Content-Length: 28
Content-Type: application/x-www-form-urlencoded
data=example1&data2=example2------------78b7f8917ad1992c
Why am I getting boundary here?
Problem: 3
And also while sending GET as:
curl localhost:8080
Output on server:
GET / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
ontent-Length: 28
Content-Type: application/x-www-form-urlencoded
data=example1&data2=example2------------78b7f8917ad1992c
I'm getting Content-Type and x-www-form-urlencoded data along with boundary.
What am I doing wrong?
Is something wrong with my code or with my understanding?
Server.c:
// Server side C program to demonstrate Socket programming
#include <stdio.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>
#define PORT 8080
int main(int argc, char const *argv[])
{
int server_fd, new_socket; long valread;
struct sockaddr_in address;
int addrlen = sizeof(address);
char buffer[1024] = {0};
char *hello = "HTTP/1.1 200 OK\nContent-Type: text/plain\nContent-Length: 12\n\nHello world!";
// Creating socket file descriptor
if ((server_fd = socket(AF_INET, SOCK_STREAM, 0)) == 0)
{
perror("In socket");
exit(EXIT_FAILURE);
}
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons( PORT );
if (bind(server_fd, (struct sockaddr *)&address, sizeof(address))<0)
{
perror("In bind");
exit(EXIT_FAILURE);
}
if (listen(server_fd, 10) < 0)
{
perror("In listen");
exit(EXIT_FAILURE);
}
while(1)
{
printf("\n+++++++ Waiting for new connection ++++++++\n\n");
if ((new_socket = accept(server_fd, (struct sockaddr *)&address, (socklen_t*)&addrlen))<0)
{
perror("In accept");
exit(EXIT_FAILURE);
}
valread = read( new_socket , buffer, 1024);
printf("%s\n",buffer );
write(new_socket , hello , strlen(hello));
printf("------------------Hello message sent-------------------\n");
close(new_socket);
}
return 0;
}

Answer 1

Problem 1
I have checked the request headers and found Expect: 100-continue header. This is the first time I have seen this header.
A simple search in Google shows this is causing the problem.
Expect: 100-Continue' Issues and Risks (I'm just gonna paste everything to avoid dead link)
How the Expect: 100-Continue Header Works
When Expect: 100-Continue is NOT present, HTTP follows approximately the following flow (from the
client's point of view):
1. The request initiates a TCP connection to the server.
2. When the connection to the server is established, the full request--which includes both the request headers and the request body--is transmitted to the server.
3. The client waits for a response from the server
(comprised of response headers and a response body).
4. If HTTP
keep-alives are supported, the request is optionally repeated from
step 2.
When the client is using the Expect: 100-Continue feature, the
following events occur:
1. The request initiates a TCP connection to the server.
2. When the
connection to the server is established, the request--including the
headers, the Expect: 100-Continue header, without the request body--is
then transmitted to the server.
3. The client then waits for a response
from the server. If the status code is a final status code, using the
prior steps above the client retries the request without Expect:
100-Continue header. If the status code is 100-Continue, the request
body is sent to the server.
4. The client will then wait for a response
from the server (comprised of response headers and a response body).
5. If HTTP keep-alives are supported, the request is optionally repeated
from step 2.
Why use Expect: 100-Continue?
API POST requests that
include the Expect: 100-Continue header save bandwidth between the
client and the server, because the server can reject the API request
before the request body is even transmitted. For API POST requests
with very large request bodies (such as file uploads), the server can,
for example, check for invalid authentication and reject the request
before the push body was sent, resulting in significant bandwidth
savings.
Without Expect: 100-Continue:
Without the Expect: 100-Continue
feature, the entire API request, including the (potentially large)
push body would have to be transmitted before the server could even
determine if the syntax or authentication is valid. However, since the
majority of our API requests have small POST bodies, the benefits of
separating the request header from the request body is negligible.
Problems when the request header and body are sent separately
Because
of the high volume of requests that Urban Airship handles, many levels
of complexity exist between our clients and the servers responsible
for responding to API requests. This is not an abnormal phenomenon for
most server configurations and strategies, but it does introduce a
risk of elevated request failures to any API POST requests using the
Expect: 100-Continue header. This is due to the fact that the request
header and the request body are sent separately from one another, and
must travel through the same connection throughout the entire API
server infrastructure.
With the number of proxies, load-balancing servers, and back-end
request processing servers that are implemented, requests with the
Expect: 100-Continue header have an increased probability of becoming
separated from one another, and hence returning with an error.
What To Expect:
We've always attempted to support Expect: 100-Continue.
However, we have determined that our customers that use Expect:
100-Continue are receiving a sub-optimal quality of service due to
elevated request failures.
Additionally, the majority of our API requests have small POST bodies,
and as a result the benefits of separating the request header from the
request body are negligible. These reasons have motivated us to
disable support for Expect: 100-Continue service-wide.
Our Recommendations:
We recommend against the use of Expect:
100-Continue. If you receive an HTTP Error 417 (Expectation failed),
retry the request without Expect: 100-Continue.
So, to prevent Expect: 100-continue header in POST form-data, include -H 'Expect:' in your `curl
curl -X POST -F "name=user" -F "password=test" localhost:8080 -H 'Expect:'
Now you can receive your entire data in one go(just like Postman) as you said in comments.
Problem 2 & 3
As #melpomene said in comments, read() doesn't put \0 after reading. That's why you are seeing data from previous requests.
So, just use valread to iterate over string to print or just declare variable in your while loop as I said in the comments.
Code:
while(1)
{
printf("\n+++++++ Waiting for new connection ++++++++\n\n");
if ((new_socket = accept(server_fd, (struct sockaddr *)&address, (socklen_t*)&addrlen))<0)
{
perror("In accept");
exit(EXIT_FAILURE);
}
char buffer[30000] = {0}; // This way you get new variable everytime. So, there is no need to iterate over the string using valread value.
valread = read( new_socket , buffer, 30000);
printf("%s\n",buffer );
write(new_socket , hello , strlen(hello));
printf("------------------Hello message sent-------------------%lu\n", valread);
close(new_socket);
}