I am hoping to utilise ZeroMQ in order to handle queuing of a large number of requests coming in at the same time to an attestation server.
I am in an HPC environment and so have a large number of compute nodes all running the same launcher program that attests with a server. This server code runs on a front-end node and after successfully attesting the client, will then release a key in order for the client to decrypt job data.
Currently standard sockets are used. When a client initially sends something to the server a new socket is spawned using accept() from sys/socket.h. This allows the client and server to send multiple messages between each other in the attestation process (checksums etc) before finally returning the key if successful.
The issue with ZeroMQ is that the attach command is not necessary and so a secondary socket for that specific attestation is not created. All the messages are dealt with from all the clients in whatever order they come in, leading to the multi-part attestation process not working. I have spent ages going through the guide and googling to try and find a similar solution but have not had any luck so far.
Is there a way I can utilise ZeroMQ to give the same behaviour in this application as a standard socket?
Q : Is there a way I can utilise ZeroMQ to give the same behaviour in this application as a standard socket?
ZeroMQ is a very smart and rather behaviour-oriented signaling / messaging-platform for distributed-systems, not a socket.
Given your intentions are to be worth for HPC-ecosystem, a solution postulated / directed to use some tool and a must to bend it as much as possible, so as it will become close to resemble a behaviour that is native, but for other tool, does not seem to be a typical HPC-grade approach.
HPC-code is typically very well-crafted so as to become computing-costs-efficient ( and bless the Boss, CFO & gov/mil-funding for all those, who are today permitted not to design the HPC-code for the ultimate performance and hardware-resources' use efficiency :o) ) - here, if one pays expenses on ZeroMQ instantiations, there seems no benefit to come from these non-zero costs of instantiations and getting "just"-a-socket-alike behaviour, at cost, has negative performance yield, without any adjustments in some future benefits from smart, cluster-wide ZeroMQ services ( be it an N+1 or N+M redundancy, low-latency smart inter-node cluster signaling, cryptography, cheap security-motivated white-listing, or anything that may represent any additional HPC-grade Project's benefit, that may justify the costs of the initial ZeroMQ instantiation ).
Defined archetype of ZMQ_STREAM may provide some tools, yet, ref. above
A socket of type ZMQ_STREAM is used to send and receive TCP data from a non-ØMQ peer, when using the tcp:// transport. A ZMQ_STREAM socket can act as client and/or server, sending and/or receiving TCP data asynchronously.
When receiving TCP data, a ZMQ_STREAM socket shall prepend a message part containing the identity of the originating peer to the message before passing it to the application. Messages received are fair-queued from among all connected peers.
When sending TCP data, a ZMQ_STREAM socket shall remove the first part of the message and use it to determine the identity of the peer the message shall be routed to, and unroutable messages shall cause an EHOSTUNREACH or EAGAIN error.
To open a connection to a server, use the zmq_connect call, and then fetch the socket identity using the ZMQ_IDENTITY zmq_getsockopt call.
To close a specific connection, send the identity frame followed by a zero-length message (see EXAMPLE section).
When a connection is made, a zero-length message will be received by the application. Similarly, when the peer disconnects (or the connection is lost), a zero-length message will be received by the application.
You must send one identity frame followed by one data frame. The ZMQ_SNDMORE flag is required for identity frames but is ignored on data frames.
ZMQ_STREAM Example:
void *ctx = zmq_ctx_new (); assert (ctx && "Context Instantiation Failed..." );
void *socket = zmq_socket (ctx, ZMQ_STREAM); assert (socket && "socket Instantiation Failed..." );
int rc = zmq_bind (socket, "tcp://*:8080"); assert (rc == 0 && "socket.bind() Failed..." );
uint8_t id [256]; /* Data structure to hold the ZMQ_STREAM ID */
size_t id_size = 256;
uint8_t raw [256]; /* Data structure to hold the ZMQ_STREAM received data */
size_t raw_size = 256;
while (1) {
id_size = zmq_recv (socket, id, 256, 0); assert (id_size > 0 && "Get HTTP request; ID frame and then request; Failed..." )
do {
raw_size = zmq_recv (socket, raw, 256, 0); assert (raw_size >= 0 && "socket.recv() Failed..." );
} while (raw_size == 256);
char http_response [] = /* Prepares the response */
"HTTP/1.0 200 OK\r\n"
"Content-Type: text/plain\r\n"
"\r\n"
"Hello, World!";
zmq_send (socket, id, id_size, ZMQ_SNDMORE); /* Sends the ID frame followed by the response */
zmq_send (socket, http_response, strlen (http_response), 0);
zmq_send (socket, id, id_size, ZMQ_SNDMORE); /* Closes the connection by sending the ID frame followed by a zero response */
zmq_send (socket, 0, 0, 0);
}
zmq_close (socket);
zmq_ctx_destroy (ctx);
ZeroMQ zmq_getsockopt() can deliver a POSIX/SOCKET descriptor, for low-level tricks
The ZMQ_FD option shall retrieve the file descriptor associated with the specified socket. The returned file descriptor can be used to integrate the socket into an existing event loop; the ØMQ library shall signal any pending events on the socket in an edge-triggered fashion by making the file descriptor become ready for reading.
The ability to read from the returned file descriptor does not necessarily indicate that messages are available to be read from, or can be written to, the underlying socket; applications must retrieve the actual event state with a subsequent retrieval of the ZMQ_EVENTS option.
The returned file descriptor is also used internally by the zmq_send and zmq_recv functions. As the descriptor is edge triggered, applications must update the state of ZMQ_EVENTS after each invocation of zmq_send or zmq_recv.
To be more explicit: after calling zmq_send the socket may become readable (and vice versa) without triggering a read event on the file descriptor.
The returned file descriptor is intended for use with a poll or similar system call only. Applications must never attempt to read or write data to it directly, neither should they try to close it.
Option value type: int on POSIX systems, SOCKET on Windows
For more details on ZeroMQ tricks one may enjoy to read solutions, performance benchmarks, latency-shaving details and other problem-solving tricks that have already been discussed here.
Related
I want to write basic chat program. I don't release that, I just want to learn socket programming. The chat program will be between client and server.
server code:
//bla bla code
new_socket = accept(server_fd, (struct sockaddr*)&address,(socklen_t*)&addrlen);
char server_msg[128];
char client_msg[128];
int running = 1;
while(running){
fgets(server_msg, 64, stdin);
send(new_socket, server_msg, strlen(server_msg), 0);
recv(new_socket, client_msg, 128, 0);
}
client code:
char server_msg[128];
char client_msg[128];
int running = 1;
while(running){
fgets(client_msg, 64, stdin);
send(server_socket, client_msg, strlen(client_msg), 0);
recv(server_socket, server_msg, 128, 0);
}
Questions:
Is the new socket fd used only once? That means; Will I do create a new socket for each sending and receiving. Or can I use this forever?
If first question answer is "FOREVER", Will I do something to new socket fd? I don't know maybe clear.
The above code is not working as expected. As I expected. Actually, The code is working very well :D. How do I perform interrupt operations such as getting input, sending messages, receiving messages?
My English is not good, I'm sorry.
Is the new socket fd used only once? That means; Will I do create a new socket for each sending and receiving. Or can I use this forever?
you open a TCP socket connection, so, once in the connected state (the ESTABLISHED state) it remains there until one side closes it (and the other side will notice it by reading 0 bytes as result, this explanation is oversimplified to make it simpler to your case, normally you should detect the end of file in the socket, so you can close it, and accept(2) another connection)
If first question answer is "FOREVER", Will I do something to new socket fd? I don't know maybe clear.
Your program is not a chat program, as the server uses the SAME socket to write what it has read, so it will return the message sent to the same client that issued it (acting more as an echo server, than a chat)
The above code is not working as expected. As I expected. Actually, The code is working very well :D. How do I perform interrupt operations such as getting input, sending messages, receiving messages?
Normally, a chat server should wait for a message to arrive at several sockets (by means of the select(2) system call) and will determine which socket needs attention. As a result, the server will read(2) from that socket (probably in a non blocking way, to avoid being blocked on a read to a socket, that will not be interrupted if data enters on another socket) The chat server should write what it has read over all the other sockets it has connections from, so every other user of your chat system receives a copy of the message (sending back the message to the sender is optional, but not very elegant, the issuer of the message should do local echo of it, although, to ensure the user sees his messages interspersed between the ones from other users)
I recommend you to read the book "Unix Network Programming" from Richard Stevens. It describes perfectly how to use the sockets API and IMHO it is the best reference you can get to learn how to use the socket interface.
One final note:
Don't use plain numbers hardwired in your code, as it is error prone (you declare a buffer string to be 128 bytes, but then read only 64, making 64 unused bytes in the array, you can use sizeof buffer as to make the compiler to use the amount declared in the variable, or you can #define BUFFSIZE 128 and then declare char buffer[BUFFSIZE]; and later do a fgets(buffer, BUFFSIZE, socket_FILE_descriptor); if you use a constant to size a buffer, use a constant name to name it :) and use it everywhere, in case you decide to give it a different value, all the occurences of it will change, instead of having to reviste all places in your code where you have used the number 64.
How to receive message from only desired client if I know that this client has been connected to my server?
recvfrom(sockfd, buf, BUFFSIZE, MSG_PEEK,
(struct sockaddr *)&addr, &caddrlen);
addr will be replaced by corresponding data, but I want receive data from only one client
How to receive message from only desired client if I know that this client has been connected to my server?
You can't really do this with a datagram socket in the AF_INET or AF_INET6 family, unless undesired clients are blocked at a lower level, such as via a firewall. At least, not if you want to be able to continue to receive messages from desired clients after one from an undesired client arrives. The network driver queues datagrams for you, and you need to handle them in order, where the C API for "handling" a datagram is to receive it via one of the several system calls that serve that purpose, such as recvfrom().
You can discriminate between messages after receiving them, such as by discarding those from undesired clients. However, it is of limited, special-purpose use to retrieve message data without dequeueing it, as the MSG_PEEK flag to recvfrom() provides. In particular, that does not serve your stated purpose -- you will still need to receive every message, via a subsequent call that does not use MSG_PEEK. Instead, I suggest simply reading the data via recvfrom(), checking the address to determine whether it's from the client you're interested in, and handling it appropriately.
If you want to handle multiple clients at the same time, then you have some alternatives. A relatively simple one is to have a function, perhaps running in a dedicated thread, that receives all incoming messages and dispatches them appropriately according to their source addresses. Another alternative is to open a new (datagram) socket for each client, each on its own port, and to set up a protocol by which you tell each client which port to use after their first contact. Datagrams from unexpected clients on these additional ports would be erroneous, and therefore safe to reject.
Of course, the latter of those is an approximation to a connection-oriented protocol. If it seems attractive to you then perhaps you should be looking at stream sockets instead of datagram sockets. Not only would that get you no-fuss client specificity, it would also provide for reliable and guaranteed in-order communication.
recvfrom(sockfd, buf, BUFFSIZE, MSG_PEEK,
(struct sockaddr *)&addr, &caddrlen);
if(addr.sin_addr.s_addr == "ADDRESS_OF_DESIRED_CLIENT")
{
//ALLOW USER
}
For IPV4 "addr.sin_addr.s_addr" is an int, but you can also get the Address in a string from.
I am believing TCP is reliable. If write(socket, buf, buf_len) and close(socket) returns without error, the receiver will receive the exact same data of buf with length buf_len.
But this article says TCP is not reliable.
A:
sock = socket(AF_INET, SOCK_STREAM, 0);
connect(sock, &remote, sizeof(remote));
write(sock, buffer, 1000000); // returns 1000000
close(sock);
B:
int sock = socket(AF_INET, SOCK_STREAM, 0);
bind(sock, &local, sizeof(local));
listen(sock, 128);
int client=accept(sock, &local, locallen);
write(client, "220 Welcome\r\n", 13);
int bytesRead=0, res;
for(;;) {
res = read(client, buffer, 4096);
if(res < 0) {
perror("read");
exit(1);
}
if(!res)
break;
bytesRead += res;
}
printf("%d\n", bytesRead);
Quiz question - what will program B print on completion?
A) 1000000
B) something less than 1000000
C) it will exit reporting an error
D) could be any of the above
The right answer, sadly, is ‘D’. But how could this happen? Program A
reported that all data had been sent correctly!
If this article is true, I have to change my mind. But I am not sure if this article is true or not.
Is this article true?
TCP is reliable (at least when the lower level protocols are), but programmers may use it in a unreliable way.
The problem here is that a socket should not be closed before all sent data has been correctly received at the other side: the close signal may destroy the connection before while the last data is still in transfert.
The correct way to ensure proper reception by peer is a graceful shutdown.
TCP/IP is reliable, for a very particular (and limited) meaning of the word "reliable".
Specifically, when write() returns 1000000, it is making you the following promises:
Your 1000000 bytes of data have been copied into the TCP socket's outgoing-data-buffer, and from now on, the TCP/IP stack is responsible for delivering those bytes to the remote program.
If those bytes can be delivered (with a reasonable amount of effort), then they will be delivered, even if some of the transmitted TCP packets get dropped during delivery.
If the bytes do get delivered, they will be delivered accurately and in-order (relative to each other and also relative to the data passed to previous and subsequent calls to write() on that same socket).
But there are also some guarantees that write() doesn't (and, in general, can't) provide. In particular:
write() cannot guarantee that the receiving application won't exit or crash before it calls recv() to get all 1000000 of those bytes.
write() cannot guarantee that the receiving application will do the right thing with the bytes it does receive (i.e. it might just ignore them or mishandle them, rather than acting on them)
write() cannot guarantee that the receiving application ever will call recv() to recv() those bytes at all (i.e. it might just keep the socket open but never call recv() on it)
write() cannot guarantee that the network infrastructure between your computer and the remote computer will work to deliver the bytes -- i.e. if some packets are dropped, no problem, the TCP layer will resend them, but e.g. if someone has pulled the power cable out of your cable modem, then there's simply no way for the bytes to get to their destination, and after a few minutes of trying and failing to get an ACK, the TCP layer will give up and error out the connection.
write() can't guarantee that the receiving application handles socket-shutdown issues 100% correctly (if it doesn't, then it's possible for any data you send just before the remote program closes to the socket to be silently dropped, since there's no receiver left to receive it)
write() doesn't guarantee that the bytes will be received by the receiving application in the same segment-sizes that you sent them in. i.e. just because you sent 1000000 bytes in a single call to send() doesn't mean that the receiving app can receive 1000000 bytes in a single call to recv(). He might instead receive the data in any-sized chunks, e.g. 1-byte-per-recv()-call, or 1000-bytes-per-recv-call, or any other size the TCP layer feels like providing.
Note that in all of the cases listed above, the problems occur after write() has returned, which is why write() can't just return an error code when these happen. (A later call to write() might well return an error code, of course, but that won't particularly help you know where the line between delivered-bytes and non-delivered bytes was located)
TLDR: TCP/IP is more reliable than UDP, but not a 100% iron-clad guarantee that nothing will ever go wrong. If you really want to make sure your bytes got processed on the receiving end, you'll want to program your receiving application to send back some sort of higher-level acknowledgement that it received (and successfully handled!) the bytes you sent it.
TCP/IP offers reliability which means it allows for the retransmission of lost packets, thereby making sure that all data transmitted is (eventually) received or you get a timeout error. Either your stuff get delivered or you are presented with a timeout error
What is the correct way of reading from a TCP socket in C/C++?
and btw, TCP/IP is reliable to some extent, it guarantees delivery assuming the network is working... if you unplug the cable TCP/IP will not deliver your packet
P.S. you should at least advance the pointer inside the buffer buffer
void ReadXBytes(int socket, unsigned int x, void* buffer)
{
int bytesRead = 0;
int result;
while (bytesRead < x)
{
result = read(socket, buffer + bytesRead, x - bytesRead);
if (result < 1 )
{
// Throw your error.
}
bytesRead += result;
}
}
Program A reported that all data had been sent correctly!
No. The return of write in program A only means that the returned number of bytes has been delivered to the kernel of operating system. It does not claim that the data have been send by the local system, have been received by the remote system or even have been processed by the remote application.
I am believing TCP is reliable.
TCP provides a reliability at the transport layer only. This means only that it will make sure that data loss gets detected (and packets re-transmitted), data duplication gets detected (and duplicates discarded) and packet reordering gets detected and fixed.
TCP does not claim any reliability of higher layers. If the applications needs this it has to be provided by the application itself.
I am doing some test with TCP client application in a Raspberry Pi (server in the PC), with PPP (Point to Point Protocol) using a LTE Modem. I have used C program with sockets, checking system call's response. I wanted to test how socket works in a bad coverage area so I did some test removing the antenna.
I have followed the next steps:
Connect to server --> OK
Start sending data (write system call) --> OK (I also check in the server)
I removed the LTE modem's antenna (There is no network, it can't do ping)
Continue sending data (write system call) --> OK (server does not receive anything!!!)
It finished sending data and closed socket --> OK (connection is still opened and there is no data since the antenna was removed)
Program was finished
I put the antenna again
Some time later, the data has been uploaded and the connection closed. But I did another test following this steps but with more data, and it did not upload this data...
I do not know if there any way to ensure that the data written to TCP server is received by the server (I thought that TCP layer ensured this..). I could do it manually using an ACK but I guess that it has to be a better way to do.
Sending part code:
while(i<100)
{
sprintf(buf, "Message %d\n", i);
Return = write(Sock_Fd, buf, strlen(buf));
if(Return!=strlen(buf))
{
printf("Error sending data to TCP server. \n");
printf("Error str: %s \n", strerror(errno));
}
else
{
printf("write successful %d\n", i);
i++;
}
sleep(2);
}
Many thanks for your help.
The write()-syscall returns true, since the kernel buffers the data and puts it in the out-queue of the socket. It is removed from this queue when the data was sent and acked from the peer. When the OutQueue is full, the write-syscall will block.
To determine, if data has not been acked by the peer, you have to look at the size of the outqueue. With linux, you can use an ioctl() for this:
ioctl(fd, SIOCOUTQ, &outqlen);
However, it would be more clean and portable to use an inband method for determining if the data has been received.
TCP/IP is rather primitive technology. Internet may sound newish, but this is really antique stuff. TCP is needed because IP gives almost no guarantees, but TCP doesn't actually add that many guarantees. Its chief function is to turn a packet protocol into a stream protocol. That means TCP guarantees a byte order; no bytes will arrive out of order. Don't count on more than that.
You see that protocols on top of TCP add extra checks. E.g. HTTP has the famous HTTP error codes, precisely because it can't rely on the error state from TCP. You probably have to do the same - or you can consider implementing your service as a HTTP service. "RESTful" refers to an API design methodology which closely follows the HTTP philosophy; this might be relevant to you.
The short answer to your 4th and 5th topics was taken as a shortcut from this answer (read the whole answer to get more info)
A socket has a send buffer and if a call to the send() function succeeds, it does not mean that the requested data has actually really been sent out, it only means the data has been added to the send buffer. For UDP sockets, the data is usually sent pretty soon, if not immediately, but for TCP sockets, there can be a relatively long delay between adding data to the send buffer and having the TCP implementation really send that data. As a result, when you close a TCP socket, there may still be pending data in the send buffer, which has not been sent yet but your code considers it as sent, since the send() call succeeded. If the TCP implementation was closing the socket immediately on your request, all of this data would be lost and your code wouldn't even know about that. TCP is said to be a reliable protocol and losing data just like that is not very reliable. That's why a socket that still has data to send will go into a state called TIME_WAIT when you close it. In that state it will wait until all pending data has been successfully sent or until a timeout is hit, in which case the socket is closed forcefully.
The amount of time the kernel will wait before it closes the socket,
regardless if it still has pending send data or not, is called the
Linger Time.
BTW: that answer also refers to the docs where you can see more detailed info
Lets imagine the following data sequence that was sent from the server to the client:
[data] [data] [data] [FIN] [RST]
And lets imagine that I'm doing the following on the client side (sockets are non-blocking):
char buf[sizeof(data)];
for (;;)
{
rlen = recv(..., buf, sizeof(buf), ...);
rerr = errno;
slen = send(..., "a", 1, ...);
serr = errno;
}
When I will see the ECONNRESET error?
I'm particularly curious about the following edge case. Let's imagine that all IP frames for the imagined sequence above are already received and ACKed by the TCP stack. However, my client application didn't send() or recv() anything yet. Will the first call to send() return an ECONNRESET? If so, will the next call to recv() succeed and allow me to read everything it has in its internal buffers (since it received the data and has it) before starting to report ECONNRESET (or returning 0 because of FIN)? Or something different will happen?
I will especially appreciate link on the documentation that explains that situation. I'm trying to grok linux tcp implementation to figure that out, but it's not that clear...
Will the first call to send() return an ECONNRESET?
Not unless it blocks for long enough for the peer to detect the incoming packet for the broken connection and return an RST. Most of the time, send will just buffer the data and return.
will the next call to recv() succeed
It depends entirely on (a) whether there is incoming that a to be read and (b) whether an RAT has been received yet.
and allow me to read everything it has in its internal buffers (since it received the data and has it) before starting to report ECONNRESET (or returning 0 because of FIN)?
If an RST is received, all buffered data will be thrown away.
It all depends entirely on the timing and on the size of the buffers at both ends.