I'm creating a C application daemon that I want to be able to interact with through named pipes. For example, I write in the shell echo hello > /tmp/appname/interface, and the application reads it and does stuff with what I just echoed into the named pipe.
I've successfully managed to implement this with a singular named pipe with read(). However, I want to be able to listen to many different named pipes at the same time. I don't think spawning a new thread for each named pipe is a very good solution. I am also concerned about whether constantly listening for something to be written to the named pipes with read() will use an unnecessarily high amount of CPU. Is there a better way to approach this?
Related
This is more design question to hear ideas from other people.
I'm writing a software written in c language completely.
It has multiple processes running and now I'm thinking to add logging.
First idea is writing a simple log function may be in separate library.
Then, I may need to add some kind of lock to prevent race condition to access the log file.
The second idea is spawning a separate log process and the log process is accessing the log file.
And the other processes can send log message to the log process using IPC mechanism such as pipe or unix socket.
In that case, is there any way to set buffer for the pipe or unix socket?
Or it is not worth to consider buffering for IPC recipient?
I'm writing a C program in which I need to pass messages between child processes and the main process. But the thing is, I need to do it without using the functions like msgget() and msgsnd()
How can I implement? What kind of techniques can I use?
There are multiple ways to communicate with children processes, it depends on your application.
Very much depends on the level of abstraction of your application.
-- If the level of abstraction is low:
If you need very fast communication, you could use shared memory (e.g. shm_open()). But that would be complicated to synchronize correctly.
The most used method, and the method I'd use if I were in your shoes is: pipes.
It's simple, fast, and since pipes file descriptors are supported by epoll() and those kind of asynchronous I/O APIs, you can take advantage from this fact.
Another plus is that, if your application grows, and you need to communicate with remote processes (processes that are not in your local machine), adapting pipes to sockets is very easy, basically it's still the same reading/writing from/to a file descriptor.
Also, Unix-domain sockets (which in other platforms are called "named pipes") let you to have a server process that creates a listening socket with a very well known name (e.g. an entry in the filesystem, such as /tmp/my_socket) and all clients in the local machine can connect to that.
Pipes, networking sockets, or unix-domain sockets are very interchangeable solutions, because - as said before - all involve reading/writing data from/to a file descriptor, so you can reuse the code.
The disadvantage with a file descriptor is that you're writing data to a stream of bytes, so you need to implement the "message streaming protocol" of your messages by yourself, to "unstream" your messages (marshalling/unmarshalling), but that's not so complicated in the most of the cases, and that also depends on the kind of messages you're sending.
I'd pass on other solutions such as memory mapped files and so on.
-- If the level of abstraction is higher:
You could use a 3rd party message passing system, such as RabbitMQ, ZMQ, and so on.
I would like to inject a shared library into a process (I'm using ptrace() to do that part) and then be able to get output from the shared library back into the debugger I'm writing using some form of IPC. My instinct is to use a pipe, but the only real requirements are:
I don't want to store anything on the filesystem to facilitate the communication as it will only last as long as the debugger is running.
I want a portable Unix solution (so Unix-standard syscalls would be ideal).
The problem I'm running into is that as far as I can see, if I call pipe() in the debugger, there is no way to pass the "sending" end of the pipe to the target process, and vice versa with the receiving end. I could set up shared memory, but I think that would require creating a file somewhere so I could reference the memory segment from both processes. How do other debuggers capture output when they attach to a process after it has already begun running?
I assume that you are in need of a debugging system for your business logic code (I mean application). From my experience, this kind of problem is tackled with below explained system design. (My experience is in C++, I think the same must hold good for the C based system also.)
Have a logger system (a separate process). This will contain - logger manager and the logging code - which will take the responsibility of dumping the log into hard disk.
Each application instance (process running in Unix) will communicate to this process with sockets. So you can have your own messaging protocol and communicate with the logger system with socket based communication.
Later, for each of this application - have a switch which can switch off/on the log.So that you can have a tool - to send signal to this process to switch on/off the message logging.
At a high level, this is the most generic way to develop a logging system. In case you need any information - Do comment it. I will try to answer.
Using better search terms showed me this question is a dup of these guys:
Can I share a file descriptor to another process on linux or are they local to the process?
Can I open a socket and pass it to another process in Linux
How to use sendmsg() to send a file-descriptor via sockets between 2 processes?
The top answers were what I was looking for. You can use a Unix-domain socket to hand a file descriptor off to a different process. This could work either from debugger to library or vice versa, but is probably easier to do from debugger to library because the debugger can write the socket's address into the target process while it injects the library.
However, once I pass the socket's address into the target process, I might as well just use the socket itself instead of using a pipe in addition.
I am writing a simple instant messenger program in C on Linux.
Right now I have a program that binds a socket to a port on the local machine, and listens for text data being sent by another program that connected to my local machine IP and port.
Well, I can have this client send text data to my program, and have it displayed using stdout on my local machine; however, I cannot program a way to send data back to the client machine, because my program is busy listening and displaying the text sent by the client machine.
How would I go about either creating a new process (that listens and displays the text sent to it by the client machine, then takes that text and sends it to the other program's stdout, while the other program takes care of stdin being sent to the client machine) or create 2 programs that do the separate jobs (sending, receiving, and displaying), and sends the appropriate data to one another?
Sorry if that is weirdly worded, and I will clarify if need be. I looked into exec, execve, fork, etc. but am confused as to whether this is the appropriate path to look in to, or if there is a simpler way that I am missing.
Any help would be greatly appreciated, Thank you.
EDIT: In retrospect, I figured that this would be much easier accomplished with 2 separate programs. One, the IM server, and the others, the IM clients.
The IM Clients would connect to the IM server program, and send whatever text they wanted to the IM server. Then, the IM server would just record the data sent to it in a buffer/file with the names/ip's of the clients appended to the text sent to it by each client, and send that text (in format of name:text) to each client that is connected.
This would remove the need for complicated inter-process/program communication for stdin and stdout, and instead, use a simple client/server way of communicating, with the client programs displaying text sent to it from server via stdout, and using stdin to send whatever text to the server.
With this said, I am still interested in someone answering my original question: for science. Thank you all for reading, and hopefully someone will benefit from my mental brainstorming, or whatever answers come from the community.
however, i cannot program a way to send data back to the client machine, because my program is busy listening and displaying the text sent by the client machine.
The same socket that was returned from a listening-socket by accept() can be used for both sending and receiving data. So your socket is never "busy" just because you're reading from it ... you can write back on the same socket.
If you need to both read and write concurrently, then share the socket returned from accept() across two different threads. Since two different buffers are being used by the networking stack for sending and receiving on the socket, a dedicated thread for reading and another dedicated thread for writing to the socket will be thread-safe without the use of mutexes.
I would go with fork() - create a child process and now you have two different processes that can do two different things on two different sockets- one can receive and the other can send. I have no personal experience with coding a client/server like this yet, but that would be my first stab at solving your issue...
As #bdonlan mentioned in a comment, you definitely need a multiplexing call like select or preferably poll (or related syscalls like pselect, ppoll ...). These multiplexing calls are the primitive to wait on several channels at once (with pselect and ppoll able to atomically wait for both I/O events and signals). Read also the select tutorial man page. Of course, you can wait for several file descriptors, and you can wait for both reading & writing abilities (even on the same socket, if needed), in the same select or poll syscall.
All event-based loops and frameworks are using these multiplexing calls (like poll or select). You could also use libevent, or even (particularly when coding a graphical user interface application) some GUI toolkit like Gtk or Qt, which are all based around a central event loop.
I don't think that having a multi-process or multi-threaded application is useful in your case. You just need some event loop.
You might also ask to get a SIGIO signal when data arrives on your socket using fcntl with F_SETOWN, but this is not very useful for you. Then you often want to have your socket non-blocking.
What are they and how do they work?
Context happens to be SQL Server
Both on Windows and POSIX systems, named-pipes provide a way for inter-process communication to occur among processes running on the same machine. What named pipes give you is a way to send your data without having the performance penalty of involving the network stack.
Just like you have a server listening to a IP address/port for incoming requests, a server can also set up a named pipe which can listen for requests. In either cases, the client process (or the DB access library) must know the specific address (or pipe name) to send the request. Often, a commonly used standard default exists (much like port 80 for HTTP, SQL server uses port 1433 in TCP/IP; \\.\pipe\sql\query for a named pipe).
By setting up additional named pipes, you can have multiple DB servers running, each with its own request listeners.
The advantage of named pipes is that it is usually much faster, and frees up network stack resources.
--
BTW, in the Windows world, you can also have named pipes to remote machines -- but in that case, the named pipe is transported over TCP/IP, so you will lose performance. Use named pipes for local machine communication.
Unix and Windows both have things called "Named pipes", but they behave differently. On Unix, a named pipe is a one-way street which typically has just one reader and one writer - the writer writes, and the reader reads, you get it?
On Windows, the thing called a "Named pipe" is an IPC object more like a TCP socket - things can flow both ways and there is some metadata (You can obtain the credentials of the thing on the other end etc).
Unix named pipes appear as a special file in the filesystem and can be accessed with normal file IO commands including the shell. Windows ones don't, and need to be opened with a special system call (after which they behave mostly like a normal win32 handle).
Even more confusing, Unix has something called a "Unix socket" or AF_UNIX socket, which works more like (but not completely like) a win32 "named pipe", being bidirectional.
Linux Pipes
First In First Out (FIFO) interproccess communication mechanism.
Unnamed Pipes
On the command line, represented by a "|" between two commands.
Named Pipes
A FIFO special file. Once created, you can use the pipe just like a normal file(open, close, write, read, etc).
To create a named pipe, called "myPipe", from the command line (man page):
mkfifo myPipe
To create a named pipe from c, where "pathname" is the name you would like the pipe to have and "mode" contains the permissions you want the pipe to have (man page):
#include <sys/types.h>
#include <sys/stat.h>
int mkfifo(const char *pathname, mode_t mode);
According to Wikipedia:
[...] A traditional pipe is "unnamed" because it exists anonymously and persists only for as long as the process is running. A named pipe is system-persistent and exists beyond the life of the process and must be "unlinked" or deleted once it is no longer being used. Processes generally attach to the named pipe (usually appearing as a file) to perform IPC (inter-process communication).
Compare
echo "test" | wc
to
mkdnod apipe p
wc apipe
wc will block until
echo "test" > apipe
executes
This is an exeprt from Technet (so not sure why the marked answer says named pipes are faster??):
Named Pipes vs. TCP/IP Sockets
In a fast local area network (LAN) environment, Transmission Control Protocol/Internet Protocol (TCP/IP) Sockets and Named Pipes clients are comparable with regard to performance. However, the performance difference between the TCP/IP Sockets and Named Pipes clients becomes apparent with slower networks, such as across wide area networks (WANs) or dial-up networks. This is because of the different ways the interprocess communication (IPC) mechanisms communicate between peers.
For named pipes, network communications are typically more interactive. A peer does not send data until another peer asks for it using a read command. A network read typically involves a series of peek named pipes messages before it starts to read the data. These can be very costly in a slow network and cause excessive network traffic, which in turn affects other network clients.
It is also important to clarify if you are talking about local pipes or network pipes. If the server application is running locally on the computer that is running an instance of SQL Server, the local Named Pipes protocol is an option. Local named pipes runs in kernel mode and is very fast.
For TCP/IP Sockets, data transmissions are more streamlined and have less overhead. Data transmissions can also take advantage of TCP/IP Sockets performance enhancement mechanisms such as windowing, delayed acknowledgements, and so on. This can be very helpful in a slow network. Depending on the type of applications, such performance differences can be significant.
TCP/IP Sockets also support a backlog queue. This can provide a limited smoothing effect compared to named pipes that could lead to pipe-busy errors when you are trying to connect to SQL Server.
Generally, TCP/IP is preferred in a slow LAN, WAN, or dial-up network, whereas named pipes can be a better choice when network speed is not the issue, as it offers more functionality, ease of use, and configuration options.
Pipes are a way of streaming data between applications. Under Linux I use this all the time to stream the output of one process into another. This is anonymous because the destination app has no idea where that input-stream comes from. It doesn't need to.
A named pipe is just a way of actively hooking onto an existing pipe and hoovering-up its data. It's for situations where the provider doesn't know what clients will be eating the data.
Inter-process communication (mostly) for Windows Applications. Similar to using sockets to communicate between applications in Unix.
MSDN
Named pipes in a unix/linux context can be used to make two different shells to communicate since a shell just can't share anything with another.
Furthermore, one script instantiated twice in the same shell can't share anything through the two instances. I found a use for named pipes when coding a daemon that contains the start() and stop() function, and I wanted to use the same script to perform the two actions.
Without named pipes (or any kind of semaphore) starting the script in the background is not a problem. The thing is when it finishes you just can't access the instance in background.
So when you want to send him the stop command you just can't: running the same script without named pipes and calling the stop() function won't do anything since you are actually running another instance.
The solution was to implement two pipes, one READ and the other WRITE when you start the daemon. Then make him, among its other tasks, listen to the READ pipe. Then the Stop() function contains a command that will write a message in the pipe, that will be handled by the background running script that will perform an exit 0. This way our second instance of the same script has only on task to do: tell the first instance to stop.
This way one and only one script can start and stop itself.
Of course you have different ways to do it by triggering the stop via a touch for example. But this one is nice and interesting to code.
Named pipes is a windows system for inter-process communication. In the case of SQL server, if the server is on the same machine as the client, then it is possible to use named pipes to tranfer the data, as opposed to TCP/IP.