In ANSI C on Windows what is the best/easiest way to transfer data between two programs running locally without needing to write/read from a file? Data will be basic text and only one program will be sending and the other receiving. Thanks.
Inter-process communication is inherently platform-dependent; "ANSI C" doesn't have anything to say about this, but you should start here for Windows:
http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx
Much depends on the kinds of applications you're talking about, and the volume of data, and how tightly coupled the processes are.
Related
I am thinking of creating a C program that will use picocom to read and write serial streams to ports (GPS module, NTPD). The c program will run on a Debian OS.
Is it possible to do this using Picocom? Is it bad design to interact with Picocom through c and system commands?
While that's technically possible, it's a bad idea; picocom is a somewhat uncommon tool, and it's primarily designed to be used interactively by a user. Trying to use it from within another application will be pretty weird.
It'll be much easier to interact with the serial port directly from your application. There's a nice introduction to doing that in the answer to How to open, read, and write from serial port in C.
I'm developing a program that will need to run on Internet servers (a back-end component to be used by several cross-platform programs). I'm familiar with the security precautions to take (to prevent buffer overflows and SQL Injection attacks, for instance), but have never written a server program before, or any program that will be used on this scale.
The program needs to be able to serve hundreds or thousands of clients simultaneously. The protocols are designed for processing speed and to minimize the amount of data that must be exchanged, and the server side will be written in C. There will be both a Windows and a Linux version from the same code.
Questions:
How should the program handle communications -- multiple threads, a single thread handling all the sockets in turn, or spawn a new process for every so many incoming connections (or for each one)?
Do I need to worry about things like memory fragmentation, since this program will need to run for months at a time?
What other design issues, specific to this kind of programming, might an experienced developer of cross-platform programs for desktop and mobile systems not be aware of?
Please, no suggestions to use a different language. That decision has already been made, for reasons I'm not at liberty to go into.
For I'd use libevent or libev and non-blocking I/O. This way the operating system will take case of most of your scheduling problems. I'd also use a thread pool for processing tasks, that by nature are blocking, so they don't block the main loop. And if you ever need to read or write large amounts of data to or from the disc, use mmap, again to let the OS handle as much as possible.
The basic advice is use the OS, as much as possible. If you want a good example of a program which does this look at Varnish, it is very well written, and performs fantastic.
With my experience running multiple servers for over 3 years of uptime, and programs with little over a year of uptime I can still recommend making the setup so that the system gracefully recovers from a program error and from a server reboot.
Even though performance gets a hit when a program is restarted, you need to be able to handle that as external circumstances can force the program to such a restart.
Don't try to reinvent the wheel when not needed, and have a look at zeromq or something like that to handle distribution of incoming communications. (If you are allowed to, prototype the backends in a more forgiving language than C like Python, then reimplement in C but keeping the communications protocol)
I have a simple server written in C. It's main purpose is to communicate with some business partners over a proprietary protocol. For that reason and a few others, it must be written in C. I have a number of other processes, however, written in other languages (e.g. Python) that must communicate with the server (locally, on the same Linux server).
What are the best options for cross-language IPC in this scenario? Specifically, I think I have a handle on transport technologies: Unix domain sockets, named pipes, shared memory, ZeroMQ (Crossroads). I'm more interested in the best way to implement the protocol, in order to keep the C code small and maintainable, while still allowing communication from other languages.
Edit: there seems to be some confusion. I'm not interested in discussion of pros/cons of domain sockets, shared memory et. al. I am interested in msgpack (thanks unwind), and other technologies/approaches for implementing the wire protocol.
It's hard to optimize (=select the "best") when the requirements are unknown. You do state that your goal is to keep the C code "small and maintainable", which seems to imply that you should look for a library. Perhaps msgpack over a local socket?
Also, your basic premise that the server must be written in C because you have a proprietary protocol seems ... weird, at least.
Edit: What you need is a "serialization framework", i.e. something can turn a memory structure into a byte stream. The best candidates are:
Protocol Buffers
MessagePack
JSON
Pros/cons:
Protocol Buffers
+ Fast
+ Easy to version (which you'll start to love very much when you need to make a change to your message format for the first time and which you will curse to hell before that)
- Solves many problems which you don't know about, yet. That makes the API a bit "strange". I assure you, there are very good reasons for what and how they do it but you will feel confused sometimes.
I don't know much about MessagePack.
Lastly:
JSON
+ Any language out there can read and write JSON data.
+ Human readable without tools
- somewhat slow
- the format is very flexible but if you need to make big changes, you need to find a strategy to figure out what format (= which fields) a message has when you read it.
As for the transport layer:
Pros/cons:
Shared memory
+ Fastest option
- You need a second channel (like a semaphore) to tell the other process that the data is now ready
- gets really ugly when you try to connect more then two processes
- OS specific
Named pipes
+ Very easy to set up
+ Fairly fast
- Only allows two processes to talk ... or rather one process to talk to another in a single direction. If you need bi-directional communication, you need several pipes
Sockets
+ Pretty easy to set up
+ Available for all and any languages
+ Allows remote access (not all processes need to be on the same machine)
+ Two-way communication with one server and several processes
- Slower than shmem and pipes
ZeroMQ
+ Like sockets but better
+ Modern API (not that old IPC/socket junk)
+ Support for many languages...
- ...but not all
If you can I'd suggest to try ZeroMQ because it's a modern framework that solves many of the problems that you'll encounter with the older technologies.
If that fails, I'd try sockets next. They are easy, well supported and docile.
I have been asked this question in some recent interviews,Whats the advantages and disadvantages of using Socket in IPC when there are other ways to perform IPC.Have not found exact answer .
Any help would be much appreciated.
Compared to pipes, IPC sockets differ by being bidirectional, that is, reads and writes can be done on the same descriptor. Pipes, unlike sockets, are unidirectional. You have to keep a pair of descriptors if you want to do both reads and writes.
Pipes, on the other hand, guarantee atomicity when reading or writing under a certain amount of bytes. Writing something less than PIPE_BUF bytes at once is guaranteed to be delivered in one chunk and never observed partial. Sockets do require more care from the programmer in that respect.
Shared memory, when used for IPC, requires explicit synchronisation from the programmer. It may be the most efficient and most flexible mechanism, but that comes at an increased complexity cost.
Another point in favour of sockets: an app using sockets can be easily distributed - ie. it can be run on one host or spread across several hosts with little effort. This depends of course on the nature of the app.
Perhaps this is too simplified an answer, yet it is an important detail. Sockets are not supported on all OS's. Recently, I have been aware of a project that used sockets for IPC all over the place only to find that they were forced to change from Linux to a proprietary OS which was POSIX, but did not support sockets the same way as Linux.
Sockets allow you a few benefits...
You can connect a simple client to them for testing (manually enter data, see the response).
This is very useful for debugging, simulating and blackbox testing.
You can run the processes on different machines. This can be useful for scalability and is very helpful in debugging / testing if you work in embedded software.
It becomes very easy to expose your process as a service
But there are drawbacks as well
Overhead is greater than IPC optimized for a single machine. Shared memory in particular is better if you need the performance, and you know your processes are all on the same machine.
Security - if your client apps can connect so can anyone else, if you're not careful about authentication. Data can also be sniffed if you're not encrypting, and modified if you're not at least signing data sent over the wire.
Using a true message queue tends to leave you with fixed sized messages. If you have a large number of messages of wildly varying sizes this can become a performance problem. Using a socket can be a way around this, though you're then left trying to wrap this functionality to become identical to a queue, which is tricky to get the detail right on, particularly aspects like blocking/non-blocking and atomicity.
Shared memory is quick but requires management (you end up writing a version of malloc to manage the SHM) plus you have to synchronise and lock it in some way. Though you can use libraries to help with this the availability depends on your environment and language.
Queues are easy but have the downsides listed as pros to my socket discussion.
Pipes have been covered by Blagovests answer to this question.
As is ever the case with this kind of stuff I would suggest reading the W. Richard Stevens books on IPC and sockets. There is no better explanation than his! :-)
I understand the differences between a multithreaded program and a program relying on inter-machine communication. My problem is that I have a nice multithreaded program written in 'C' that works and runs really well on an 8-core machine. There is now opportunity to port this program to a cluster to gain access to more cores. Is it worth the effort to rip out the pthread stuff and retrofit MPI (which I've never used) or are we better off recoding the whole thing (or most of it) from scratch? Assume we are "stuck" with C so a wholesale change of language isn't an option.
Depending on how your software is written, there may or may not be advantages to going to MPI over keeping your pthread implementation.
Unfortunately (or fortunately), message passing is a very different beast than pthreading - the basic assumption is quite different. I love this quote from Joshua Phillips of the Maestro team: "The difference between message-passing and shared-state communication is equivalent to the difference between sending a colleague an e-mail requesting her to complete a task and opening up her organizer to write down the task directly in her to-do list. More than just being rude, the latter is likely to confuse her – she might erase it, not notice it, or accidentally prioritize it incorrectly."
Unfortunately, the way you share data is very different. There is no direct access to data in other threads (since it can be on other machines), so it can be a very daunting task to migrate from pthreads to MPI. On the other hand, if the code is written so each thread is isolated, it can be an easy task, and definitely worthwhile.
In order to determine how useful this will be, you'll need to understand the code, and what you hope to achieve by switching. It can be worthwhile as a learning experience (you learn a LOT about synchronization and threading by working in MPI), but may not be practical if the gains will be minor.
Re. your comment to Reed -- this sounds like an easy, low-overhead conversion to MPI. Just be careful: not all MPI APIs support dynamic creation of processes, i.e., you start your program with N processes (specified at startup) and you're stuck with N processes throughout the life-time of the program.