I want to send lot of strings (~250000) for <1sec from C application to a C# application. When I do it with WM_COPYDATA and SendMessage, my C# application hangs. What else can I do? Named pipes are included only in .NET 4, and I'm using .NET 2.
EDIT:
I'm gonna stick to WM_COPYDATA and appending to a list (which is a fast operation). Then post processing this list.
The fastest option is probably to use named pipes via P/Invoke. This is still much higher performance than most other IPC options.
Shared memory or MMF is the fastest method. It's as fast as kernel objects, used for signalling about data availability are. And, more importantly, you can first open the shared memory, then put your data directly there (saves you one copy operation) and signal to other application. That other application can consume the data directly from shared memory (again, no need to copy).
Not the fastest on win32 currently, but worth investigating: 0mq
Uses TCP sockets on Windows, but very efficiently.
For a closed source solution I don't think 29 West's Ultra Messaging can easily be trumped, includes a rare feature of zero-copy messaging in .net
Related
I have an application that began its life as a C#-based Windows GUI that used marshalling to talk to a C DLL.
I now need to separate the Windows client and DLL so that the client is installed on a remote PC and communicates with the C DLL over the internet.
A further complication is that I want to have multiple Windows clients connecting to the C DLL.
This whole world is new to me, so excuse me if the following are naive questions.
My questions:
0) What is the best method for having the client communicate with the DLL over the internet? TCP/IP Sockets?
1) I need to make modifications to my DLL to have it service multiple clients. But I need some piece of middleware that collects the queries from the different clients, feeds them to the DLL, and then sends the results back to the appropriate client. Is there any code (such as node.js) that would facilitate this?
Regarding: What is the best method for having the client communicate with the DLL over the internet?
Your suggestion of using TCP/IP could certainly (and likely will) be part of the solution, but there will be other components of the solution as well. The direction you choose will in part be made by answering whether you are using standard marshaling (COM), or custom? At the very least, your problem description suggests a scenario requiring interprocess communications.
There are many ways to implement. This diagram maps out a general approach, that based on your description might apply:
Components of Interprocess Communications
Read more here
Regarding: make modifications to my DLL to have it service multiple clients...
The dll is simply a file like any other. Several processes can read, and subsequently own content from, a file as long as the processes doing the reading adhere to common file access rules. I do not think you will have to modify your dll, at least for that reason. Just make sure the processes accessing the dll comply with safe file access protocols. (Safe file access).
I have a Windows UI, written in C#, that calls a DLL, written in C.
Data is exchanged between the C# UI and C DLL using the marshaling techniques available through pInvoke. Both the UI and the the DLL are legacy code.
All of the software runs on the cloud; specifically, on Amazon Web Services (AWS). But it is portable to any cloud service provider (Azure, Google, etc).
I need to write a new piece of C code ("NewCode"), that runs on a separate AWS (or other) instance, that does nothing except read data from a proprietary database and service data requests from the existing DLL.
For lots of reasons, this NewCode needs to run on its own instance, so that it has its own, exclusive access to memory, cpu, and disk. Newcode needs to service a variety of data requests: a single number, a char string, an array of numbers, array of strings, etc. NewCode will be portable C, so it can run under Linux, Unix, etc.
My question:
What are my options for having the existing C DLL communicate with NewCode? I know it is too broad a topic to ask for a list of options and their relative merits, so all I'm asking for here is to what should be on the list so I can begin my research. I am a complete newbie in this area, but so far I have determined that on the list should be sockets and pipes. What else should be on the list?
Since NewCode will be communicating over the network, I would look into protocol buffers. Protocol buffers would likely be the most efficient for communicating between to processes on separate machines who may be running different operating systems. There are protocol buffer implementations for many different languages, all of which use the same predefined structure definitions.
Of course, there are other options, like XML, JSON, or your own binary protocol.
https://code.google.com/p/protobuf/
I am trying to create two applications. One application should take inputs from user like name, address, phone number and send that information to the other application to store it. This should also be capable of reading the stored address information from the other application.
My assumptions for this:
I am planning to use system() process in application1 to create application2.
For communication between these processes, shared memory as IPC.
Can anyone suggest me whether this is the correct way for this task or is there any best and easy approach for this task.
Thank you.
You could take a look at google protocol buffers if you are looking for communication between processes in python , java or c++.
It is clean and elegant and works across files and sockets.
You can communicate using sockets between the two processes.
I'm currently writing an HTTP server in C so that I'll learn about C, network programming and HTTP. I've implemented most of the simple stuff, but I'm only handling one connection at a time. Currently, I'm thinking about how to efficiently add multitasking to my project. Here are some of the options I thought about:
Use one thread per connection. Simple but can't handle many connections.
Use non-blocking API calls only and handle everything in one thread. Sounds interesting but using select()s and such excessively is said to be quite slow.
Some other multithreading model, e.g. something complex like lighttpd uses. (Probably) the best solution, but (probably) too difficult to implement.
Any thoughts on this?
There is no single best model for writing multi-tasked network servers. Different platforms have different solutions for high performance (I/O completion ports, epoll, kqueues). Be careful about going for maximum portability: some features are mimicked on other platforms (i.e. select() is available on Windows) and yield very poor performance because they are simply mapped onto some other native model.
Also, there are other models not covered in your list. In particular, the classic UNIX "pre-fork" model.
In all cases, use any form of asynchronous I/O when available. If it isn't, look into non-blocking synchronous I/O. Design your HTTP library around asynchronous streaming of data, but keep the I/O bit out of it. This is much harder than it sounds. It usually implies writing state machines for your protocol interpreter.
That last bit is most important because it will allow you to experiment with different representations. It might even allow you to write a compact core for each platform local, high-performance tools and swap this core from one platform to the other.
Yea, do the one that's interesting to you. When you're done with it, if you're not utterly sick of the project, benchmark it, profile it, and try one of the other techniques. Or, even more interesting, abandon the work, take the learnings, and move on to something completely different.
You could use an event loop as in node.js:
Source code of node (c, c++, javascript)
https://github.com/joyent/node
Ryan Dahl (the creator of node) outlines the reasoning behind the design of node.js, non-blocking io and the event loop as an alternative to multithreading in a webserver.
http://www.yuiblog.com/blog/2010/05/20/video-dahl/
Douglas Crockford discusses the event loop in Scene 6: Loopage (Friday, August 27, 2010)
http://www.yuiblog.com/blog/2010/08/30/yui-theater-douglas-crockford-crockford-on-javascript-scene-6-loopage-52-min/
An index of Douglas Crockford's above talk (if further background information is needed). Doesn't really apply to your question though.
http://yuiblog.com/crockford/
Look at your platforms most efficient socket polling model - epoll (linux), kqueue (freebsd), WSAEventSelect (Windows). Perhaps combine with a thread pool, handle N connections per thread. You could always start with select then replace with a more efficient model once it works.
A simple solution might be having multiple processes: have one process accept connections, and as soon as the connection is established fork and handle the connection in that child process.
An interesting variant of this technique is used by SER/OpenSER/Kamailio SIP proxy: there's one main process that accepts the connections and multiple child worker processes, connected via pipes. The parent sends the new filedescriptor through the socket. See this book excerpt at 17.4.2. Passing File Descriptors over UNIX Domain Sockets. The OpenSER/Kamailio SIP proxies are used for heavy-duty SIP processing where performance is a huge issue and they do very well with this technique (plus shared memory for information sharing). Multi-threading is probably easier to implement, though.
The user, administrators and support staff need detailed runtime and monitoring information from a daemon developed in C.
In my case these information are e.g.
the current system health, like throughput (MB/s), already written data, ...
the current configuration
I would use JMX in the Java world and the procfs (or sysfs) interface for a kernel module. A log file doesn't seem to be the best way.
What is the best way for such a information interface for a C daemon?
I thought about opening a socket and implementing a bare-metal http or xmlrpc server, but that seems to be overkill. What are alternatives?
You can use a signal handler in your daemon that reacts to, say USR1, and dumps information to the screen/log/net. This way, you can just send the process a USR1 signal whenever you need the info.
You could listen on a UNIX-domain socket, and write regularly write the current status (say once a second) to anyone who connects to it. You don't need to implement a protocol like HTTP or XMLRPC - since the communication will be one-way just regularly write a single line of plain text containing the state.
If you are using a relational database anyway, create another table and fill it with the current status as frequent as necessary. If you don't have a relational database, write the status in a file, and implement some rotation scheme to avoid overwriting a file that somebody reads at that very moment.
Write to a file. Use a file locking protocol to force atomic reads and writes. Anything you agree on will work. There's probably a UUCP locking library floating around that you can use. In a previous life I found one for Linux. I've also implemented it from scratch. It's fairly trivial to do that too.
Check out the lockdev(3) library on Linux. It's for devices, but it may work for plain files too.
I like the socket idea best. There's no need to support HTTP or any RPC protocol. You can create a simple application specific protocol that returns requested information. If the server always returns the same info, then handling incoming requests is trivial, though the trivial approach may cause problems down the line if you ever want to expand on the possible queries. The main reason to use a pre-existing protocol is to leverage existing libraries and tools.
Speaking of leveraging, another option is to use SNMP and access the daemon as a managed component. If you need to query/manage the daemon remotely, this option has its advantages, but otherwise can turn out to be greater overkill than an HTTP server.